For context, Raph Koster is not only the person who designed Ultima Online, but generally considered one of the top theoricians in game design. This isnt some guy who has a small blog and came up with a neat explanation for how some word came about in the early days of the web. Its the guy who did it.
I find it incredibly hard to believe that ultima online was the first application of database sharding, but it may very well be the source of the name. "key-based partitioning" doesn't have the same ring to it, but has existed since the early 90's and possibly late 80's.
I can't seem to find it right now, but I read an article, shared in the "awesome-scalability" GitHub repo, which referenced a 1978 paper, written by a couple of IBM researchers, about sharding and a lot of seemingly modern database things.
Agreed. I have worked at a Fortune 100 firm where its business ran (still runs) off of an ERP sharded into 100 instances, each representing one key facility since the mid-90s. It took extra work in the early days to centralize analytics and manage deployments and data, but it has been very resilient (low blast radius in today's terminology) and natural for low risk deployments, rolling out of upgrades to one site, then the rest in batches.
> I spent some time crawling through the proceedings of Very Large Databases (VLDB) and the ACM Digital Library, and I could find no instances of "shard" used to mean the partitioning of a database prior to 2001. (That paper is "Minerva: An automated resource provisioning tool for large-scale storage systems" in Transactions on Computer Systems, free-to-read at https://dl.acm.org/doi/abs/10.1145/502912.502915.)
> Other the other hand, I found many papers citing the SHARD paper - more than the official count. That's a difficulty with citation counts of old papers: a lot of the papers citing it are also old papers, and we're not consistent at tracking the citations of old papers. Personally, I don't have a conclusion. The SHARD paper is decently cited, and its usage is close to the modern one. On the other hand, I can't find any smoking gun pre-1997 usage of "shard" in the modern meaning.
I started my digging thinking I would quickly find a paper using "shard" in the modern database context that predated Ultima Online. I could not find it, so now I think it's plausible.
A lot of innovations came from MMOs, so it shouldn’t be that surprising. Especially the big ones push the cutting edge in a way that no other applications do.
One MMO tech I always got a kick out of was WoW using Bit Torrent for updates at a time when it was massive and that bandwidth would have been brutal to output.
"serve [seemingly] <50ms latency N:N coordination between thousands of clients, sometimes who are all literally in the same place (events, cities etc) expecting to observe their and other users actions ~immediately"
This still sounds like a really hard problem, I can't imagine trying to do it 20 years ago. (WoW turns 20 in Nov, and it wasn't the first, though perhaps the first to reach such supercilious scale?)
Interesting, I had always assumed shard came from a specific company's technology and never gave it much thought. Kinda like how we talk about blade servers because of ibm's marketing.
That was the best seller in history, at the time, but there are mechanical computers that played similar tricks going back to the early days of punch cards.
The Hollerith Tabulator won a competition in 1888.
Yes, the technique is obviously predated by Ultima Online. The question we're trying to answer is specifically about the etymology of sharding in the database context.
Dan, is there a way to automatically add related older discussions to the top of a new item, in a similar way to submitting the same URL within a certain time will take you to the existing discussion rather than create a new one?
I have never played modern Ultima. I played a very early version on the Mac SE when I was about 6 and I didn’t understand anything that was going on. I just rode around aimlessly. The one thing I remember is that the menu had a “negate time” option and when I selected it a message came up that said “you don’t have two coins to rub together”.
So I just rode around not knowing what was going on but hoping to find two coins so I could rub them together and negate time.
Ultima Online in the age of dialup and early DSL was a history-altering game for MMO's. I still log on to UO Forever servers every once in a while (a modified game server with accelerated skill learning) to level up a character and scrabble together enough gold to build and furnish a house. It was such a vast game and the sheet depth of it for a game of that time was unprecedented. I still think to this day it is one of the best "just go do what you want to do" MMOs out there, something that New World didn't even get right 20 years after with considerably more resources and advances behind it.
Runescape is the only other game from that era (and which shares a lot of gameplay similarities) that has that much staying power.
For me, the best part of UO was the ability to extend it, for me OSI was never pleasant to play on, but I was on an unofficial shard with Dragonlance saga background and it was the best time of my life, I don't think any other MMO could've been so extendible to fully implement that saga at those level of details
since Ultima deals with time travel, I accept your "history altering" claim. Long live Lord British! (RG has been to the deepest sea and the space station, so who can say he doesnt also have a time machine. He is a real adventurer who made Ultima- one of the best adventure rpg lines ever)
Ultima Online is a very unique social experience compared to today's landscape of multiplayer games, and there are different "Eras" of the game which can impart a particular flavor to the experience, eras like: UO: Renassaince, UO: Lost Lands (aka T2A mechanics), and others.
Its a sandbox game just like Arcage, Neverwinter Nights, Albion Online, Everquest, Star Wars Galaxy, Eve Online, Black Desert Online, etc in which the game's goals tacitly produce a local society, typically from players' in-game avatar "Characters" just playing the game. PvP is emphasized on many of the servers.
Right now there is a very good UO Freeshard server called "UO Outlands" with hybrid rules that are quite satisfying and with updated maps/dungeons/landscapes designed to encourage the classic vibes you've heard about and aren't really available in other, modern games.
If you just want to take a look, there is a Youtuber named "Trammie", he prefers to play a pure-thief character and he semi-regularly records streams of live play. Its a good look at one high-level play style that is nearly unique to UO and still available.
Raph Koster is also currently working on a new game, that aims to take this ideas of social experience and player-driven virtual society and push it beyond what they were able to do with UO. (Also with less PK-ing, griefing, etc. - things he regretted about UO, but many people today consider synonymous with sandbox)
Interestingly enough, this new game is NOT going to have any shards or sharding at all - not even instancing - but just many (procedurally generated) planets, that are all part of the same shared universe and economy.
Thus, some of the very unique ambitions behind UO and SWG might actually become part of a new, more modern game:
Trammie Surprise, as he was formerly known, also runs the SQZD clan, who are often spotted in the wilds and around their homes, PvPing, thieving, and whatnot. They're one portal for easing into the local communities in Outlands.
Easing into the mechanics, minutiae, and overall obscurity of the game is a different beast entirely, but a lot of things in there, while worn out over time, have a handworn feel like none other, which itch just the right niche.
You played Ultima II (1982). It had a "negate time" command to stop time for enemies in combat, but it used "strange coins". I think it only used one, the "two coins to rub together" was just a figure of speech.
Reading the manual would probably have helped but being 6, I understand how you didn't (or maybe didn't even have access to it). I had similar experiences with other games when I was a similar age :)
Haha yeah I don't think there was a manual. I'm pretty sure it was pirated on a floppy disk or something with a label written in biro in my step father's desk.
It wasn't until much later (maybe mid-teens) that I saw/heard the expression "not having two coins to rub together" as being used as a euphamism for being poor and it clicked into place.
I played Renaissance and the following one on unofficial servers for about 6 years between 2003 and 2010. My offline and online friends would easily play 6-8 hours per day, and when we were offline we were talking about it. It was like a second life and an incredible escape, but looking back at it we were all addicted to it pretty badly. I get flashbacks and cravings to this day.
semi-related, but also helps me to believe that this is the case (and not only because the different regional servers were called "shards" in Ultima Online):
in the "Game Coding Complete, Fourth Edition" book by two programmers who worked on Ultima and Sims (and other Origin/EA games of the time) back in the day, they share some war stories of programming, and if memory serves there is a portion where they talk about the original design, and the realization that lead to the sharding and how the login and shard system worked in the game.
Also, unrelated, a really neat war story about a guy who put in debug code to generate certain audio cues while a game was running to catch a bug.
The book all in all was a fun read if only for all these stories, and generally remember good coding guidelines as well but it is using older C++ that may not stand up to modern critique.
Ha re: audio cues for debugging… your pc speaker is truly an underused tool when debugging something infrequent… for example our system processes a lot of xml data and usually it’s fine but for our test suite hearing beeps and knowing there are server side issues immediately is a great thing
I ended up digging the book out and finding the passage; hopefully it's ok to share because it's an amazing story and helps illustrate what makes that book so great to me:
"The best caveman debugging solution I ever saw was one that used the PC speaker. Herman was a programmer who worked on Ultima V through Ultima IX, and one of his talents was perfect pitch. He could tell you the difference between a B and a B flat and get it right every time. He used this to his advantage when he was searching for the nastiest crasher bugs of them all - they didn't even allow the debugger window to pop up. He wrote a special checker program that output specific tones through the PC speaker and peppered the code with these checks. If you walked into his office while his spiced-up version of the game was running, it sounded a little like raw modem noise, until the game crashed. Because the PC speaker wasn't dependent on the CPU, it would remain emitting the tone of his last check. "Hmm...that's a D," he would say, and zero in on the line of code that caused the crash."
I bought a copy of Game Coding Complete a long time ago, but couldn't remember which edition and your comment made me curious. Turns out I have the first edition from 2003.
Hmmm... I vaguely recall the term "shards" when referring to DB/2 running on S36, S38, and later newfangled AS/400 across the world. When presenting the data in a single pane, some "shards" would come in late, or be broken, and require reconnecting.
The wikipedia entry for this confirms two sources depending on context, and in your case it came from a late 80s paper:
Sarin, DeWitt & Rosenberg, Overview of SHARD: A System for Highly Available Replicated Data, Technical Report CCA-88-01, Computer Corporation of America, May 1988
From a quick search right now, the term seems to have come from that same system, but the first reference appears to be older than 1988. It looks to be possibly 1985.
In a comment at the bottom it references a paper published by a few people working jointly with Computer Corporation of America, MIT & Boston University.
If you view that referenced paper "Correctness Conditions for Highly Available Databases" by N. Lynch, B. Blaustein & M. Siegel (https://apps.dtic.mil/sti/pdfs/ADA171427.pdf), and look at section 1.2 it clearly describes "SHARD: (System for Highly Available Replicated Data)" as being underdevelopment at CCA. It also says if you want to learn more about Shard, see the paper's reference [SBK]. Checking out the references section of that paper it lists the following for [SBK]:
Sarin, S. K., Blaustein, B. T., and Kaufman, C. W., "System Architecture for Partition Tolerant Distributed Databases," IEEE Transactions on Computers C-34, 12 (December 1985). pp. 1158-1163.
Which means there was a paper published in 1985, describing the in development Shard system.
It is possible that in 1985 they hadn't yet named the system "Shard", and it only got that name by 1988 - but it'd be interesting to check out that 1985 paper and see if they used the term Shard at all.
I seem to remember a conversation here on HN not too long ago where people tried to reconstruct the history behind "sharding" and, in particular, tried to find that 1980s paper you mentioned – without success. I believe they even contacted one of the authors.
I found that paper title as well when looking into this exact question. That paper does not have the number of citations I would expect if it is the source of the term. It's possibly the source, but it's not obviously the source.
I worked with db2 people back in 98 and they talked about shards. They worked down in some cave in the basement that smelled like folgers coffee, feet, and camel filter bbq sauce. I would offer them a roll of TP each and every time they said the word. Their office chairs had seat-cushions that looked like they were pulled out of a dumpster fire, worn out and melty looking. With all the red meat, nicotine, and caffeine, Im assuming someone sharded there at least once. Maybe that is what got those seat cushions all worn out.
AS one of the early big players of UO (In that it consumed an entire wall of machines in the Intel Game 'DRG' Lab in the late 90s - Shards came from UO.
And the concept as described in how he brought it from the Sosarian Lore is laser etched into my head, because along the same fantastical lines we also have the infamous The Dark Crystal - and so having that be a strong element in the SciFi-Fantasy DNA of anyone of my generation into gaming, sci-fi etc - it was completely grokked immediately and understood.
Readit News
https://www.youtube.com/watch?v=zyVTxGpEO30&pp=ygUPcmFwaCBrb...
Deleted Comment
Deleted Comment
However, I then dug into it a bit. From my digging 4 years ago (https://news.ycombinator.com/item?id=23460200):
> I spent some time crawling through the proceedings of Very Large Databases (VLDB) and the ACM Digital Library, and I could find no instances of "shard" used to mean the partitioning of a database prior to 2001. (That paper is "Minerva: An automated resource provisioning tool for large-scale storage systems" in Transactions on Computer Systems, free-to-read at https://dl.acm.org/doi/abs/10.1145/502912.502915.)
> Other the other hand, I found many papers citing the SHARD paper - more than the official count. That's a difficulty with citation counts of old papers: a lot of the papers citing it are also old papers, and we're not consistent at tracking the citations of old papers. Personally, I don't have a conclusion. The SHARD paper is decently cited, and its usage is close to the modern one. On the other hand, I can't find any smoking gun pre-1997 usage of "shard" in the modern meaning.
I started my digging thinking I would quickly find a paper using "shard" in the modern database context that predated Ultima Online. I could not find it, so now I think it's plausible.
This still sounds like a really hard problem, I can't imagine trying to do it 20 years ago. (WoW turns 20 in Nov, and it wasn't the first, though perhaps the first to reach such supercilious scale?)
[1] https://pages.cs.wisc.edu/~dewitt/includes/paralleldb/ieee90...
https://en.m.wikipedia.org/wiki/IBM_1401
That was the best seller in history, at the time, but there are mechanical computers that played similar tricks going back to the early days of punch cards.
The Hollerith Tabulator won a competition in 1888.
https://www.census.gov/history/www/innovations/technology/th...
Anyway, yahoo probably sharded by the mid to late 1990s.
https://news.ycombinator.com/item?id=35479553 (90 comments)
https://news.ycombinator.com/item?id=23438399 (172 comments)
https://news.ycombinator.com/item?id=425765 (3 comments)
https://news.ycombinator.com/item?id=17926566 (1 comment)
https://news.ycombinator.com/item?id=16343939 (1 comment)
And related:
https://news.ycombinator.com/item?id=36848605
Database “sharding” came from Ultima Online? - https://news.ycombinator.com/item?id=35479553 - April 2023 (89 comments)
Database “sharding” came from Ultima Online? (2009) - https://news.ycombinator.com/item?id=34694397 - Feb 2023 (1 comment)
Database “sharding” came from Ultima Online? (2009) - https://news.ycombinator.com/item?id=23438399 - June 2020 (172 comments)
Database “sharding” came from Ultima Online? (2009) - https://news.ycombinator.com/item?id=17926566 - Sept 2018 (1 comment)
Database “sharding” came from Ultima Online? (2009) - https://news.ycombinator.com/item?id=16343939 - Feb 2018 (1 comment)
Did the term 'sharding' start with Ultima Online? - https://news.ycombinator.com/item?id=425765 - Jan 2009 (3 comments)
As for https://news.ycombinator.com/item?id=36848605 - I've added it to https://news.ycombinator.com/highlights. Thanks!
So I just rode around not knowing what was going on but hoping to find two coins so I could rub them together and negate time.
Runescape is the only other game from that era (and which shares a lot of gameplay similarities) that has that much staying power.
Its a sandbox game just like Arcage, Neverwinter Nights, Albion Online, Everquest, Star Wars Galaxy, Eve Online, Black Desert Online, etc in which the game's goals tacitly produce a local society, typically from players' in-game avatar "Characters" just playing the game. PvP is emphasized on many of the servers.
Right now there is a very good UO Freeshard server called "UO Outlands" with hybrid rules that are quite satisfying and with updated maps/dungeons/landscapes designed to encourage the classic vibes you've heard about and aren't really available in other, modern games.
If you just want to take a look, there is a Youtuber named "Trammie", he prefers to play a pure-thief character and he semi-regularly records streams of live play. Its a good look at one high-level play style that is nearly unique to UO and still available.
Interestingly enough, this new game is NOT going to have any shards or sharding at all - not even instancing - but just many (procedurally generated) planets, that are all part of the same shared universe and economy.
Thus, some of the very unique ambitions behind UO and SWG might actually become part of a new, more modern game:
https://starsreach.com/ (Fair warning: the graphics are still very early...)
Easing into the mechanics, minutiae, and overall obscurity of the game is a different beast entirely, but a lot of things in there, while worn out over time, have a handworn feel like none other, which itch just the right niche.
Reading the manual would probably have helped but being 6, I understand how you didn't (or maybe didn't even have access to it). I had similar experiences with other games when I was a similar age :)
It wasn't until much later (maybe mid-teens) that I saw/heard the expression "not having two coins to rub together" as being used as a euphamism for being poor and it clicked into place.
in the "Game Coding Complete, Fourth Edition" book by two programmers who worked on Ultima and Sims (and other Origin/EA games of the time) back in the day, they share some war stories of programming, and if memory serves there is a portion where they talk about the original design, and the realization that lead to the sharding and how the login and shard system worked in the game.
Also, unrelated, a really neat war story about a guy who put in debug code to generate certain audio cues while a game was running to catch a bug.
The book all in all was a fun read if only for all these stories, and generally remember good coding guidelines as well but it is using older C++ that may not stand up to modern critique.
[0]: https://www.amazon.com/dp/1133776574
I ended up digging the book out and finding the passage; hopefully it's ok to share because it's an amazing story and helps illustrate what makes that book so great to me:
"The best caveman debugging solution I ever saw was one that used the PC speaker. Herman was a programmer who worked on Ultima V through Ultima IX, and one of his talents was perfect pitch. He could tell you the difference between a B and a B flat and get it right every time. He used this to his advantage when he was searching for the nastiest crasher bugs of them all - they didn't even allow the debugger window to pop up. He wrote a special checker program that output specific tones through the PC speaker and peppered the code with these checks. If you walked into his office while his spiced-up version of the game was running, it sounded a little like raw modem noise, until the game crashed. Because the PC speaker wasn't dependent on the CPU, it would remain emitting the tone of his last check. "Hmm...that's a D," he would say, and zero in on the line of code that caused the crash."
- Game Coding Complete, Fourth Edition
Gamedev people came up with creative devices, like using pedal to debug-break into the game.
At guru level, of course, you no longer need the radio.
Old gamedev books really have some wild stuff.
Sarin, DeWitt & Rosenberg, Overview of SHARD: A System for Highly Available Replicated Data, Technical Report CCA-88-01, Computer Corporation of America, May 1988
https://en.wikipedia.org/wiki/Shard_(database_architecture)#...
Following this link here: https://shkspr.mobi/blog/2021/06/where-is-the-original-overv...
In a comment at the bottom it references a paper published by a few people working jointly with Computer Corporation of America, MIT & Boston University.
If you view that referenced paper "Correctness Conditions for Highly Available Databases" by N. Lynch, B. Blaustein & M. Siegel (https://apps.dtic.mil/sti/pdfs/ADA171427.pdf), and look at section 1.2 it clearly describes "SHARD: (System for Highly Available Replicated Data)" as being underdevelopment at CCA. It also says if you want to learn more about Shard, see the paper's reference [SBK]. Checking out the references section of that paper it lists the following for [SBK]:
Sarin, S. K., Blaustein, B. T., and Kaufman, C. W., "System Architecture for Partition Tolerant Distributed Databases," IEEE Transactions on Computers C-34, 12 (December 1985). pp. 1158-1163.
Which means there was a paper published in 1985, describing the in development Shard system.
It is possible that in 1985 they hadn't yet named the system "Shard", and it only got that name by 1988 - but it'd be interesting to check out that 1985 paper and see if they used the term Shard at all.
And the concept as described in how he brought it from the Sosarian Lore is laser etched into my head, because along the same fantastical lines we also have the infamous The Dark Crystal - and so having that be a strong element in the SciFi-Fantasy DNA of anyone of my generation into gaming, sci-fi etc - it was completely grokked immediately and understood.
UO is one of the golden eras of my gaming DNA.
Let me cut through the hype: they’re better than you think. Real legends.
I was early in the distributed database game, these guys were earlier.
Legends.