Show HN: A Database Written in Golang

Cool!

You watched CMU's intro to Database sytems, right? It's really good and thorough. It will save you of some common pitfalls and can help you navigate the trade-offs.

https://www.youtube.com/watch?v=otE2WvX3XdQ&list=PLSE8ODhjZX...

Tostino · 10 months ago

I really have liked the current Optimizer series. So much amazing info.

I hope some company that heavily relies on Postgres sponsors a project to replace the optimizer with a more modern implementation. I feel like for any large applications with olap style queries (even if on the OLTP database) will benefit incredibly from that work. SQL Servers optimizer was light-years ahead of Postgres when I worked with it last over a decade ago.

alecco · 10 months ago

Sadly, most of those corporations would rather spend millions in proprietary solutions than give back 100k to the community. I've seen it first hand many times. They only give money if they absolutely have to.

skrtskrt · 10 months ago

I also recommend "Database Internals" to people. An invaluable book, the logical next read after "Designing Data-Intensive Applications". It's the best and most approachable resource on distributed systems and database consistency / performance tradeoffs I have found.

BiraIgnacio · 10 months ago

Oh thanks for the link, this is really great!

Sahil121 · 10 months ago

sure man

hobs · 10 months ago

Looks fun, but the intro requirement is "Knows C++" - that's kinda a non starter right?

alecco · 10 months ago

I'm sure that's for the homework.

Dead Comment

I didnt look into it yet but i already wanted to say - cuz of curiosity i build my own graph database in golang :) and i learned alot. so i absolutly understand why you did it and what experiences you probably made on the way :D

congratz !

varelaseb · 10 months ago

Is this published anywhere?

voodooEntity · 10 months ago

Kinda. So first i went for a standalone application back than to use it as storage/database. This part is archived now. it was released under: https://github.com/voodooEntity/slingshotdb

At this point i think its important to mention that its nativly an in-memory graph database/storage.

At some point (4-5 years ago) i decided to change the way i handle it. Since im using the storage in many private projects, and there was no real traction (while also maintaining a database is quite some work) i decided i gonne change the way i handle it away from a standalone database to a library that works as an in-memory graph storage.

Also i added a custom query language including builder (the queries are 100% json compatible so you can in theory build queries from whatever language and transmit it via json.

The library is released under : https://github.com/voodooEntity/gits

Currently im completly reworking the query parser but since this is a privat project and im the only maintainer it may take some to to finish it. The update will takle some smaller bugs in the query parser that can occur on very complex nested queries and also will optimize query speeds. I cant provide a date when the update will occure i only can say it will (because i need it myself).

Finally said: The strength of gits is to be very fast (hence in memory handling) and easy to use from golang. Its threadsafe by default.

Sahil121 · 10 months ago

thanks man

dveeden2 · 10 months ago

Other databases written in Go:

- TiDB by PingCAP

- Vitess by PlanetScale

Both are basically only the SQL part as TiDB uses TiKV (written in Rust) and Vitess uses MySQL.

For those who want to implement a database in Go but without having to implement a network protocol there is go-mysql, which allows you to do this: https://github.com/go-mysql-org/go-mysql/blob/master/cmd/go-... As demonstration I created a networked SQLite: https://github.com/dveeden/go-mysql-examples/blob/main/go-my...

Both TiDB and Vitess have parsers that can be used outside standalone. So if you only want to implement your own on disk format, this can help.

Note that I'm working for PingCAP on TiDB and I'm also a co-maintainer for go-mysql.

aqueueaqueue · 10 months ago

We use Tidb at work at scale. Great product! Was looking at the source today to understand an error code.

raggi · 10 months ago

Vitess came from YouTube

CBLT · 10 months ago

Didn't the Vitess team found planetscale?

twalla · 10 months ago

Someone asked about resources and I ran into this while evaluating embedded db options for a golang project - it's a collection of db components implemented in golang:

https://github.com/thomasjungblut/go-sstables

tjungblut · 10 months ago

thank you for referring, feel free to ask any questions you may have

littlemerman · 10 months ago

You might be interested in this golang db from mit’s database systems course:

https://github.com/MIT-DB-Class/go-db-2024 https://dsg.csail.mit.edu/6.5830/

will surely checkout thanks man

mbreese · 10 months ago

This looks like a good exploratory project!

One thing I’d add to the readme is an example of how you’d use the database in an example application. From the docs, it’s clear that this isn’t a sql database (yet?), so it would be good to have an example to see how to use the database.

It might also be nice to have a description of what happens when you insert or get a record, so others can learn from the code too. Or can you comment here about what your favorite part of the code is? What did you figure out that you didn’t know before? If you’re using the project to learn about databases, what have you learned so far?

thanks man yes currently it is not a fully sql database (have to add query support) My fav part of code is the data retrieval code because had some issues there & got to know lot about it

Learnt about B+ Trees, transactions, managing concurrent reads & data presistence

bob1029 · 10 months ago

If you are wanting proper SQL command support, you could copy the SQLite parser approach. Properly parsing all valid command texts is not a problem that I would find compelling unless I was being compensated for it.

https://www.sqlite.org/lemon.html

You could probably use something like participle, but you'd have to translate the grammar.

https://github.com/alecthomas/participle

qaq · 10 months ago

You can use https://github.com/cockroachdb/cockroachdb-parser which is basically PGSQL compatible parser implemented in golang

mingodad · 10 months ago

I do have a big collection of LALR(1) grammars to test/study/develop/document here https://mingodad.github.io/parsertl-playground/playground/ including sqlite, tidb, vites, postgresql, mysql, ...

yes man have to work on adding the sql cmd support thanks for the links

pighive · 10 months ago

Nice, did you follow a course or any resources to build it? Please share, I have a similar goal. Thank you.

rebelmackerel · 10 months ago

I took a quick glance at the code, I believe it may build upon "Build Your Own Database From Scratch in Go" [0]. The first part of the book is available for free on the author's website, along with information on how to purchase the full book (which includes source code).

I share the same goal and am working through the material after working through Codecrafters' "Build your own SQLite" [1]. Good luck!

I apologize in advance for mistakes (formatting, et cetera). I just registered this account to point you toward resources I found helpful.

[0] https://build-your-own.org/database/

[1] https://app.codecrafters.io/courses/sqlite/overview

croskcool · 10 months ago

Does something like this exist for java, don't want to take the burden of a new language.