First, it uses CMake to build - for a long time Google projects had seemed pretty anti-CMake (for example using gyp, plain Makefiles or autotools) so it's nice to see them using CMake. IMO it's the best build tool, though all build tools generate various levels of hate :-)
Second it's another Google project that generates good developer docs from source code using doxygen and markdown. These docs look good on github directly (https://github.com/google/flatbuffers/tree/master/docs/sourc...) as they are markdown, and even better on the dedicated site where they have custom css.
If I were to write a C++ library, I'd definitely copy these 2 approaches.
Thanks! I guess it's because we're a game development group inside Google, who are more externally focused than most Google engineers. We wanted to ensure the library is attractive to outside developers, hence CMake and other choices (like not having any dependencies).
In short: They first used JSON, if I understood correctly. Then
"In last six months, we have transitioned most of Facebook on Android to use FlatBuffers as the storage format. Some performance improvement numbers include:
Story load time from disk cache is reduced from 35 ms to 4 ms per story.
Transient memory allocations are reduced by 75 percent.
This is indeed a very interesting encoding. One part of the optimization comes from the use of binary encoding and the other from using offset tables to optimize random access. Offset tables also allows to modify random data in the structure without having to reencode the whole information. This comes at the price of less compact data.
It looks like I could update my YABE encoding which is a straight JSON binary encoding.
For those are are totally new to Flatbuffers, here is a good video by Colt from Google. It was primarily created for game developers but technically any app can use it - those who are not relying on a library that handles networking and has implementation for JSON.
He also mentioned Flatbuffer use in his recent at Android meetup in San Francisco and we debated it's benefits again. It has a learning curve but well worth it: https://www.youtube.com/watch?v=iQTxMkSJ1dQ
OK, so if I understand this correctly, a binary format where you can seek to arbitrary positions to read only the data you need to currently display is a huge performance win versus a big blob of text you have to read and fully parse. As someone who has had to uninstall FaceBook from my Android phone due to its poor performance, excuse me if I'm bewildered that they didn't realize this years ago (FlatBuffers isn't the first binary serialization format) and write it that way in the first place!
Facebook(the company) values putting any functioning software out there over putting optimized software out. This is intentional, but it leads to bad experiences for people like you.
There's a lot more to it than just being binary. The big deal is that it can be accessed in-place while still being portable and forwards/backwards compatible. For example, Protocol Buffers is binary too, but requires unpacking, causing lots of object allocation, etc.
I didn't like the writeup, because I can't tell how the layout works from his bad description of it. The author makes that classic mistake of technical writing in which the writing assumes the reader already knows what they are talking about.
I ran across Cap'n Proto (https://capnproto.org/) a while back and at a glance it seems to do something similar. Is that true? Are you familiar with the tradeoffs between the two?
They are missing the second major reason why flatbuffers is awesome, cache locality.
It's incredibly hard to layout objects in memory with Java but if you don't mind the lookup hit on bytebuffers flatbuffers is a great way to structure data in the patterns you access it.
All this stuff is pretty old-hat to game dev people bit it's nice to see mainstream dev start caring a bit more about performance.
Good point. Lookup in Java is a bit slower than in the language FlatBuffers was originally designed for (C++), but being able to bypass Java's object allocation may well make up for that.
In general, objects allocated in sequence will be consecutive in Eden. After GC moves stuff around, objects referred to by object X often follow object X itself. YMMV.
Nice to see Facebook acknowledge contributions from Google. It's a departure from typical not-invented-here mentality prevalent at large tech companies.
Readit News
First, it uses CMake to build - for a long time Google projects had seemed pretty anti-CMake (for example using gyp, plain Makefiles or autotools) so it's nice to see them using CMake. IMO it's the best build tool, though all build tools generate various levels of hate :-)
Second it's another Google project that generates good developer docs from source code using doxygen and markdown. These docs look good on github directly (https://github.com/google/flatbuffers/tree/master/docs/sourc...) as they are markdown, and even better on the dedicated site where they have custom css.
If I were to write a C++ library, I'd definitely copy these 2 approaches.
"In last six months, we have transitioned most of Facebook on Android to use FlatBuffers as the storage format. Some performance improvement numbers include:
Story load time from disk cache is reduced from 35 ms to 4 ms per story.
Transient memory allocations are reduced by 75 percent.
Cold start time is improved by 10-15 percent.
We have reduced storage size by 15 percent."
It looks like I could update my YABE encoding which is a straight JSON binary encoding.
So they took advantage of that fact by not inflating ("flattening") the data structure using a stream.
He also mentioned Flatbuffer use in his recent at Android meetup in San Francisco and we debated it's benefits again. It has a learning curve but well worth it: https://www.youtube.com/watch?v=iQTxMkSJ1dQ
FWIW, I help maintain FlatBuffers for Go and Python. I'm happy to answer any questions I can.
The filesystem is a fantastic way to persist data for 80-90% of mobile applications.
Well Sqlite provides an extensive query language. I don't think Android's file system does.
It's incredibly hard to layout objects in memory with Java but if you don't mind the lookup hit on bytebuffers flatbuffers is a great way to structure data in the patterns you access it.
All this stuff is pretty old-hat to game dev people bit it's nice to see mainstream dev start caring a bit more about performance.