Leading with unfair benchmarks is the fastest way to lose credibility in the database space. The companies that last are the ones doing honest technical work.
I remember a friend showing me some stuff about this when they were still trying to pursue the MMORPG route and I am surprised this company is still alive given the amount of unnecessary overengineering that went into that effort.
The model of putting ECS like logic as code and putting all that code as stored procs solves a handful of db transaction issues that none of the major mmorpgs really had issues with 20 years ago when mmorpgs were all the rage and when computers were 100x-1000x slower than today. A lot of engineering is going to solving and cutting out these database round trips that were not even an issue 20 years ago and even less of an issue today.
On the other hand the actual hard parts about building a multiplayer game, such as roll back, physics, collisions, all of that sort of stuff, well this actually doesn't really help with any of that. If anything it makes all of those things even more difficult. I imagine that the developers of this game probably had to come up with some pretty interesting hacks that probably didn't benefit from their DB to get anything resembling physics to work.
Layered on top of that are the known anti-patterns around using stored procs within databases, which spacetimedb does't address at all. You can't really run your tests just on your client because the application logic is split between client and db-server leading to spaghetti code and poor iteration cycles for game design. The database itself also doesn't horizontally scale beyond a single node.
Just take a peek at the steam reviews for BitCraft. The recent ones are actually pretty scathing and it seems like, despite all this engineering, the developers haven't really listened to their playerbase. With all of this effort centered around building a custom database, users still can't, say, build homes for themselves in this game, something that you can certainly do in probably most all other MMORPGs.
I think this will go down as a cautionary example of not listening to your users and spending engineering effort on the wrong types of problems. I think one of the most important skills for an engineer is actually not just having good technical ideas but knowing the right types of problems to solve and, most importantly, listening to your audience, your users, and your stakeholders. I wish the developers the best of luck and hope they can pivot successfully.
The SpacetimeDB launch video came up on my YouTube feed - and I was surprised the video didn't go at all into how it was implemented. I assumed it was proprietary magic, but then I was surprised to find it was opensource. That confused me more - to get the kind of semantics they were talking about I had assumed that it was pretty novel and if it was open source they should be leading with that.
Kind of bummed to see its essentially 2015-era React Flux in Rust around a mutex.
Broadly enjoyed my read, though I do have a nitpick here:
>The absence of side effects or stalls cannot be enforced by the type system ...
Side effects is probably correct, but stalls would imply wasm code that calls out to a long-time blocking function - that's generally quite easy to type-system-ify, and wasm stuff often does so with promise-like constructs. So there would be a need for spacetimedb-side markers for "this func might do HTTP", but that kind of marker for WASM-contact-able code is very much a normal expectation.
If they don't have that kind of marker, and do allow blocking calls in their beta API, then yeah - huge problem with that kind of internal structure (shared global lock), completely agreed.
There’s also an excellent paper called “Fair Benchmarking Considered Difficult:
Common Pitfalls In Database Performance Testing”.
Benchmarks are genuinely hard. Modern systems and hardware are notoriously complex and have different behaviors across runs and with minor tweaks to performance settings.
Yep, absolutely... and in the end benchmarks aren't going to sell a database anyways. Reliability and provable correctness and ease of integration are probably going to win in the end. It's a dubious marketing edge to lead with.
PostgreSQL itself is remarkably slow in many configurations and leaves a lot of performance on the table due to its architecture and assumptions. But it works and people trust it. So they use it.
I appreciate the very even-handed way this was written, not fanning flames but bringing it back to a choice between trade-offs that all DB developers can/should make.
Very cool watching late stage software shops undiscover n tier architecture. Of course, breaking from hard learned wisdom for niche use cases is fine… until you bury the reality in a misleading and edge lord-ish PR sweep.
I watched a few of their talks, and while I'm somewhat impressed by the results, it's not exactly surprising to me that you can get those numbers when you make the trade-offs they're making. Most applications cannot make those trade-offs and, thus, cannot ever hope to see those numbers. The lack of rigor in database benchmarking in any of the videos I saw was an obvious indicator of something to be wary of.
One of the problems with running the application code inside the database server is that you need to make application developers use the programming language(s) you prepared for them. The other is that you need to trust them to not do crash out, or do infinite loops. Good luck moving application developers from their choice into the right choice.
This is easy to fix. The SpacetimeDB folks could just add a phase to analyze the application developer’s code before running it to prove that it will definitely halt.
This is why you need to compile your code for spacetimedb to WebAssembly and it runs inside a VM. Use the language of your choice, and assume the code is only semi-trusted with the ability to have a watchdog. That part makes sense.
"Because the system is, well, a hash table with a lock in front of it."
This actually surprises me? I was a very early employee/contractor working on this product but left after 3 months. But this wasn't the shape it had when I was there. It would have been better described as a CoW tree of various sorts, which lets reads and writes advance on separate transactions and reconciles at commit with what I recall was fairly minimal lock-holding time.
When I was there there was no global lock being held while a WASM "reducers" was executing. They operated on their own separate transaction in their own thread.
I haven't looked at their source since -- and I really don't want to because I work on similar things I don't want IP contamination as last I looked they had some sort of weird license -- but there's been .. 3 years? of development on it since and I know Tyler well enough and he hired smart-ish people.
I had a lot of criticisms of what he was making and the way it was being made... but ehh....I would want to confirm in source that this claim is accurate, it makes me a bit suspicious of the accuracy and fairness of the rest of TFA.
Ok, so I had a clanker go look. It actually sounds as bad as the article says. For some reason they apparently lock things while "reducer"s are running.
Robot brain summarized:
"Your historical description may be completely accurate.
Your request that the author substantiate the claim was warranted.
But the source substantiates it extremely clearly.
The blog’s concurrency conclusion is right.
Its “hash table,” fairness, “cannot sync WAL,” and “questionable to call it a database” language is deliberately contemptuous and sometimes technically sloppy."
The model of putting ECS like logic as code and putting all that code as stored procs solves a handful of db transaction issues that none of the major mmorpgs really had issues with 20 years ago when mmorpgs were all the rage and when computers were 100x-1000x slower than today. A lot of engineering is going to solving and cutting out these database round trips that were not even an issue 20 years ago and even less of an issue today.
On the other hand the actual hard parts about building a multiplayer game, such as roll back, physics, collisions, all of that sort of stuff, well this actually doesn't really help with any of that. If anything it makes all of those things even more difficult. I imagine that the developers of this game probably had to come up with some pretty interesting hacks that probably didn't benefit from their DB to get anything resembling physics to work.
Layered on top of that are the known anti-patterns around using stored procs within databases, which spacetimedb does't address at all. You can't really run your tests just on your client because the application logic is split between client and db-server leading to spaghetti code and poor iteration cycles for game design. The database itself also doesn't horizontally scale beyond a single node.
Just take a peek at the steam reviews for BitCraft. The recent ones are actually pretty scathing and it seems like, despite all this engineering, the developers haven't really listened to their playerbase. With all of this effort centered around building a custom database, users still can't, say, build homes for themselves in this game, something that you can certainly do in probably most all other MMORPGs.
I think this will go down as a cautionary example of not listening to your users and spending engineering effort on the wrong types of problems. I think one of the most important skills for an engineer is actually not just having good technical ideas but knowing the right types of problems to solve and, most importantly, listening to your audience, your users, and your stakeholders. I wish the developers the best of luck and hope they can pivot successfully.
Kind of bummed to see its essentially 2015-era React Flux in Rust around a mutex.
>The absence of side effects or stalls cannot be enforced by the type system ...
Side effects is probably correct, but stalls would imply wasm code that calls out to a long-time blocking function - that's generally quite easy to type-system-ify, and wasm stuff often does so with promise-like constructs. So there would be a need for spacetimedb-side markers for "this func might do HTTP", but that kind of marker for WASM-contact-able code is very much a normal expectation.
If they don't have that kind of marker, and do allow blocking calls in their beta API, then yeah - huge problem with that kind of internal structure (shared global lock), completely agreed.
There’s also an excellent paper called “Fair Benchmarking Considered Difficult: Common Pitfalls In Database Performance Testing”.
Benchmarks are genuinely hard. Modern systems and hardware are notoriously complex and have different behaviors across runs and with minor tweaks to performance settings.
PostgreSQL itself is remarkably slow in many configurations and leaves a lot of performance on the table due to its architecture and assumptions. But it works and people trust it. So they use it.
This actually surprises me? I was a very early employee/contractor working on this product but left after 3 months. But this wasn't the shape it had when I was there. It would have been better described as a CoW tree of various sorts, which lets reads and writes advance on separate transactions and reconciles at commit with what I recall was fairly minimal lock-holding time.
When I was there there was no global lock being held while a WASM "reducers" was executing. They operated on their own separate transaction in their own thread.
I haven't looked at their source since -- and I really don't want to because I work on similar things I don't want IP contamination as last I looked they had some sort of weird license -- but there's been .. 3 years? of development on it since and I know Tyler well enough and he hired smart-ish people.
I had a lot of criticisms of what he was making and the way it was being made... but ehh....I would want to confirm in source that this claim is accurate, it makes me a bit suspicious of the accuracy and fairness of the rest of TFA.
Robot brain summarized:
"Your historical description may be completely accurate. Your request that the author substantiate the claim was warranted. But the source substantiates it extremely clearly. The blog’s concurrency conclusion is right. Its “hash table,” fairness, “cannot sync WAL,” and “questionable to call it a database” language is deliberately contemptuous and sometimes technically sloppy."
touche