0

I am interested how multi datacenter (multi homing in Google terms) systems handle writes to strongly consistent user data? Is the master replica for each user placed in the same datacenter and writes are done through log structured queue (or directly forcing the user to wait)?

Alternatively, do companies use an architecture when the user has some "home" cluster dependend on user location? Do they migrate user data if she travels around the globe?

For this question's purposes, exclude Dynamo like databases with eventual consistency.

Robert Harvey
  • 199,517
TohaSt
  • 1
  • 1
    At that sort of scale, everything is eventual consistency. A "successful write" is merely writing it to enough master replicas that a plurality agree that the write was successful. – Telastyn Apr 11 '19 at 22:46
  • What you're looking for is a consensus algorithm, e.g. Paxos. This can be done with hard consistency guarantees, even across multiple datacenters, but it's difficult to make it fast enough. You might want to read the Google paper on the Spanner database. – Daniel Pryden Apr 11 '19 at 23:28
  • have you read amazon/google docs on the matter? AFAIK, AWS RDS has decent information on their architecture and what to expect – aaaaa says reinstate Monica Apr 12 '19 at 00:45
  • This borders on being way too broad. Each company/service likely works somewhat differently after all. – jwenting Apr 12 '19 at 10:13
  • @DanielPryden Quoting Spanner paper: "For example, an application might store each end-user’s data in its own directory, which would enable user A’s data to have three replicas in Europe, and user B’s data to have five replicas in North America." Do they migrate data if user moves to other continent? – TohaSt Apr 12 '19 at 14:16
  • @TohaSt: If you have replicas on every continent, then you don't need to migrate data at all. One common solution is to pick an initial set of regions at random, but to add a replica in a new region each time data gets "hot" there. This can be because the user in question has been sending you requests from there, or can be from other activities (e.g. if you run your batch processing in Europe, then you might get a replica there, even if your users are in Asia, or vice versa). – Daniel Pryden Apr 12 '19 at 16:55
  • @TohaSt: I think the key insight is that, at scale, you need to assume that you have users everywhere. And as your scale increases, more and more of your users aren't stationary: they move from place to place. And your users in different regions interact with each other, which means that that data needs to flow back and forth; geographic location of data can't be static. – Daniel Pryden Apr 12 '19 at 16:59
  • @DanielPryden If we have replicas on every continent, I don't think every replica can accept writes. GCP Spanner three continent offering lets write to US datacenter only, so users in Asia experience significant latency for writes. Based on Spanner white paper, internally it is possible to select preferred datacenter for Paxos leader replica, however I don't think it is done on per user data. I think, a user is hashed to a shard with a preselected datacenter for the leader participant. However, there should be migration process – TohaSt Apr 12 '19 at 19:44
  • @TohaSt: If you want ACID guarantees, you need to accept latency. Most applications at scale solve this problem by using an eventually-consistent model instead, which I am nearly certain is what you really want. If you really feel like you need ACID guarantees, it might be better to explain in the question why you think you need those guarantees. I suspect an eventually consistent model will suit your needs better overall, especially if global latency is an important requirement. – Daniel Pryden Apr 12 '19 at 22:08
  • @TohaSt: Also, I pointed you at the Spanner paper not because I was recommending that you just use the GCP offering, but because the Spanner team solved this problem, at least internal to Google. If you want to build your own solution with the same guarantees, you'll need to solve the same problems they did. (Also, this is exactly what I was referring to in my earlier comment: "This can be done with hard consistency guarantees, even across multiple datacenters, but it's difficult to make it fast enough.") – Daniel Pryden Apr 12 '19 at 22:12

0 Answers0