Distributed Systems Design and Development
Systems that keep running when a machine dies
A distributed system is one that runs on many machines at once and keeps working when some of them fail. That sounds simple and is not. Clocks disagree, networks partition, and "exactly once" turns out to be a lie.
We have built these systems for telecoms and for our own USP controller, on Cassandra, Kafka, Hazelcast and Kubernetes. We know where the sharp edges are.
Contact usWhat we pay attention to
Failure is normal
We design assuming any node, disk or network link will fail, and test by making them fail. The system should degrade a little, not stop.
Throughput comes from spreading work
Partition the data well and a hundred nodes process a hundred times the events. Partition it badly and one hot node does all the work. Most of the design effort goes into the keys.
Eventual consistency, where it is acceptable
Many things do not need to be consistent this millisecond: a device's last-seen time, a usage counter. Accepting that (the BASE model) buys availability. We are careful about where it is acceptable and where it is not.
CAP is a budget, not a slogan
When the network splits, each part of the system has to choose between staying available and staying consistent. We make that choice explicitly, per component, and write it down so operations knows what to expect.
Messaging between services
If two services need to agree on something, a queue between them is usually cleaner than a synchronous call. We have run Kafka, ActiveMQ Artemis and RabbitMQ in production and know what each is good and bad at.
Testing the failures, not just the features
Kill a Cassandra node mid-write. Partition the Kafka cluster. Restart half the services during a load test. If it has not been done in staging, it will be done in production by accident.
Operations is part of the design
Metrics, alerts, runbooks and rolling upgrades are designed in, not added later. We help deploy, we stay on call for the first weeks, and we can keep maintaining it if you want.
Why go distributed at all
You add machines, not rewrites
When the load grows, the answer is more nodes. There is no biggest-server-we-can-buy ceiling.
One failure is not an outage
Other nodes pick up the work. The on-call engineer replaces the failed one in the morning.
Problems stay local
A crashed process, a bad disk or a flaky link affects its own corner of the system and nothing else.
Work happens in parallel
A hundred nodes each doing a hundredth of the job finish long before one large machine doing all of it.
A word of caution
Not everything should be distributed. A well-tuned single database is faster and simpler than a cluster for a surprising number of workloads, and we will say so if that is your situation. When you do need a distributed system, we have built enough of them to build yours without the usual first-year surprises.
Write to us about what you are building. If you already have a system and want a second opinion on it, see our distributed systems consultancy.