Database Sharding
How Sharding Splits a Database
Database sharding is a way to scale a database by splitting its rows across several separate servers, each holding one portion of the data. It is a form of horizontal partitioning, because whole rows are divided into groups rather than columns being split. Each group is called a shard, and each shard is a complete database with its own storage and processing.
A shard key decides which shard holds a given row. A common example is a customer ID: customers in one range are stored on one server and the rest on another. The application, or a routing layer in front of the databases, uses the key to send each query to the right shard.
Partitioning Versus Sharding
Partitioning and sharding are related but different. Partitioning usually means dividing a table into parts inside one database server, for example by month, which can make maintenance and some queries easier. Sharding goes a step further by placing those parts on different machines, which adds capacity for storage and writes.
Sharding also differs from replication. Replication copies all of the data to several servers, while sharding gives each server a different portion. Replication mainly helps with read load and availability; sharding helps with data size and write volume. Document databases such as MongoDB include built-in support for sharding.
Trade-Offs and Costs of Sharding
Sharding adds complexity. A query that needs data from several shards must be sent to each one and the results combined, and joins across shards are slow or awkward to write. Transactions that span shards are harder to keep consistent, so many systems avoid them. A poor shard key can create hot spots, where one shard receives most of the traffic while others sit idle. Moving data later to rebalance shards is a significant operation.
For these reasons, sharding is usually a late step. Optimizing queries and indexes, using a larger server, sending read traffic to replicas and adding caching often solve the problem first.
A Multi-Tenant Application
A software company serves thousands of business customers from one application. Each customer’s data is kept together under a shard key such as the customer ID. A very large customer can be moved to a dedicated shard, while smaller customers share another. Most queries touch only one customer’s data, so they stay on a single shard, which is what makes this design work.
Frequently asked questions
When does a database need sharding?
Usually when a single server cannot hold the data or keep up with writes, even after optimization and a larger server. Sharding is rarely needed for a typical website or small application. Read load is often easier to scale with replicas, so sharding is reserved for very large data sets or heavy write loads.
What makes a good shard key?
A good shard key spreads rows and requests evenly across shards and keeps related data together, so most queries touch a single shard. Customer or tenant IDs are common choices. A steadily increasing value, such as a timestamp, can send all new writes to one shard, creating a hot spot.