SOSP paper on DynamoDB mentions :
“Data is distributed across multiple servers using partitioning, and each partition is further replicated to provide availability. The technique for distributing (aka partitioning) is consistent hashing”.
If you see MongoDB’s online manual:
“Sharding is a method to distribute data across multiple different servers. We achieve horizontal scalability through sharding”.
I really don’t see any difference in the implied meaning of these two statements from the literature on two different databases. Both are trying to distribute data to multiple machines and building a scale out database architecture.
Both these databases (and many other NoSQL databases) distribute the data across multiple different machines in a clustered deployment. It is pointless to talk about the strategy used for distributing.
The basic thing to understand is this. Let’s use the word partition for now.
I have a set of data items (records). Each record has a key. I can use this key topartition (distribute) these records into multiple different units. If we talk about Oracle RDBMS, table partitioning has been there for quite some time. Using the partition key, records of a table are sub-divided into 2 or more partitions. These partitions are still in the control of same DB Instance : share the same CPU, memory, I/O, storage resources with other peer partitions and also other non-partitioned tables.
When a query comes, we first determine which partition does the query have its data on. The corresponding partition’s data is then processed to return the results of query. There is no need to touch other partitions.
Oracle supports Hash, Range, and List based partitioning. The purpose of these and many other distribution schemes is simple: given a key for a record, determine the destination partition that it will belong to.
Now let’s talk about sharding. Note that concept of partitioning explained in the context of Oracle had all the partitions under the supervision of same DB instance (hence same physical machine).
It turns out that partitioning across different physical machines/nodes is termed as sharding. Now each partition sits on an entirely different physical machine, and thus in a different schema and under the control of a separate database instance. This is what is done in MongoDB. The approaches to distribute data across multiple machines are hash and range.
Similarly this is what is also done in DynamoDB and Cassandra where the distribution technique is Consistent hashing.
This difference between sharding and partitioning is acceptable -
“sharding is distribution or partition of data across multiple different machines whereas partitioning is distribution of data on the same machine”.
I would personally want to go with this difference although nothing really prevents from saying “sharding is partitioning across different machines”.
Both have to work with a distribution key. Now we can call it as “shard key” or “partition key”. It does not really matter. In fact MongoDB’s documentation uses both “partitioning” and “sharding” terms. DynamoDB and Cassandra only use “partitioning” term.
In Oracle RAC, we can partition the data across multiple instances. A table T when stored in a RAC environment can be hash partitioned (using the hash value of partition key) across the RAC nodes. The hash value will determine which node the key (and its record) will go to. Going by the definition of sharding, what we can do in RAC is kind of related to sharding but we call it as partitioning. Again an overlap.
There is one key difference. RAC is _not_ a shared nothing architecture. It is _not_ a shared everything architecture as well. It is shared disk architecture, and that is why I think sharding is not the right term to use here although the data is still getting distributed across different physical nodes.
This is the reason why sharding is related to shared nothing architecture where we distribute/partition data across multiple different nodes, and nodes do not share any type of resources. This is the key difference to understand.