1. What is Database Sharding?

• Database sharding is a data architecture strategy that increases database performance by splitting up data into chunks and then spreading these chunks intelligently across multiple database servers or database instances.
• These chunks of data are called shards, while each shard contains a subset of our data. All shards represent the entire set of data, and each row of data exists in only one shard.

2. Why do we need database sharding?

• Scalability: Sharding allows you to distribute your data across multiple servers, enabling the system to handle more data and traffic without overwhelming a single server.
• Improved Performance: Reduced load on individual servers by splitting the data into smaller shards, the query load on each server is reduced, improving response times and overall database performance.
• Avoiding Single Point of Failure: In a single instance of database, failure of a database server can lead to a complete system outage. Sharding reduces this risk by spreading the data across multiple independent servers.
• Handling Large Data: Distribute large datasets when databases grow to a size that a single server can no longer efficiently store or manage, sharding divides the data into manageable chunks.
• Cost Efficiency: Instead of investing in high-end, powerful servers for vertical scaling, sharding allows the use of multiple servers, reducing infrastructure costs.

3. Difference between database sharding and partitioning?

• Database Sharding
  • Divides a database into smaller, autonomous units (shards) distributed across multiple servers.
  • Shards are independent, meaning each shard is responsible for its own subset of the data.
  • Ideal for large-scale distributed systems needing high scalability.
  • Fault tolerance is achieved through shard distribution across multiple servers.
• Database Partitioning
  • Partitions can exist on the same server or across servers, but distribution is optional.
  • Partitions are part of the same database instance.
  • Primarily used to improve data organization and performance in a single system.
  • Fault tolerance depends on partition replication or distribution but isn’t inherent.

4. What are the different methods of database sharding?

• Key Based Database Sharding:
  • A unique key (e.g., User ID or Order ID) is selected from the dataset. This key determines which shard the data will be stored in.
  • A hash function is applied to the shard key to map it to a specific shard. This ensures an even distribution of data across all shards.
  • The hashing process distributes data evenly across shards, helping avoid hotspots.
    
• Range Based Database Sharding:
  • A specific key (e.g., Date, User ID) is selected, and data is partitioned into shards based on the range of this key’s values.
  • Each shard holds data within a specific range of values for the sharding key. For example, Shard 1 might store data for IDs 1-1000, Shard 2 for IDs 1001-2000.
  • Since each shard contains data from a well-defined range, the system can easily route queries to the correct shard based on the range of the sharding key in the query.
    
• Directory-based database sharding:
  • Directory-based Sharding, also known as metadata-based Sharding, employs a separate service or metadata store to maintain a mapping of data to shards.
  • Each piece of data contains metadata or attributes that describe which shard it belongs to.
  • Directory-based Sharding offers flexibility in distributing data based on a variety of criteria, including business logic and data attributes.
  • 

5. What are the pros and cons of database sharding?

• Pros
  • The sharding pattern is well suited for large, distributed enterprise applications.
  • Sharding allows for the fast execution of a command or a query.
  • Storage segmentation, which is a key feature of the sharding pattern, enables the physical infrastructure to scale in a controlled manner.
• Cons
  • Sharding requires DBAs to have domain expertise and experience with best practices in relevant database technologies for managing servers.
  • Shards distributed over many geolocations can be susceptible to performance degradation due to excessive network traffic.
  • Some database technologies are better suited to the sharding pattern than others. Thus, you need to choose wisely.
  • Added hardware means a higher total cost of ownership of the service.