Introduction to Non-Relational and Large-Scale Data Storage
Welcome to the world of Big Data! In the previous chapters of A.3 Databases, we looked at relational databases which use neat tables, rows, and columns. But what happens when the data is too messy, too fast, or simply too massive for a traditional table to handle? That is where non-relational and large-scale storage systems come in. In this chapter, we will explore why these systems exist, how they handle "Big Data," and how they differ from the relational databases you have already studied.
Note: For a refresher on traditional table-based systems, see the chapter "Relational database design and modelling."
1. What is "Big Data"?
Before we look at the storage tools, we need to understand the problem they solve. We often use the "Three V's" to describe Big Data:
- Volume: The sheer amount of data. We aren't just talking about thousands of records, but petabytes of data (that's millions of gigabytes!).
- Velocity: The speed at which data is created and moves. Think of Twitter/X or financial stock markets where thousands of pieces of data arrive every second.
- Variety: Data comes in many forms. It isn't just numbers and names; it's also videos, social media posts, GPS signals, and sensor readings.
Quick Review: If a database has to store 10,000 high-definition videos every minute, it is facing a challenge of both Volume and Velocity.
2. Non-Relational (NoSQL) Databases
The term NoSQL stands for "Not Only SQL." These databases do not use the strict table-and-row structure of relational databases. Instead, they are schema-less, meaning they are very flexible about how data is formatted.
Why use NoSQL?
Imagine you are collecting data from different types of smart home devices. A smart lightbulb might only send "Status: On," but a smart fridge might send a list of "Items," "Temperature," and "Filter Life." In a traditional relational database, you would have to pre-define every possible column. In a NoSQL database, you can just save whatever data each device sends without worrying about matching a rigid structure.
Common Types of NoSQL Databases
- Document Databases: These store data in "documents" (often using a format called JSON). Each document can have different fields. Analogy: A folder full of different Word documents rather than a strict Excel spreadsheet.
- Key-Value Stores: The simplest type. Every item has a unique "key" (like an ID) and a "value" (the data). Analogy: A coat check where you give a ticket (key) to get your specific bag (value).
- Graph Databases: These focus on the relationships between data points. They are used for social networks or recommendation engines (like "People you may know").
- Column-family Stores: These store data in columns rather than rows, which makes them incredibly fast for analyzing specific types of data across billions of records.
Key Takeaway: Non-relational databases offer flexibility and speed for data that doesn't fit neatly into tables.
3. Scaling: Vertical vs. Horizontal
When a database gets too big for its current computer, we need to "scale" it. This is one of the biggest differences between relational and non-relational systems.
Vertical Scaling (Scaling Up)
This means making your existing server more powerful. You add more RAM, a faster CPU, or more storage space.
Pros: Easy to manage.
Cons: There is a limit to how powerful a single computer can get, and it becomes very expensive.
Horizontal Scaling (Scaling Out)
This means adding more computers (nodes) to a network to share the load. This is how large-scale data storage works.
Pros: You can keep adding cheap computers forever. If one fails, the others keep working.
Cons: It is much more complex to coordinate data across many different machines.
Did you know? Relational databases are usually better at Vertical Scaling, while NoSQL databases are designed specifically for Horizontal Scaling.
4. Comparing Relational and Non-Relational
Don't worry if you aren't sure which one is "better"—the truth is, they are used for different jobs!
| Feature | Relational (SQL) | Non-Relational (NoSQL) |
|---|---|---|
| Data Structure | Structured (Tables/Rows) | Unstructured or Semi-structured |
| Schema | Fixed (must be defined first) | Dynamic (flexible) |
| Scaling | Vertical (Bigger server) | Horizontal (More servers) |
| Best Use Case | Financial systems, complex queries | Big Data, Real-time apps, Social media |
Common Mistake: Students often think NoSQL is meant to replace SQL. In reality, most large companies (like Amazon or Netflix) use both—SQL for things like billing/orders and NoSQL for things like product recommendations and user activity logs.
5. Large-Scale Storage Challenges
Managing data across hundreds of servers introduces new problems. One of the biggest concepts is Data Consistency.
In a small relational database, when you update a piece of data, it is updated instantly for everyone. In a large-scale system spread across the world, it might take a few milliseconds for a change made in a server in New York to reach a server in Singapore. This is sometimes called Eventual Consistency—the idea that the data will be the same everywhere eventually, but maybe not at this exact microsecond.
Example: If you "Like" a photo on Instagram, your friend might not see the updated "Like count" for a split second because the data is still traveling across the large-scale network. This is acceptable for social media, but would be dangerous for a bank!
Summary Checklist
- Can you define the 3 V's of Big Data? (Volume, Velocity, Variety)
- Do you know the difference between Vertical Scaling (bigger computer) and Horizontal Scaling (more computers)?
- Can you explain why a NoSQL database is "schema-less"?
- Do you understand that Relational databases are best for structured data, while Non-Relational are best for large-scale, flexible data?
Don't worry if the technical details of horizontal scaling seem complex! For the IB DP, focus on the concepts: why we need these systems (Big Data) and how they differ from the traditional tables we use in SQL.