Disclaimer: I have found it a lil tough to translate my mental model of this stuff into words for juniors. I can explain it to them, but I am making up stuff on the fly when I do that. So I am writing it down properly, once.
you are a junior. you gotta store some data. for your application ofcourse.
Where though? Before you pick a tool, you need to know your data. Then the tool picks itself.
ask these questions first
- What shape is the data?
- Structured (spreadsheets, tables)?
- Semi-structured (JSON, logs)?
- Unstructured (images, videos)?
- How is it accessed?
- Frequent small reads/writes (user profiles)?
- Rarely updated, but heavily queried (analytics)?
- Bulk storage with no direct querying (backups)?
- How much data is there? Gigabytes? Terabytes? Petabytes?
- What’s the performance need?
- Milliseconds for queries (user-facing apps)?
- Hours for batch processing (reports)?
- Who or what uses it?
- Humans (via dashboards)?
- Machines (APIs, apps)?
the usual suspects
Storage systems are tools for specific jobs. These are the ones you will keep running into:
| Archetype | Best for | Example tools | Analogy |
|---|---|---|---|
| Relational database | Structured data needing ACID* transactions, complex queries | PostgreSQL, MySQL, RDS | A filing cabinet with labeled folders. Strict rules, perfect for precise lookups. |
| Object store | Unstructured/semi-structured data, cheap bulk storage | S3, GCS, Azure Blob | A warehouse for boxes. Dump anything, retrieve by barcode (no search). |
| Data warehouse | Analytics on large structured/semi-structured datasets | Redshift, Snowflake, BigQuery | A research library. Built for big questions (“Total sales last year?”). |
| NoSQL database | Semi-structured data, flexible schemas, scale horizontally | MongoDB, DynamoDB | A flexible shelf. Store varied items (books, tools, toys), search by tags. |
| File system | Hierarchical data (directories of files) | NFS, EFS, HDFS | A physical office desk. Folders inside drawers, familiar but limited scalability. |
*ACID = Atomicity, Consistency, Isolation, Durability. Critical for transactions like banking.
deciding
- Start with the data shape:
- Structured → relational DB or warehouse
- Unstructured → object store
- Semi-structured → NoSQL or warehouse
- Then the access pattern:
- Need to fetch one row fast? → relational DB
- Need to scan millions of rows? → warehouse
- Just storing files? → object store
- Then scale:
- Small datasets → almost anything works
- Massive datasets → object store, warehouse, or NoSQL
stuff to keep in mind
- Hot vs cold data. Hot (frequently accessed) goes on fast storage (databases). Cold (rarely accessed) goes on cheap storage (object store).
- Cost vs performance. Faster and structured costs more. Slower and unstructured is cheaper.
- Flexibility. Object stores are dumb but scale forever. Databases and warehouses are smart, but the schema constrains you.
Start with the problem, not the tool.
never a hammer finding a nail to hammer please