A small, solid data warehouse on AWS

Most growing companies do not need a big data platform. They need one place where the numbers agree. When finance, sales and product each pull their own numbers from production databases, every meeting starts with an argument about whose number is right.

This is the simple setup I built for a fintech team, and I would still recommend its shape today.

The setup

• S3 as the landing zone. Every source lands here first, raw and untouched, organised by source and date.
• Airflow runs every step on a schedule, with retries and alerts.
• Snowflake holds the clean, modelled tables that people actually query.
• Power BI and a small React app read from Snowflake, never from production databases.

Three rules that saved us

1. Never change raw data

Keep the original files exactly as they arrived. When a bug shows up three months later, you can rebuild everything from raw. Without this, a bug becomes permanent.

2. Every load can run twice

Loads must be idempotent: running the same day twice gives the same result. Airflow will retry, people will rerun jobs by hand, and you never want duplicate rows in a financial report.

3. One owner per table

Each important table has a named owner who answers questions about it. It sounds bureaucratic. It saves hours every week.

"Real time" usually means hourly

Many teams ask for real time data. When you ask what decision they make with it, hourly is almost always enough. Hourly batch is far cheaper and simpler to run than streaming. Start there and move to streaming only for the cases that truly need it.

What I would add today

The same shape still works. I would add dbt for the transformations, so models are versioned and tested like code, and data quality checks that run on every load. The core ideas stay the same: keep raw data, make loads repeatable, and make someone own each table.