Data Engineering Pipelines
Learn how real data pipelines move and reshape data: ingestion, batch vs streaming, ETL/ELT, schemas, transformation, orchestration, warehouses, and monitoring.
0 of 10 lessons complete
About this course
A practical, intermediate tour of data engineering. You will learn what a data pipeline is and who relies on it, how data gets ingested from files/APIs/databases/events, the difference between batch and streaming, ETL vs ELT, why schemas and data contracts matter, how to write idempotent transformations, how incremental loads and watermarks avoid reprocessing everything, how orchestration tools schedule and retry work with DAGs, how warehouses and lakes model data for analytics, and how to monitor data quality and pipeline health in production.
Syllabus
Pipeline Foundations
What a data pipeline is, who depends on it, how data is ingested, and batch vs streaming.
- What a Data Pipeline Is (and Who Depends on It)Finish the project "Build a Mini Data Pipeline: Ingest, Clean, and Summarize" to unlock this lesson.12m
- Sources and Ingestion: Files, APIs, Databases, EventsFinish the project "Build a Mini Data Pipeline: Ingest, Clean, and Summarize" to unlock this lesson.14m
- Batch vs Streaming ProcessingFinish the project "Build a Mini Data Pipeline: Ingest, Clean, and Summarize" to unlock this lesson.12m
Shaping and Trusting Data
ETL vs ELT, schemas and data contracts, validation, transformation, and idempotency.
- ETL vs ELTUnlimitedFinish the project "Build a Mini Data Pipeline: Ingest, Clean, and Summarize" to unlock this lesson.12m
- Schemas, Data Contracts, and ValidationUnlimitedFinish the project "Build a Mini Data Pipeline: Ingest, Clean, and Summarize" to unlock this lesson.14m
- Transformation and IdempotencyUnlimitedFinish the project "Build a Mini Data Pipeline: Ingest, Clean, and Summarize" to unlock this lesson.14m
Running Pipelines in Production
Incremental loads, orchestration and scheduling, warehouses/lakes, and data quality monitoring.
- Incremental Loads and WatermarksUnlimitedFinish the project "Build a Mini Data Pipeline: Ingest, Clean, and Summarize" to unlock this lesson.14m
- Orchestration, DAGs, Retries, and BackfillsUnlimitedFinish the project "Build a Mini Data Pipeline: Ingest, Clean, and Summarize" to unlock this lesson.15m
- Warehouses, Lakes, and Modelling for AnalyticsUnlimitedFinish the project "Build a Mini Data Pipeline: Ingest, Clean, and Summarize" to unlock this lesson.15m
- Data Quality Monitoring and Pipeline ObservabilityUnlimitedFinish the project "Build a Mini Data Pipeline: Ingest, Clean, and Summarize" to unlock this lesson.14m