javascript · intermediate
Data Engineering Pipelines
Learn how real data pipelines move and reshape data: ingestion, batch vs streaming, ETL/ELT, schemas, transformation, orchestration, warehouses, and monitoring.
A practical, intermediate tour of data engineering. You will learn what a data pipeline is and who relies on it, how data gets ingested from files/APIs/databases/events, the difference between batch and streaming, ETL vs ELT, why schemas and data contracts matter, how to write idempotent transformations, how incremental loads and watermarks avoid reprocessing everything, how orchestration tools schedule and retry work with DAGs, how warehouses and lakes model data for analytics, and how to monitor data quality and pipeline health in production.
Pipeline Foundations
What a data pipeline is, who depends on it, how data is ingested, and batch vs streaming.
- What a Data Pipeline Is (and Who Depends on It)Finish the project "Build a Mini Data Pipeline: Ingest, Clean, and Summarize" to unlock this lesson.12m
- Sources and Ingestion: Files, APIs, Databases, EventsFinish the project "Build a Mini Data Pipeline: Ingest, Clean, and Summarize" to unlock this lesson.14m
- Batch vs Streaming ProcessingFinish the project "Build a Mini Data Pipeline: Ingest, Clean, and Summarize" to unlock this lesson.12m
Shaping and Trusting Data
ETL vs ELT, schemas and data contracts, validation, transformation, and idempotency.
- ETL vs ELTFinish the project "Build a Mini Data Pipeline: Ingest, Clean, and Summarize" to unlock this lesson.12m
- Schemas, Data Contracts, and ValidationFinish the project "Build a Mini Data Pipeline: Ingest, Clean, and Summarize" to unlock this lesson.14m
- Transformation and IdempotencyFinish the project "Build a Mini Data Pipeline: Ingest, Clean, and Summarize" to unlock this lesson.14m
Running Pipelines in Production
Incremental loads, orchestration and scheduling, warehouses/lakes, and data quality monitoring.
- Incremental Loads and WatermarksFinish the project "Build a Mini Data Pipeline: Ingest, Clean, and Summarize" to unlock this lesson.14m
- Orchestration, DAGs, Retries, and BackfillsFinish the project "Build a Mini Data Pipeline: Ingest, Clean, and Summarize" to unlock this lesson.15m
- Warehouses, Lakes, and Modelling for AnalyticsFinish the project "Build a Mini Data Pipeline: Ingest, Clean, and Summarize" to unlock this lesson.15m
- Data Quality Monitoring and Pipeline ObservabilityFinish the project "Build a Mini Data Pipeline: Ingest, Clean, and Summarize" to unlock this lesson.14m