← Teaching materials← Bahan ajar
Data EngineeringData Engineering
Data engineering course at STMIK Tazkia in two parts: big-data engineering and the cold path (data modelling and columnar formats, large-scale storage and tiering, lakehouse and the medallion architecture, single-machine ETL with Arrow, Polars and DuckDB, orchestration and dbt, storage engines and NoSQL) and the hot path (from Hadoop to Spark at scale, Kafka/NATS/CDC streaming, hot and cold architectures, dashboards and data apps, data security, governance, GRC and observability).Mata kuliah Data Engineering STMIK Tazkia dengan dua capaian: big data engineering dan cold path (pemodelan data dan format columnar, storage skala besar dan tiering, lakehouse dan medallion architecture, ETL di satu mesin dengan Arrow, Polars, dan DuckDB, orchestration dan dbt, storage engine dan NoSQL) serta hot path (dari Hadoop ke Spark skala besar, streaming dengan Kafka/NATS/CDC, arsitektur hot dan cold, dashboard dan data app, data security, governance, GRC, dan observability).
Data Engineering 101Data Engineering 101
- 📄 Slide Pertemuan 1 v04102026
Data modelling and columnar formatsPemodelan data dan format columnar
- 📄 Slide Pertemuan 2 v04102026
Large-scale storage and tieringStorage skala besar dan tiering
- 📄 Slide Pertemuan 3 v04102026
Lakehouse and the medallion architectureLakehouse dan medallion architecture
- 📄 Slide Pertemuan 4 v04102026
ETL on a single machine: Arrow, Polars, DuckDBETL di satu mesin: Arrow, Polars, DuckDB
- 📄 Slide Pertemuan 5 v04102026
Orchestration, dbt, and data qualityOrchestration, dbt, dan data quality
- 📄 Slide Pertemuan 6 v04102026
Storage engines, NoSQL, and servingStorage engine, NoSQL, dan serving
- 📄 Slide Pertemuan 7 v04102026
Midterm - group project: a bronze-silver-gold ETL pipelineUTS - tugas besar: pipeline ETL bronze-silver-gold
- 📄 Brief Tugas Besar UTS v04102026
From Hadoop to SparkDari Hadoop ke Spark
- 📄 Slide Pertemuan 9 v04102026
Spark at scale: HPC for dataSpark skala besar: HPC data
- 📄 Slide Pertemuan 10 v04102026
Kafka, NATS, CDC, and stream processingKafka, NATS, CDC, dan stream processing
- 📄 Slide Pertemuan 11 v04102026
Hot and cold architectures in industryArsitektur hot dan cold di industri
- 📄 Slide Pertemuan 12 v04102026
Data-driven: from metrics to dashboards and data appsData-driven: dari metrik ke dashboard dan data app
- 📄 Slide Pertemuan 13 v04102026
Data securityData security
- 📄 Slide Pertemuan 14 v04102026
Data governance, GRC, and observabilityData governance, GRC, dan observability
- 📄 Slide Pertemuan 15 v04102026
Final - group project: a large-scale Spark pipeline through to a dashboard or appUAS - tugas besar: pipeline data skala besar dengan Spark sampai dashboard atau aplikasi
- 📄 Brief Tugas Besar UAS v04102026