Hendri Karisma

← Teaching materials← Bahan ajar

Data EngineeringData Engineering

TI-DE-303 · Ganjil 2026/2027

Data engineering course at STMIK Tazkia in two parts: big-data engineering and the cold path (data modelling and columnar formats, large-scale storage and tiering, lakehouse and the medallion architecture, single-machine ETL with Arrow, Polars and DuckDB, orchestration and dbt, storage engines and NoSQL) and the hot path (from Hadoop to Spark at scale, Kafka/NATS/CDC streaming, hot and cold architectures, dashboards and data apps, data security, governance, GRC and observability).Mata kuliah Data Engineering STMIK Tazkia dengan dua capaian: big data engineering dan cold path (pemodelan data dan format columnar, storage skala besar dan tiering, lakehouse dan medallion architecture, ETL di satu mesin dengan Arrow, Polars, dan DuckDB, orchestration dan dbt, storage engine dan NoSQL) serta hot path (dari Hadoop ke Spark skala besar, streaming dengan Kafka/NATS/CDC, arsitektur hot dan cold, dashboard dan data app, data security, governance, GRC, dan observability).

Materials on this page are open. Assignments and quizzes belong to the class — log in to see them.Materi di halaman ini terbuka. Tugas dan kuis milik kelas — masuk dulu untuk melihatnya.
1

Data Engineering 101Data Engineering 101

2

Data modelling and columnar formatsPemodelan data dan format columnar

3

Large-scale storage and tieringStorage skala besar dan tiering

4

Lakehouse and the medallion architectureLakehouse dan medallion architecture

5

ETL on a single machine: Arrow, Polars, DuckDBETL di satu mesin: Arrow, Polars, DuckDB

6

Orchestration, dbt, and data qualityOrchestration, dbt, dan data quality

7

Storage engines, NoSQL, and servingStorage engine, NoSQL, dan serving

UTS

Midterm - group project: a bronze-silver-gold ETL pipelineUTS - tugas besar: pipeline ETL bronze-silver-gold

9

From Hadoop to SparkDari Hadoop ke Spark

10

Spark at scale: HPC for dataSpark skala besar: HPC data

11

Kafka, NATS, CDC, and stream processingKafka, NATS, CDC, dan stream processing

12

Hot and cold architectures in industryArsitektur hot dan cold di industri

13

Data-driven: from metrics to dashboards and data appsData-driven: dari metrik ke dashboard dan data app

14

Data securityData security

15

Data governance, GRC, and observabilityData governance, GRC, dan observability

UAS

Final - group project: a large-scale Spark pipeline through to a dashboard or appUAS - tugas besar: pipeline data skala besar dengan Spark sampai dashboard atau aplikasi