Data Engineer Intern

Responsibilities

Built and maintained a Change Data Capture platform on Apache Kafka (KRaft), Debezium, Prometheus, and Grafana to evaluate log-based replication across 15+ Oracle source tables.
Delivered Oracle to Oracle CDC pipelines for 5 tables using Debezium on Kafka Connect, validating replication with no changes to the source application.
Delivered Oracle to Apache Iceberg pipelines for 10 tables holding 40M+ records, combining Debezium connectors with cron-scheduled Apache Spark batch jobs.
Engineered a Spark-based bulk initial load for a 600M row (200 GB) Oracle table where the Debezium snapshot path was not practical at that volume, then transitioned the table onto standard Debezium delta streaming.
Developed custom Python Prometheus exporters for Kafka offset lag, Spark processing lag, and Debezium connector health, and performed Iceberg table optimization to keep performance predictable under sustained load.