Data Engineering

Kafka

Development
- Built kafka based events processing pipeline for third party integrations and SDK events
- GDPR compliance implementation on Kafka for PII and sensitive information storage and retrieval
Deployment
- Manual deployment of multi node Kafka cluster on vanilla VMs
- Setup kafka cluster in Azure cloud with HD Insights

Development
- Batch processing pipeline design and implementation for
  - Digestion to AI Subsystem
  - Business Metrics and reporting
  - Full Stack Microservices
- Scalatra Based API microservice for AI systems to process data in expressive SQL format that works on top on spark
- Built multinode spark cluster deployment service with notebook access (github repo here)
Deployment
- Setup Databricks spark cluster on Azure Cloud
- Manual Deployment of spark cluster on vanilla VMs

Design
- Data Schema design of Cassandra data storage for
  - AI subsystem Consumption
  - Transformed and processed data storage
  - Time series data storage
  - Reporting and business metrics calculation
Deployment
- Manual deployment of multi node Cassandra cluster in use for all production systems (Blog post here)

Streaming platform prototyping and design for real time data processing requirement
- Kafka
- Kafka Streams
- Faust
- Spark Streaming

Lead building external data integration with following platforms

Designed and implemented data pipeline for batch data processing and streaming data processing
Tech stack included
- Spark
- Kafka
- Cassandra
- Scala
- Faust
- Kafka streams

Last updated 4 years ago