Data Engineer

Location : Remote

Experience Level : 5 to8 years in Data Engineering, Data Pipelines, and Cloud-based Data Platforms

Department : Data & AI Engineering

Compensation : PKR 600,000 to 850,000 (based on experience)

Role Summary

The Data Engineer will design and build large-scale, high-performance data pipelines to support segmentation, pricing simulation, and offer decisioning. They will ensure efficient data ingestion from telco systems (CDRs, usage, recharge, offer purchase), transformation, and integration with ML models and orchestration modules.

Key Responsibilities

• Design and develop scalable ETL / ELT data pipelines to process 50M+ customer records daily.
• Ingest data from OCS, CRM, DWH, and Adobe RT-CDP or other customer data platforms.
• Build and maintain Customer Profile Store and Feature Store for real-time and batch processing.
• Implement data validation, quality, and lineage frameworks.
• Optimize query performance and cost efficiency for batch and streaming workloads.
• Collaborate with Data Scientists to prepare model training datasets and deploy inference pipelines.
• Integrate outputs with Decision Engine and Real-Time Offer Orchestration Module.
• Automate pipelines using CI / CD and maintain environment configurations across Dev, UAT, and Prod.

Required Skills

Tools & Technologies

Data Platform : Databricks, AWS Glue, Azure Data Factory, Snowflake, BigQuery

Streaming : Kafka, Kinesis, Spark Streaming

Storage : S3, Delta Lake, Parquet, Hive

Workflow Orchestration : Airflow, dbt, Dagster, Prefect

Scripting : Python, SQL, PySpark

DevOps : Git, Jenkins, Terraform

Monitoring & Validation : Great Expectations, Deequ, DataDog

Preferred (Nice-to-Have)

#J-18808-Ljbffr