Lead Confluent Kafka Engineer / Architect

Job Title : Lead Confluent Kafka Engineer / Architect Location : Remote (Pak) / Hybrid (Riyadh, KSA) Employment Type : Full-Time / Contract-to-Hire Experience Level : 10+ years (Senior / Lead) Role Summary As a Lead Confluent Kafka Engineer , you'll architect, design, setup, install, implement and optimize high-throughput data streaming solutions using Confluent Platform and Apache Kafka.

You'll lead a team of engineers in delivering production-grade pipelines, ensuring scalability, reliability, and security.

This role involves hands-on development, mentoring, and collaborating with data architects, DevOps, and stakeholders to implement event-driven architectures.

You'll champion best practices in real-time data processing, from proof-of-concepts to enterprise deployments, including full lifecycle management from installation to optimization.

Key Responsibilities Architecture & Design : Lead the design of scalable Kafka clusters and Confluent-based ecosystems (e.g., Kafka Streams, ksqlDB, Schema Registry, Connect) for on-prem, hybrid, and multi-cloud (GCP) environments.

Implementation & Development : Build and maintain real-time data pipelines, integrations, and microservices using Kafka producers / consumers; integrate with tools like Flink, Spark, or ML frameworks for advanced analytics.

Installation & Setup : Oversee the end-to-end installation and initial configuration of Confluent Platform and Apache Kafka clusters, including : Deploying Confluent Enterprise / Community editions on Kubernetes (via Helm / Operator), bare-metal servers, or managed cloud services (e.g., Confluent Cloud, GCP).

Configuring brokers, ZooKeeper / KRaft mode, topics, partitions, replication factors, and security settings (e.g., SSL / TLS, SASL, ACLs) using Ansible, Terraform, or Confluent CLI.

Setting up auxiliary components like Schema Registry, Kafka Connect clusters, and monitoring agents (e.g., JMX exporters) with automated scripts for reproducible environments.

Performing initial health checks, load testing (e.g., with Kafka's performance tools), and integration with existing infrastructure (e.g., VPC peering, load balancers).

Operations & Maintenance : Oversee monitoring, troubleshooting, performance tuning, and lifecycle management (upgrades, patching) of Kafka / Confluent instances; implement DevSecOps practices for CI / CD pipelines.

Team Leadership : Mentor junior engineers, conduct code reviews, and drive technical proofs-of-concept (POCs); gather requirements and define standards for Kafka as a managed service (e.g., access controls, documentation).

Optimization & Innovation : Ensure high availability (>

99.99%), fault tolerance, and cost-efficiency; explore emerging features like Kafka Tiered Storage or Confluent Cloud integrations for AI workloads.

Collaboration & Delivery : Partner with cross-functional teams (data engineers, architects, product owners) to align streaming solutions with business goals; provide thought leadership on event-driven patterns.

Security & Compliance : Implement RBAC, encryption, and auditing; conduct root-cause analysis for incidents and ensure GDPR / HIPAA compliance in data flows.

Required Qualifications & Skills Bachelor's / Master's in Computer Science, Engineering, or related; certifications like Confluent Developer / Administrator a plus.

10+ years in software engineering; 5+ years hands-on with Apache Kafka & Confluent Platform (Cloud / Enterprise editions).

Proficiency in Java / Scala / Python (8 / 11+); Kafka Streams / Connect / ksqlDB; Schema Registry; REST / gRPC APIs.

Event-driven / microservices design; data pipeline optimization; handling high-volume streams (TB / day scale).

Expertise in containerization (Docker / Kubernetes); CI / CD (Jenkins / GitHub Actions); Terraform / Ansible for IaC.

Multi-cloud experience (AWS, GCP, Azure); monitoring tools (Prometheus, Grafana, Confluent Control Center).

Experience with streaming integrations (e.g., Flink, Spark Streaming for CDC).

Contributions to open-source Kafka projects or publications on streaming architectures.

Knowledge of AI / ML data pipelines (e.g., Kafka + TensorFlow / PyTorch).

Familiarity with observability tools and security (OAuth, Kerberos).

Strong problem-solving, communication, and leadership; experience leading POCs and cross-team projects.

Agile / Scrum leadership in fast-paced environments.

Experience in client facing roles and leading teams.

Powered by JazzHR