Data Engineer in Test

We are undergoing a significant data transformation to ensure that accurate and consistent data is available precisely when and where it's needed.

Responsibilities

Write PySpark and SQL scripts to validate data pipelines, transformations, and integrations.

Design and run tests for data validation, storage, and retrieval using Azure services like Data Lake, Synapse, and Data Factory, adhering to industry standards.

Continuously enhance automated tests as new features are developed, ensuring Participate in data reconciliation and verify Data Quality frameworks to maintain data accuracy, completeness, and consistency across the platform.

Formulate and maintain test strategies—including smoke, performance, functional, and regression testing—to ensure data processing and ETL jobs meet requirements.

Collaborate with development teams to assess changes in data workflows and update test cases to preserve data integrity.

Share knowledge and best practices by collaborating with business analysts and technology teams to document testing processes and findings.

Communicate testing progress effectively with stakeholders, highlighting issues or blockers, and ensuring alignment with business objectives.

Maintain a comprehensive understanding of the Azure Data Lake platform's data landscape to ensure thorough testing coverage.

Requirements

Bachelor’s degree in computer science or related discipline

3-6 years of hands‑on experience in Data Engineering particularly in Data Lake environments on Azure's cloud platform.

Proficient in Azure Data Factory, Azure Synapse Analytics and Databricks for big data processing and scaled data quality checks.

Proficiency in SQL, capable of writing and optimizing both simple and complex queries for data validation and testing purposes.

Proficient in PySpark, with experience in data manipulation and transformation, and a demonstrated ability to write and execute test scripts for data processing and validation.

Hands‑on experience with Functional & system integration testing in big data environments, ensuring seamless data flow and accuracy across multiple systems.

Knowledge and ability to design and execute test cases in a behaviour‑driven development environment.

Fluency in Agile methodologies, with active participation in Scrum ceremonies and a strong understanding of Agile principles.

Familiarity with tools like Jira, including experience with X‑Ray or Jira Zephyr for defect management and test case management.

Proven experience working on high‑traffic and large‑scale software products, ensuring data quality, reliability, and performance under demanding conditions.

Identify and resolve issues in complex Big Data environments, ensuring data accuracy and system efficiency.

Initiate and manage tasks, delivering high‑quality results within deadlines in a dynamic setting.

Confidently challenge existing assumptions and processes to enhance data engineering and QA practices.

Maintain organized testing schedules and documentation, ensuring thorough coverage of all scenarios.

Exhibit strong interpersonal skills, working effectively in cross‑functional, multicultural teams with minimal supervision.

Comfortable in Agile and Scrum environments, proficient with tools like Jira, and adaptable to changing priorities.

Manage multiple projects simultaneously, demonstrating strong planning and organizational skills to ensure timely task completion.

We have an amazing team of 700+ individuals working on highly innovative enterprise projects & products. Our customer base includes Fortune 100 retail and CPG companies, leading store chains, fast‑growth fintech, and multiple Silicon Valley startups.

What makes Confiz stand out is our focus on processes and culture. Confiz is

ISO 9001 : 2015

(QMS),

ISO 27001 : 2022

(ISMS),

ISO 20000-1 : 2018

(ITSM),

ISO 14001 : 2015

(EMS),

ISO 45001 : 2018

(OHSMS) Certified. We have a vibrant culture of learning via collaboration and making workplace fun.

People who work with us work with cutting‑edge technologies while contributing success to the company as well as to themselves.

To know more about Confiz Limited, visit :