Senior Data Engineer – Databricks, Python & Azure
Don't apply into the void.
Most applications for this Recruiterv role vanish into an ATS. With jobfinder-ai, your agent finds the actual hiring manager or founder behind this opening and sends a tailored email from your own inbox — so a real person reads your pitch and replies. We then follow up until you land on the calendar.
Reach the decision-maker — $5About the role
**Senior Data Engineer – Databricks, Python \& Azure**
**Location:** Chennai **Client:** US\-based IT services and consulting firm **Work Mode:** Work from Office (5days/week) **Experience:** 10\+ Years **Employment Type:** Full\-time
About the Role
We are looking for an experienced **Senior Data Engineer** with deep expertise in **Databricks, Python, Apache Spark, SQL, and Azure** to design and deliver enterprise\-scale Lakehouse solutions. This role requires hands\-on experience in building scalable data platforms, modern ETL/ELT pipelines, and cloud\-native data engineering solutions.
The ideal candidate will have strong architectural experience, excellent stakeholder management skills, and a passion for building high\-performance data platforms that support analytics and AI initiatives.
Key Responsibilities
* Design and implement enterprise\-grade Lakehouse solutions using **Databricks**. * Build scalable batch and real\-time data pipelines using **Apache Spark (PySpark)** and **SQL**. * Develop ETL/ELT pipelines, metadata\-driven ingestion frameworks, and incremental data processing. * Implement and optimize Databricks features including **Delta Lake, Delta Live Tables, Autoloader, Structured Streaming, Unity Catalog, and Workflows**. * Design and maintain enterprise data warehouse solutions using **3NF and dimensional data modeling**. * Establish data governance, security, data quality, and metadata management frameworks. * Lead ML lifecycle management using **MLflow** and support AI initiatives such as **RAG** and **AI/BI**. * Deploy and manage cloud\-native data solutions on **Microsoft Azure**. * Build CI/CD pipelines and automate deployments using **Azure DevOps**. * Optimize Spark workloads for scalability, reliability, and performance. * Integrate enterprise applications such as **ServiceNow** with data platforms. * Collaborate with business stakeholders to translate requirements into scalable technical solutions. * Mentor engineering teams and provide technical leadership throughout project delivery.
Required Skills
* 10\+ years of software engineering experience with at least **5\+ years in Data Engineering**. * Strong hands\-on experience with **Databricks** and end\-to\-end Lakehouse implementation. * Expertise in **Python**, **SQL**, and **Apache Spark (PySpark)**. * Strong experience with **Delta Lake, Delta Live Tables, Autoloader, Structured Streaming, Unity Catalog, and Databricks Workflows**. * Experience designing enterprise\-scale data warehouses using **3NF** and **dimensional modeling**. * Experience working on both batch and real\-time data processing solutions. * Architecture\-level experience owning solution design through deployment. * Good understanding of AI services, machine learning concepts, or Generative AI. * Strong stakeholder management and requirement gathering experience with US or UK clients. * Experience working in an IT Services or IT Consulting environment.
Preferred Skills
* Azure Databricks and Azure Data Services * Azure DevOps and CI/CD implementation * MLflow and MLOps * AI use cases including RAG and AI/BI * Terraform or Infrastructure as Code (IaC) * Databricks Asset Bundles (DABs) * ServiceNow or enterprise application integrations * Databricks, Azure, or AWS certifications
What We're Looking For
* Strong analytical and problem\-solving skills. * Excellent communication and stakeholder management abilities. * Passion for modern data engineering and cloud technologies. * Experience mentoring teams and leading technical initiatives. * Ability to deliver scalable, secure, and high\-performance data solutions.
Benefits
* Free **AWS and Azure certification training**. * Group Medical Insurance coverage of **₹5 Lakhs**. * Meal allowance. * Opportunity to work on enterprise\-scale cloud and AI\-driven data platforms. * Excellent learning and career growth opportunities.
**Note:** Compensation package includes a **5% variable pay** component.
Pay: Up to ₹3,000,000\.00 per year
Benefits:
* Provident Fund
Application Question(s):
* Are you comfortable appearing for a face\-to\-face interview in the office? Final round will be face\-face (mandatory) * Are you available to join immediately or within 15 days in office? * If you are serving NP, when is your last working day?
Experience:
* total: 10 years (Required)
Location:
* Chennai, Tamil Nadu (Chennai District) (Required)
Work Location: In person
Ready to reach the decision-maker?
Set this role as a target and your agent does the sourcing, finds the verified email, writes the pitch, and follows up — on autopilot.
Start your hunt