Senior Data Engineer – Financial Services

Job Title: Senior Data Engineer – Financial Services (Data Lakehouse & Real-Time Systems)

About the Role

We are seeking seasoned Data Engineering and architecture leaders to architect and deliver a next-generation data lake/house platform for a leading financial services client. This engagement will involve thorough data architecture and engineering components, analysis of complex data and its ecosystem and integrating real-time trade, crypto, securities and fixed income data, analytics, and risk systems for various use-cases. The ideal candidate combines deep domain experience in retail brokerage with hands-on expertise in AWS, MS- Azure Databricks, Spark, and modern data lake/house architectures, and can balance delivery under pressure with architectural foresight and team mentorship. This role requires a deep skill-set in real-time architectural systems and pipelines, adept with financial services specifically, investment domain and a hands-on engineer with a knack of innovative insight on integration of various data architecture/s within the cloud ecosystem.

Key Responsibilities

• Architect and lead the design, implementation, and optimization of large-scale data lake/house systems on Azure Databricks using Spark, Delta Lake, and Python. (advanced)

• Build and optimize streaming pipelines for real-time ingestion, processing, and analytics of trade and market data across asset classes (FX, equities, futures, options, bonds, and crypto). (intermediate)

• Collaborate closely with the CTO and data architects to evolve the end-to-end application architecture for scalability, resilience, and low-latency performance. (advanced)

• Mentor and guide junior and mid-level engineers, establishing best practices in data modelling, orchestration, and operational monitoring.

• Ensure data integrity, lineage, and governance within regulatory and compliance frameworks. (advanced)

• Partner cross-functionally with business and technology teams to align data infrastructure with business KPIs and capital markets product goals.

• Drive automation and CI/CD for data pipelines using Terraform, GitHub Actions, and cloud-native DevOps practices. (advanced)

• Deliver under tight timelines, managing sprint-level execution and ensuring production readiness within a short time-line for GO-LIVE.

Required Skills and Qualifications:

• 15+ years of professional experience, with at least 10 years in financial services and 8+ years in data engineering leadership roles.

• Proven track record building real-time, high-throughput data platforms at enterprise scale.

• Strong proficiency in Azure Databricks, Apache Spark, and Python (PySpark, Panda).

• Solid experience with streaming technologies (Kafka, Kinesis, Event Hubs, or Flink).

• Deep understanding of data lake/house design principles (Cloud Engineering and architecture, AWS, Delta Lake, Iceberg, or Hudi) and data modelling for analytics and ML pipelines.

• Strong background in retail brokerage and trading systems, including exposure to FX, equities, futures, options, bonds, and crypto.

• Experience building or integrating risk management systems, trade surveillance, or crypto exchanges.

• Practical knowledge of data governance, lineage, and metadata management tools (e.g., Azure Purview).

• Proven ability to design scalable, cost-efficient pipelines on Azure using Data Factory, Synapse, and related services.

• Excellent communication, collaboration, and documentation skills, with the ability to influence senior stakeholders.

Preferred Qualifications

• Experience working with microservices-based or event-driven architectures supporting capital markets workloads.

• Familiarity with machine learning pipelines for predictive risk and trading analytics.

• Exposure to Shopify-like multi-tenant or marketplace data architectures.

• Cloud certifications: Microsoft Certified: Azure Data Engineer Associate or equivalent.

• Previous experience in start-up or fast-paced delivery environments with adaptability and velocity.

Core Attributes

• Strong sense of ownership, reliability, and confidentiality in managing critical financial data.

• Ability to balance delivery urgency with architectural integrity.

• A collaborative mentor who uplifts engineering standards across the team.

• Strategic mindset with hands-on execution capability.

Engagement Details

• Number of Positions: 3

• Location: Mumbai

• Engagement Type: Full-Time / Contract-to-Hire

• Timeline: Immediate start; production launch targeted for January

• Reports To: Engagement Lead in conjunction with Client CTO.