Azure Databricks & Generative AI Engineer
Experience
3–6 Years
Job Type
Full-Time
Role Overview
We are looking for an experienced Azure Databricks & Generative AI Engineer to build scalable data and AI solutions using Azure Databricks, Python, and Generative AI technologies. The ideal candidate should have strong expertise in data engineering, Databricks, Azure cloud services, and hands-on experience developing or integrating Gen AI/LLM-based solutions.
Key Responsibilities
- Design, develop, and maintain scalable data solutions using Azure Databricks.
- Develop data processing and transformation pipelines using PySpark and Python.
- Work with Delta Lake, Spark SQL, and Databricks Workflows.
- Build and integrate Generative AI and LLM-based applications.
- Develop AI-powered solutions using Azure OpenAI and Azure AI services.
- Implement RAG (Retrieval-Augmented Generation) solutions using enterprise data.
- Work with embeddings, vector databases, and AI Search for intelligent information retrieval.
- Integrate structured and unstructured data into Gen AI applications.
- Develop data pipelines to support AI/ML and analytics use cases.
- Optimize Databricks jobs, Spark workloads, queries, and data processing performance.
- Implement data quality, security, governance, and access controls.
- Collaborate with data scientists, software engineers, business teams, and cloud architects.
- Monitor and troubleshoot data pipelines and AI workloads.
- Follow best practices for cloud architecture, CI/CD, version control, and deployment.
Required Technical Skills
- Strong hands-on experience with Azure Databricks.
- Strong knowledge of PySpark, Python, and SQL.
- Experience with Delta Lake and Databricks Workflows.
- Good understanding of Azure cloud services.
- Hands-on knowledge of Generative AI and Large Language Models (LLMs).
- Experience with Azure OpenAI / Azure AI services.
- Understanding of RAG architecture, embeddings, prompt engineering, and vector search.
- Experience working with structured and unstructured data.
- Knowledge of data engineering and ETL/ELT concepts.
- Experience with Git and CI/CD practices.
Preferred Skills
- Experience with Azure AI Search and vector databases.
- Knowledge of MLflow and MLOps.
- Experience with Databricks Unity Catalog.
- Knowledge of Azure Data Lake Storage (ADLS Gen2).
- Experience with REST APIs and AI application integration.
- Familiarity with LangChain, LlamaIndex, or similar Gen AI frameworks.
- Experience with Docker and cloud-based deployment.
- Understanding of Responsible AI and enterprise AI security.
Education
Bachelor’s or Master’s degree in Computer Science, Information Technology, Engineering, Data Science, or a related field.
Key Competencies
- Strong analytical and problem-solving skills.
- Good understanding of data and AI architectures.
- Ability to work with cross-functional teams.
- Strong communication and documentation skills.
- Ability to troubleshoot complex data and AI issues.
- Willingness to learn and adopt emerging Gen AI and Azure technologies.
Work Location: Hybrid remote in Noida, Uttar Pradesh (Noida)