Job Title

Infrastructure Data Engineer

Job Description Summary

Job Description

We are seeking a highly skilled Azure Cloud Engineer with deep expertise in designing, implementing, and optimizing Azure Databricks environments. This role will also support a broad range of Azure services—including Web Applications, Azure Data Factory, Application Gateway, Storage Accounts, Lakebase, Azure SQL/Databases, and Azure Monitor—to ensure our cloud ecosystem is secure, scalable, cost-efficient, and aligned with the Five Pillars of the Azure Well-Architected Framework: Operational Excellence, Security, Reliability, Performance Efficiency, and Cost Optimization.

The ideal candidate combines strong hands-on engineering skills with a strategic mindset for cloud governance, automation, and continuous improvement.

Key Responsibilities

Primary Focus: Azure Databricks Engineering

·Design, deploy, and optimize Azure Databricks workspaces, clusters, jobs, and pipelines.Implement best practices for cluster sizing, autoscaling, job orchestration, and cost governance.Collaborate with data engineering teams to build scalable ETL/ELT pipelines using Databricks notebooks, Delta Lake, and Spark.Integrate Databricks with Azure Data Lake, Azure SQL, Azure Data Factory, and other platform services.Implement security controls including Unity Catalog, RBAC, service principals, secret scopes, and network isolation.Monitor Databricks performance and reliability using Azure Monitor, Log Analytics, and Databricks native tools.

Azure Platform Engineering Responsibilities

Azure Application & Integration Services

Deploy, configure, and maintain Azure Web Apps, Function Apps, and API Management.Build and maintain Azure Data Factory pipelines for data ingestion, transformation, and orchestration.Configure and optimize Azure Application Gateway, WAF policies, routing rules, and SSL/TLS certificates.

Azure Storage & Database Services

Manage Azure Storage Accounts, lifecycle policies, replication, encryption, and access controls.Support Azure SQL, Cosmos DB, and other database services including performance tuning and security hardening.

Monitoring, Governance & Optimization

Use Azure Monitor, Log Analytics, Application Insights, and Azure Advisor to proactively identify and resolve issues.Implement alerting, dashboards, and automated remediation workflows.Continuously optimize cloud resources for cost, performance, and reliability.Contribute to cloud governance standards including tagging, resource organization, RBAC, and policy enforcement.

Automation & Infrastructure as Code

Build and maintain IaC templates using Terraform, Bicep, or ARM.Automate deployments, configuration, and operational tasks using Azure DevOps, GitHub Actions, or PowerShell.Support CI/CD pipelines for application and data workloads.

Responsibilities Aligned to the 5 Pillars of the Cloud Operating Model

Operational Excellence

Develop repeatable deployment patterns and IaC modules.Implement monitoring, logging, and alerting standards across all Azure services.Document architectures, runbooks, and operational procedures.

Security

Enforce Zero Trust principles across identity, network, and data layers.Implement RBAC, managed identities, Key Vault, private endpoints, and network security groups.Support compliance initiatives and security assessments.

Reliability

Design resilient architectures using availability zones, redundancy, and failover strategies.Implement backup, disaster recovery, and business continuity plans.Monitor system health and proactively address reliability risks.

Performance Efficiency

Optimize Databricks clusters, SQL databases, and application services for throughput and latency.Conduct performance testing and capacity planning.Use Azure Monitor and Advisor insights to drive continuous improvement.

Cost Optimization

Implement cost-efficient Databricks cluster policies and autoscaling.Monitor and optimize Azure consumption across compute, storage, and networking.Provide recommendations for reserved instances, rightsizing, and lifecycle management.
 

Required Qualifications

3–7+ years of hands-on experience with Microsoft Azure.Strong expertise in Azure Databricks, Spark, Delta Lake, and data engineering concepts.Experience with Azure services such as Web Apps, Data Factory, Application Gateway, Storage Accounts, and Azure SQL.Proficiency with Terraform, Bicep, ARM, or similar IaC tools.Strong scripting skills (PowerShell, Python, Bash).Experience with Azure Monitor, Log Analytics, and Azure Advisor.Solid understanding of cloud networking (VNETs, private endpoints, firewalls, routing).Familiarity with the Azure Well-Architected Framework.

Preferred Qualifications

Azure certifications (AZ-104, AZ-204, DP-203, AZ-305, or Databricks certifications).Experience with CI/CD pipelines in Azure DevOps or GitHub Actions.Experience with Unity Catalog, Databricks SQL, or MLflow.Background in data engineering, DevOps, or cloud architecture.

Soft Skills

Strong communication and documentation abilities.Ability to collaborate across data, application, and infrastructure teams.Problem-solving mindset with a focus on automation and continuous improvement.Comfortable working in fast-paced, cloud-first environments

 

 

 

INCO: “Cushman & Wakefield”