Overview As a Data Engineering Lead, you will own the technical direction and delivery of modern, cloud-native data platforms across AWS and Azure, leveraging AWS native services, Databricks and Microsoft Fabric. You will lead a team of data engineers, drive architectural decisions, and serve as the primary technical point of contact on client engagements. You will set the standard for engineering excellence across the full data lifecycle, ingestion, transformation, modelling, and serving, enabling real-time analytics, reporting, and AI use cases at scale. Key Responsibilities Leadership and Client Engagement Lead and manage a team of data engineers, providing mentorship, technical guidance, and career development support. Own client relationships from a technical perspective, scoping engagements, setting expectations, and ensuring delivery excellence. Translate business requirements into data platform strategies, aligning technical solutions to measurable outcomes. Drive estimation, planning, and resourcing across concurrent projects. Establish and enforce engineering standards, code review practices, and delivery processes within the team. Collaborate with senior stakeholders and client leadership to communicate progress, risks, and architectural trade-offs. Contribute to pre-sales activities including proposals, solution design, and technical demonstrations. Architecture and Platform Design Define and own data platform architectures across multi-cloud environments (AWS, Azure). Make informed architectural trade-offs across cost, performance, scalability, and governance. Design reference architectures and reusable patterns that accelerate delivery across engagements. Evaluate and recommend tooling and technology choices (Databricks, Microsoft Fabric, AWS native services) based on client context and constraints. Software Engineering Foundations Strong grounding in software engineering fundamentals (data structures, algorithms, design patterns). Proficiency in Python and SQL (additional languages advantageous). Experience with Git, CI/CD pipelines, and modern development practices. Familiarity with Terraform or Bicep for Infrastructure as Code. Comfortable working in Linux-based environments. Data Ingestion and Streaming Architect and build scalable ingestion pipelines across hybrid and cloud environments. Real-time streaming: AWS Kinesis / MSK (Kafka), Azure Event Hubs / Kafka. Batch ingestion: AWS DataSync, DMS, Azure Data Factory / Synapse Pipelines / Fabric Pipelines. Integration via APIs, JDBC/ODBC, and CDC pipelines. Storage, Lakehouse and Fabric Design and manage data lakes using Amazon S3, Azure Data Lake Storage Gen2 (ADLS). Implement lakehouse architectures using Databricks (Delta Lake, Unity Catalog), Microsoft Fabric (OneLake, Lakehouse, Warehouse). Work with modern data formats: Parquet, Avro, JSON. Experience with relational databases (Postgres, SQL Server, Aurora), NoSQL (DynamoDB, Cosmos DB), and caching (Redis). Data Processing and Transformation Build and oversee scalable ETL/ELT pipelines using Databricks (PySpark, Delta Live Tables, Workflows), AWS Glue / Lambda, Azure Databricks / Synapse Spark / Fabric Data Engineering. Implement and champion medallion architecture (Bronze/Silver/Gold). Ensure pipelines are reusable, testable, and production-grade. Analytics and AI Enablement Design platforms that support business intelligence, advanced analytics, machine learning and AI use cases. Work with Amazon Redshift / Athena, Microsoft Fabric (Semantic Models, Direct Lake), Power BI, Databricks SQL & ML capabilities. Support feature engineering, data science workflows, and real-time decisioning systems. Implement data quality, observability, and lineage frameworks. Security, Governance and Compliance Implement secure, enterprise-grade data platforms: AWS IAM / Azure Entra ID (AAD), RBAC, Managed Identities. Governance: Databricks Unity Catalog, Microsoft Purview, AWS Lake Formation. Networking: VPC / VNets, Private Endpoints, Direct Connect / ExpressRoute. Encryption: KMS / Key Vault / TLS. Orchestration and Operations Build orchestrated pipelines using Databricks Workflows, AWS Step Functions / MWAA (Airflow), ADF / Synapse / Fabric Pipelines. Monitoring & observability: Cloud-native monitoring tools (CloudWatch, Azure Monitor). Apply best practices across reliability, performance optimisation, and cost optimisation (FinOps). Requirements Bachelor's degree in Engineering, Computer Science, or a related field. 8+ years of experience in data engineering, with a proven track record of designing and delivering production data platforms. 2+ years of experience leading engineering teams, including hiring, mentoring, performance management, and workload planning. Demonstrated experience leading client-facing engagements, owning technical delivery, managing expectations, and communicating with senior stakeholders. Deep hands-on expertise across at least one major cloud platform (AWS or Azure), with working knowledge of the other. Strong architectural thinking, ability to design end-to-end data platforms and make sound trade-offs across competing concerns. Experience with Databricks and/or Microsoft Fabric in production environments. Knowledge of and experience with Infrastructure as Code (Terraform, Bicep), CI/CD, and container-based workflows (Docker, Kubernetes). Certifications such as AWS, Microsoft, or Databricks certifications are advantageous. Knowledge of data security and compliance standards. Excellent verbal and written communication skills, able to articulate technical concepts to both engineering and business audiences. Strong problem-solving and analytical skills. Self-organising with the ability to prioritise and manage multiple workstreams simultaneously. Ability to work collaboratively with clients, peers, and team members. Willingness to travel to clients as and when required. What We Offer A culture of engineering excellence and an environment where ideas are heard, and builders can build. Competitive compensation and bonus structure, commensurate with a leadership role. A flexible and supportive work environment that values diversity, work-life balance, and personal growth. A clear leadership track with opportunities for career advancement into Principal and Director-level roles. Ongoing learning and development opportunities to enhance your skills. Engagement with cutting-edge technologies and high-impact client projects. Access to a talented team of professionals and mentors. For reference, if you have not heard back from us within 30 days, please consider your application unsuccessful. However, we invite you to keep an eye out for future opportunities and to continue applying. #J-18808-Ljbffr