Data Engineer - Google Cloud - GCP - Airflow - Cloud Composer - Python - Docker - IAC - Terraform

Summary: The Data Engineer role focuses on developing and maintaining scalable data pipelines to support AI/ML user experiences and ensure robust product performance. The position requires collaboration with various teams to deliver high-quality data solutions while adhering to security and compliance standards. The ideal candidate will have extensive experience in data engineering, particularly within the Google Cloud ecosystem. Strong technical skills in Python, SQL, and infrastructure management are essential for success in this role.

Key Responsibilities:

• Build and maintain scalable data pipelines for BigQuery using Cloud Composer and Airflow;
• Provide observability and monitoring using Monte Carlo and Looker as well as operational tools such as NewRelic and Splunk, driving reliability, data quality, data quality and robustness;
• Build critical data quality checks to ensure that the data we process and provide to partner teams meets security and compliance requirements;
• Participate in code and technical design reviews;
• Scope and plan work together with product engineering, data science and the business;
• Support other teams for all tasks related to data tracking, collection, ingestion, quality monitoring and alerting.

Key Skills:

• 3+ years of experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt;
• Working knowledge of Python, SQL and Shell Scripting;
• Experience with Docker/k8s and infrastructure as a code tools such as Terraform, Salt or similar;
• 3+ years of significant professional experience developing applications written in a modern programming environment - preferably Scala/Java and/or Python;
• Professional experience across the entire software development life cycle, from gathering requirements to production deployment using CI & CD;
• Confident at articulating complex technical content to peers and partners across the organization via Technical Design Documents.

Salary (Rate): £350 per day

City: London

Country: United Kingdom

Working Arrangements: undetermined

IR35 Status: inside IR35

Seniority Level: undetermined

Industry: IT

Detailed Description From Employer:

Data Engineer - Google Cloud - GCP - Airflow - Cloud Composer - Python - Docker - IAC - Terraform

My client who are leaders in their field are looking for a Data Engineer who will own engineering efforts to provide partner teams within the organisation with data sources that build our AI/ML user experience and help run the client's product robustly and at scale.

Responsibilities:

• Build and maintain scalable data pipelines for BigQuery using Cloud Composer and Airflow;
• Provide observability and monitoring using Monte Carlo and Looker as well as operational tools such as NewRelic and Splunk, driving reliability, data quality, data quality and robustness;
• Build critical data quality checks to ensure that the data we process and provide to partner teams meets security and compliance requirements;
• Participate in code and technical design reviews;
• Scope and plan work together with product engineering, data science and the business
• Support other teams for all tasks related to data tracking, collection, ingestion, quality monitoring and alerting.

Requirements:

• 3+ years of experience managing batching and/or streaming data pipelines and infrastructure using technologies such as GCP, Dataflow/Apache Beam, Dataproc, Spark/Flink, dbt;
• Working knowledge of Python, SQL and Shell Scripting;
• Experience with Docker/k8s and infrastructure as a code tools such as Terraform, Salt or similar;
• 3+ years of significant professional experience developing applications written in a modern programming environment - preferably Scala/Java and/or Python;
• Professional experience across the entire software development life cycle, from gathering requirements to production deployment using CI & CD;
• Confident at articulating complex technical content to peers and partners across the organization via Technical Design Documents.