Data Engineer GCP

Activity description and concrete tasks:

1. Infrastructure Deployment & Management: Efficiently deploy and manage infrastructure on Google Cloud Platform (GCP) including network architectures (Shared VPC, Hub-and-Spoke), security implementations (IAM, Secret Manager, firewalls, Identity-Aware Proxy), DNS configuration, VPN, and Load Balancing.
2. Data Processing & Transformation: Utilize Hadoop cluster with Hive for querying data, and PySpark for data transformations. Implement job orchestration using Airflow.
3. Core GCP Services Management: Work extensively with services like Google Kubernetes Engine (GKE), Cloud Run, BigQuery, Compute Engine, and Composer, all managed through Terraform.
4. Application Implementation: Develop and implement Python applications for various GCP services.
5. CI/CD Pipelines: Integrate and manage GitLab Magenta CI/CD pipelines for automating cloud deployment, testing, and configuration of diverse data pipelines.
6. Security & Compliance: Implement comprehensive security measures, manage IAM policies, secrets using Secret Manager, and enforce identity-aware policies.
7. Data Integration: Handle integration of data sources from CDI, Datendrehscheibe (FTP servers), TARDIS API´s and Google Cloud Storage (GCS).
8. Multi-environment Deployment: Create and deploy workloads across Development (DEV), Testing (TEST), and Production (PROD) environments.
9. AI Solutions: Implement AI solutions using Google’s Vertex AI for building and deploying machine learning models.
10. Certification Desired: Must be a certified GCP Cloud Architect or Data Engineer.

Skills Required:

• Strong proficiency in Google Cloud Platform (GCP)
• Expertise in Terraform for infrastructure management
• Skilled in Python for application implementation
• Experience with GitLab CI/CD for automation
• Deep knowledge of network architectures, security implementations, and management of core GCP services
• Proficiency in employing data processing tools like Hive, PySpark, and data orchestration tools like Airflow
• Familiarity with managing and integrating diverse data sources
• Certified GCP Cloud Architect and Data Engineer