Immediate Start Permanent Opportunities CBD location We’re partnering with a large, well-established enterprise organisation that is seeking a Lead Data Engineer to support a high-profile security insights platform. This role offers long-term stability, a strong team culture, and the opportunity to lead the development of secure, scalable data pipelines. About the role As a Lead Data Engineer, you will be responsible for designing, building, and maintaining robust data pipelines using PySpark. You'll be working within a collaborative engineering environment focused on data transformation, ingestion, and performance optimisation. You’ll also play a key role in governance and stakeholder engagement, acting as a technical leader within the team. What you’ll be doing Designing and developing scalable data pipelines using PySpark and Python Ingesting data from various sources including REST APIs, SFTP, SQL, and flat files Working within a Linux environment using tools like Bash and managing network components such as proxies and firewalls Collaborating with teams to optimise CI/CD workflows and improve data platform operations Leading technical discussions and providing direction across key data initiatives Supporting internal stakeholders around data governance, access control, and quality management Tuning performance through techniques like partitioning, pagination, and broadcasting What we’re looking for Proven experience with Apache Spark, ideally using PySpark Solid understanding of data ingestion, transformation, and distributed computing Experience working in Agile teams and familiarity with tools like Jira and ServiceNow Ability to manage and support data services within large, enterprise-scale environments Experience working as a senior or lead engineer, with strong stakeholder engagement skills Exposure to platforms like Databricks, Snowflake, or Cloudera is a bonus Familiarity with data governance, quality, and master data management is highly regarded Consultant [email protected] Reference number: BH-59766 Profession:ICTInformation Management (BI / DW / DG) Company: Bluefin Resources Date posted: 4th May, 2025