Data Engineer Looking for an exciting job opportunity with a dynamic and growing Cybersecurity & Big Data company? This role offers hybrid opportunities, numerous benefits and a fantastic, supportive work culture. About the role: We are seeking an experienced Data Engineer to join our team. This project will involve working in a high-profile project team within an agile enterprise client . The ideal candidate will be proficient in managing and analysing large volumes of data, with expertise in big data technologies within an Azure Platforms environment. This position offers the opportunity to work on cutting-edge projects in a dynamic and challenging environment. The position includes all aspects of: Design, build, and maintain scalable big data pipelines tailored for data analysis, processing, and reporting, ensuring high performance and reliability. Connect to diverse source systems, ingest data into Data Warehouses, and manage end-to-end workflows for advanced analytics. Utilise Apache Spark (Scala) for comprehensive large-scale data transformation tasks, addressing both streaming and batch processing needs to support advanced analytics. Manage and optimise warehousing solutions, focusing on schema design, data partitioning, and query performance to support large-scale data storage and retrieval. Developing complex SQL views, stored procedures, and transformations to enable seamless data processing and modelling. Collaborate effectively with cross-functional teams to understand business data requirements, translating them into technical solutions that leverage cutting-edge big data technologies. Maintain high standards of data quality and integrity, implementing best practices and methodologies to ensure accurate and reliable data processing workflows. Provide expert technical guidance and support concerning big data technologies, contributing to the team's continuous improvement in big data practices and strategies. The successful candidate will have: A minimum of 5 years of relevant experience in a Data Engineer role, with proven experience with Azure Data Platforms, including Azure Synapse, SQL, Azure Data Factory, and Azure Databricks. Proficiency in working with Linux-based platforms, showcasing strong operational skills and understanding. Strong communication and interpersonal abilities, capable of engaging effectively in a client-facing consulting capacity. Deep expertise in Azure Data Factory, Apache NiFi, Apache Spark (Scala) , Apache Hive, and a solid understanding of Hadoop ecosystem components, demonstrating comprehensive experience in managing and processing big data. Demonstrated success in building ADF pipelines for robust data workflows. Advanced SQL development expertise, including the use of stored procedures, Common Table Expressions (CTEs), and views. Experience in data modelling, ETL processes, and the management of data warehousing solutions, contributing to the efficiency and effectiveness of data operations. Competence in additional programming languages, including Python and Shell scripting, to support diverse development tasks and automation efforts. An understanding of cloud services (such as AWS, Azure, Google Cloud) and their data-related offerings is desirable, adding value to cloud-based project deployments. Excellent problem-solving skills, with the ability to tackle complex issues independently or collaboratively within a team environment. Demonstrated proficiency in version control and source code management using Git, along with an understanding of continuous integration/continuous deployment (CI/CD) practices, enhancing collaboration and code quality. Contract Details Duration: Fixed Term Contract. 6 months with possibility of extension. Location: Melbourne Start Date: March 2024 Compensation: Competitive, based on experience What’s in it for you? The position presents an excellent opportunity to work in an innovative, agile environment with highly skilled professionals. In addition to: Staff discounts on Tech Salary packaging options Flexible working environment A great team culture Training allowances Rewarding salary based on skills and experience A good match for you or simply curious to learn more? Head to our careers page on our website to share your details with the Skillfield team & we will be in touch. We value a diverse workplace and strongly encourage people from all backgrounds and minority groups to apply. In the meantime, learn more about us by visiting our website and following Skillfield on LinkedIn.