Unison Consulting Looking for Data Engineer (Spark ,Scala, Elastic Search) at Wellington, Wellington Region, New Zealand Full Time

• Design, develop, and implement Spark Scala applications and data processing pipelines to process large volumes of structured and unstructured data
• Integrate Elasticsearch with Spark to enable efficient indexing, querying, and retrieval of data
• Optimize and tune Spark jobs for performance and scalability, ensuring efficient data processing and indexing in Elasticsearch
• Collaborate with data engineers, data scientists, and other stakeholders to understand requirements and translate them into technical specifications and solutions
• Implement data transformations, aggregations, and computations using Spark RDDs, DataFrames, and Datasets, and integrate them with Elasticsearch
• Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
• Troubleshoot and resolve issues related to data processing, performance, and data quality in the Spark-Elasticsearch integration

Requirements

• 5+ Years as Data Engineer
• 5 Years experience in Spark, Scala, Elastic Search
• Develop and maintain scalable and fault-tolerant Spark applications, adhering to industry best practices and coding standards
• Monitor and analyze job performance metrics, identify bottlenecks, and propose optimizations in both Spark and Elasticsearch components
• Stay updated with emerging trends and advancements in the big data technologies space to ensure continuous improvement and innovation