Globaldev Group

<p>Kanda Software is a dynamic company based in the US, known for delivering innovative software solutions and technology services. We are currently seeking a highly skilled Senior Data Engineer to join our team and contribute to our exciting projects.</p><h3><strong>Responsibilities:</strong></h3><ul> <li>Lead the design, development, and optimization of complex data systems, including data flows, lakes, warehouses, and ETLs, across various cloud platforms.</li> <li>Architect and manage Elasticsearch clusters, focusing on performance optimization, security, and scalability.</li> <li>Create and optimize indices for performance and storage efficiency, leveraging index templates and mappings.</li> <li>Build and optimize Elasticsearch queries using the Query DSL, including complex aggregations and full-text search.</li> <li>Implement data ingestion pipelines to load data from various sources, including relational databases (RDBMS), into Elasticsearch.</li> <li> Oversee performance monitoring, cluster management, security protocols, backup and restore procedures, and version management of Elasticsearch clusters.</li> <li>Develop and maintain data pipelines using industry-standard tools such as Apache Spark, Apache Hive, Apache Airflow, and Stitch.</li> <li>Implement data pipeline observability strategies to ensure reliable data flow and system transparency.</li> <li>Apply modern software development practices, including Agile methodologies, Test-Driven Development (TDD), and Continuous Integration/Continuous Deployment (CI/CD), to enhance data engineering workflows.</li> </ul><h3>Requirements:</h3><ul> <li>A minimum of 5 years of experience as a technical lead, building and extending complex data systems.</li> <li>Advanced proficiency in Python, with at least 3 years of experience.</li> <li>Strong SQL writing and tuning skills.</li> <li>Expert-level knowledge of Elasticsearch, including configuration, administration, and optimization techniques.</li> <li>Demonstrated experience with building and maintaining data pipelines using tools such as Spark, Hive, Airflow, and Stitch.</li> <li>Proven experience with data pipeline observability strategies and tools.</li> <li>Strong understanding of software development practices, including Agile, TDD, and CI/CD.</li> </ul><h3>Will be a plus:</h3><ul> <li>Experience building or maintaining streaming platforms using Apache Kafka.</li> <li>Proficiency in PySpark for distributed data processing.</li> <li>Skills in implementing and managing containerization solutions using Kubernetes, Docker, etc., including observability of containerized applications.</li> <li>Experience with Big Data as a Service (BDaaS) tools, such as AWS EMR or Azure HDInsight.</li> <li>Experience in building and extending Machine Learning and Data Science platforms.</li> <li>Familiarity with integrating data flows with third-party Business Intelligence (BI) platforms like Domo, Tableau, Sisense, etc.</li> </ul><h3>What we offer:</h3><ul> <li>Competitive salary and benefits package;</li> <li>Flexible remote work arrangements;</li> <li>Opportunities for professional development and growth;</li> <li>Periodic review of the salary;</li> <li>Paid events attendance;</li> </ul>