Job Description:
We are seeking a highly skilled and experienced Data Engineer Lead who possesses expertise in Python, Airbyte, and Cassandra to join our dynamic team. As the Data Engineer Lead, you will be responsible for designing, developing, and maintaining data infrastructure, ensuring the efficient and reliable collection, integration, and processing of data from various sources.
Responsibilities:
Design, develop, and optimize data pipelines and ETL processes using Python, Airbyte, and other relevant technologies to extract, transform, and load data from multiple sources.
Architect and maintain scalable and reliable data storage solutions using Cassandra, ensuring high availability, performance, and data integrity.
Implement data quality checks and validation procedures to ensure the accuracy, consistency, and completeness of data.
Troubleshoot and resolve data-related issues, providing timely and effective resolutions to minimize disruptions to data pipelines and analytics processes.
Develop and maintain documentation, including data models, data dictionaries, and technical specifications, to facilitate understanding and collaboration across teams.
Collaborate with cross-functional teams, including data scientists, analysts, and stakeholders, to understand data requirements and translate them into scalable data solutions.
Stay up-to-date with emerging trends, technologies, and best practices in data engineering, and proactively apply them to drive innovation and continuous improvement.
Requirements:
Proven experience as a Data Engineer, with a track record of successfully leading data engineering projects.
Strong proficiency in Python for data processing and ETL tasks.
Solid understanding and practical experience with Cassandra for data storage and management.
Strong attention to detail, with a focus on data accuracy, quality, and integrity.
Ability to work in a fast-paced, dynamic environment and adapt to changing priorities and requirements.
Awareness of CI/CD knowledge is a plus
Certification on Cloud is preferable
Familiarity with other data engineering tools and technologies, such as Apache Kafka, Spark, or Hadoop, is a plus.
A consultant with AirByte tool knowledge will also be helpful to start the project.
Location: Hyderabad
Required Keywords
AirByte, ETL, Python, Cassendra, SQL, Big Data, Pipeline
Optional Keywords
Pyspark, Hadoop, Data Engineer, Data Quality
Years of Exp. 8-9 Years
Immediate joiner