Job Description
Skill: DataOps
exp- 4-5 years
Np- 15days to Immediate
Job Description:
As a DataOps Engineer, you will play a key role in client's data operations, working closely with cross-functional teams to ensure efficient and reliable data management and delivery.
You will be responsible for implementing DataOps principles, practices, and tools to streamline the data processes and enhance collaboration among data professionals.
Your expertise in data engineering, automation, and DevOps will contribute to the success of client data-driven initiatives.
Responsibilities:
1. Collaborate with cross-functional teams: Work closely with data engineers, data scientists, analysts, and other stakeholders to understand data requirements, identify bottlenecks, and implement efficient data operations strategies.
2. Automation:
Implement automation tools and processes to streamline data operations, including data ingestion, transformation, data quality checks, and deployment. Leverage appropriate technologies and frameworks to maximize efficiency.
3. Data quality and monitoring: Implement data quality checks and validation processes to ensure the accuracy, completeness, and consistency of data.
Set up monitoring systems to proactively identify and address data issues or bottlenecks.
4.
Performance optimization:
Identify opportunities for performance optimization in data operations, including data processing, storage, and retrieval. Optimize data workflows to enhance overall system efficiency.
5. Stay updated with industry trends: Keep abreast of emerging trends, technologies, and best practices in DataOps, data engineering, and DevOps. Continuously enhance your knowledge and skills to contribute to the growth and innovation of the DataOps team.
Must-Have Skills:
- Awareness of PySPark with exposure to it
- Strong experience in data engineering, data integration, or related roles.
- Proficiency in programming languages such as Python, Java, or Scala.
- Understanding of SQL and experience with relational and NoSQL databases.
- Familiarity with data pipeline orchestration tools like Apache Airflow, Luigi, or similar frameworks.
- Hands-on experience with cloud platforms such as AWS, Azure, or GCP.
- Experience with CI/CD tools and processes, version control systems, and automated testing frameworks.
- Knowledge of data warehousing concepts, data modeling, and ETL/ELT processes.
- Excellent problem-solving and analytical skills, with attention to detail.
- Effective communication and collaboration skills, with the ability to work effectively in cross-functional teams.
- Self-motivated and able to work independently, managing multiple priorities and meeting deadlines.
Good-to-Have Skills:
- Experience with big data technologies such as Hadoop, Spark, or Kafka.
- Familiarity with containerization technologies like Docker and orchestration tools like Kubernetes.
- Knowledge of data governance and data security best practices.
- Proficiency in data visualization tools such as Tableau, Power BI, or similar.
- Experience with data streaming and real-time data processing.
- Familiarity with agile development methodologies.
- Knowledge of data privacy regulations and compliance.
- Certification in relevant technologies or frameworks (e.g., AWS Certified Big Data Specialty, Google Cloud Data Engineer, etc.).
V 0.0.1