Work Authorizations: Open to All

Requirements:

• Hadoop
• Spark / Python
• Kafka

JD/Requirements:

• Perform data migration and conversion activities on different applications and platforms.
• Design , development and testing of data ingestion pipelines, perform end to end automation of ETL process for various datasets that are being ingested into the big data platform.
• Perform data profiling/analysis, discovery, analysis, suitability and coverage of data, and identify the various data types, formats, and data quality issues which exist within a given data source.
• Develop transformation logic, interfaces and reports as needed to meet project requirements.
• Participate in discussion for technical architecture, data modeling and ETL standards, collaborate with Product Managers, Architects and Senior Developers to establish the physical application framework (e.g. libraries, modules, execution environments)
• Develop integrated automated test suites to validate end to end data pipeline flow, data transformation rules, and data integrity.
• Develop tools to measure the data quality and visualize the anomaly pattern in source and processed data.
• Integrate automated processes into continuous integration workflows.
• Contribute to data quality assurance standards and procedures.