POSITION OVERVIEW
The Senior Data Engineer will expand and optimize the data and data pipeline architecture, as well as optimize data flow and collection for cross functional teams. The Senior Data Engineer will perform data architecture analysis, design, development and testing to deliver data applications, services, interfaces, ETL processes, reporting and other workflow and management initiatives. The role will work closely with the business, data analysts and IT teams to support on data strategy initiatives and will ensure optimal data delivery architecture is consistent throughout the strategy. The role also will follow modern SDLC principles, test driven development and source code reviews and change control standards in order to maintain compliance with policies. This role requires a highly motivated individual with strong technical ability, data capability, excellent communication and collaboration skills including the ability to develop and troubleshoot a diverse range of problems. This role can work out of New York City, Hartford, or Boston .
RESPONSIBILITIES
- Design and develop enterprise data / data architecture solutions using AWS Glue, Lambda and other data technologies like Spark, Scala, Java, Python, SQL etc.
- Design and develop machine learning algorithms and AI models for business requirements.
- Study and transform data science prototypes and apply appropriate machine learning algorithms and tools.
- Run machine learning tests and experiments, and document findings and results.
- Train, retrain, and monitor machine learning systems and models as needed.
- Assemble large, complex data sets that meet functional / non-functional business requirements.
- Create and maintain optimal data pipeline architecture.
- Devise and execute continual improvement initiatives in all Data Management Service delivery and technology, with a focus on delivery velocity and quality.
- Partner with business leaders to determine and prioritize delivery initiatives.
- Define or influence system, technical and application architectures for major areas of development.
- Devise and execute in software development life cycle including requirements gathering, development, testing, release management, and maintenance.
- Engage with business partners to report (formally and informally) on technology strengths, weaknesses, successes, and challenges on a regular basis.
- Ability to do analytical programming in EDW architecture to bridge the gap between a traditional DB architecture and a Hadoop centric architecture.
- Highly organized and analytic, capable of solving business problems using technology.
- Ensure appropriate change management and other technology methodologies are carried out on a consistent basis over time.
- Should be an individual with in-depth technical knowledge and hands-on experience in the areas of Data Management, BI Architecture, Product Development, RDBMS and non-RDBMS platforms.
- Should have excellent analytical skills, able to recognize data patterns and troubleshoot the data.
- Will be responsible for design and delivery of data solutions to empower data migration initiatives, BI initiatives, dashboards development etc.
QUALIFICATIONS:
- At least 8 to 10 years of overall IT experience and 3 years’ relevant experience in design and development complete end-end to design of enterprise-wide big data solution, Machine Learning Algorithms.
- Master’s in data science preferred, Bachelor's degree in computer science or related field required.
- Design and Develop big data solutions using Spark, Scala, AWS Glue etc.
- Strong Analytical and Leadership skills.
- Strong ML experience.
- AWS Glue and Lambda experience is must.
- Application development experience in Scala/Python.
- Strong Database SQL experience preferably Redshift.
- Highly organized and analytic, capable of solving business problems using technology.