The Senior Data Platform Engineer will work 100% in the Microsoft Azure cloud to build, operate, and enhance a lean, efficient, and powerful data platform that identifies new pharmaceuticals expected to enter the global market within the next 2.5 years, and track them until market entry has been achieved.
The Senior Data Platform Engineer will use Python, Azure DataBricks, Azure Data Factory, and https://scrapy.org to automate ingestion of publicly available information via RSS feeds, scraping, and API calls that are persisted to multiple types and tiers of Azure storage. The Senior Data Engineer will support the processing, aggregation, and reshaping of the data to enable analytics and ML work conducted by ECRI Data Scientists and Specialists. Curated and processed data will be distributed by API's to a public facing website as well as used internally by ECRI researchers developing reports to highlight pharmaceuticals with high potential to cause a significant impact to one or more areas of healthcare in certain countries. Reports will include healthcare use, infrastructure, service delivery, disease management, patient health outcomes, and healthcare costs.
In addition the system will support regular and frequent human curation, review and feedback to ensure all records in the system are up to date and of high quality.
The Senior Data Platform Engineer will work closely with ECRI product development, enterprise architecture, and business experts to continually review, improve, and refine the ingestion of data from harvested trials, news releases & reports, and other sources being captured on a daily basis.
This system must be of high quality, excellent performance and optimal costs to operate month to month. Therefore a thorough understanding of Azure pricing models and which services are best for which use cases is critical.
The Senior Data Platform Engineer is a contractor role, expected to last for four months.
Reasonable Accommodations Statement
To accomplish this job successfully, an individual must be able to perform, with or without reasonable accommodation, each essential function satisfactorily. Reasonable accommodations may be made to help enable qualified individuals with disabilities to perform the essential functions.
Essential Functions Statements(s)
- Assemble large, complex data sets that meet functional / non-functional business requirements.
- Perform root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement.
- Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
- Assist in gathering and document business requirements.
- Support and work with cross-functional teams.
- Participate in team meetings and code reviews.
- Work independently and within a team when needed.