Building and improving ETL pipelines to enable enhanced cancer treatment

Established in 2011 by a multidisciplinary team of medical professionals, engineers, and data experts, COTA is dedicated to bringing clarity to the often fragmented and hard-to-access real-world medical data landscape. Leveraging the unique technology, sophisticated analytical methods, and extensive knowledge, COTA aims to structure intricate data to deliver a holistic view of cancer, thereby enhancing both treatment and scientific investigation.

COTA developed a suite of products for gathering, organizing and analyzing medical data related to cancer. The software allows for creating detailed representations of cancerous diseases, which are then utilized in various use cases striving to ensure patients receive the best possible care. The process involves sourcing quantitative medical data from various medical providers, organizing it and then presenting it either via an advanced data platform or in the form of specialized reports.

Business

COTA was looking for data engineers to support the existing engineering team in building a data ETL pipeline to transform and warehouse non-PHI clinical data. The challenge laid in the unique requirements of the software - the stack involved PostgreSQL, Google Cloud Platform, and NextGen Connect (Mirth) - a niche solution in healthcare that’s based on interface programming.

Technology

Technical challenges in the project were related to improving the data intake pipeline and included:

• creating services to aggregate and map data from diverse sources,
• improving the performance of these services,
• creating migration and cleanup routines to standardize data sets,
• automating monitoring services to ensure data quality, accuracy, and consistency,
• thoroughly testing the system to guarantee its reliability.

Within weeks, we selected engineers that matched the client's requirements, had successfully onboarded them to the project and integrated with COTA’s in-house engineering team.

Under client’s management, the team has delivered measurable improvements to ETL pipelines, e.g.:

• revamping COTA’s integration service, making it faster and more streamlined,
• setting up a warehouse structure on Google Cloud to efficiently transform and securely store clinical data,
• rigorously testing the system to ensure its reliability, which is critical in healthcare.

We continue our partnership with COTA, steadily growing the data engineering team and working on new projects.