This publicly traded Fortune 25 company is the largest Medicaid and Medicare managed care provider in the country focused on transforming the health of the communities it serves.
An Enterprise Eventing Platform using Kafka
This healthcare organization has 26.6 million members and over 72,500 employees across 50 states. Through mergers and acquisitions, their aggressive growth strategy has increased revenue by 13% year-over-year to $126 billion in 2021.
This company’s ambitious growth mindset created data integration and data migration challenges. Departments could not discover or share valuable data and often needed permission from database owners, which slowed down business processes. They needed to remove data silos that hindered data discoverability between teams, and they needed to move from batch processing to real-time stream processing to improve data accuracy and accelerate business processes.
They identified Apache Kafka as a solution but lacked the expertise for an enterprise-wide implementation that ensured SOC compliance. Once the Kafka implementation was complete, they needed help migrating and transforming massive amounts of data from multiple newly acquired systems into a standard format. The unprecedented scale of this project included billions and billions of data points from over ten different systems, making each decision crucial to the overall long-term success of the technology implementation.
Phase 1: Kafka Implementation
For this project, we refactored monolithic systems into distributed micro-services, developing an enterprise eventing platform to accommodate the transformation to event-driven architecture. Our applications provided a representational state transfer (REST) and remote procedure call (gRPC) interface to allow development teams to publish and consume events. The platform included:
Structured schema evolutions with Confluent Schema Registry and AVRO
Configurable (self-service) event-routing with K-Tables
Dynamic event-aggregation with Kafka Streams
Distributed event tracing with Jaeger
Event querying against a MongoDB event-store hydrated by Kafka Connect
Self-Service Portal UI and API to allow teams to onboard and maintain their configurations
Phase 2: Data Normalization as a Service
Once we implemented Kafka, the platform around Kafka, and trained the development team on the software, we began the normalization of billions of pieces of data across over ten systems. The primary tech stack included Java/Spring Boot apps running in Kubernetes talking to Apache Kafka and databases such as Mongo, Neo4j, and Postgres. In recent months, they’ve adopted Quarkus as an alternative to Boot. A frontend VueJS app was built from scratch to dynamically manage the mappings between data formats. We developed Gitlab CI/CD pipelines for deployments and built out monitoring and metrics through Kibana/Grafana. We ran this project across the entire stack, and we trained their staff on the technologies.
The Business Benefits
This company’s internal teams now share data in real-time without the hindrance of data silos and permissions hurdles, accelerating business operations. The company is able to interact and engage with providers in real-time through event streams, giving providers real-time updates on the status of their claim appeals. Since its launch, the system has processed billions of records. This client is set up to continue to evolve their products—like adding Confluent Cloud capabilities—without incurring technical debt.
Our consultants had the skills and knowledge to lead this enterprise streaming strategy with Apache Kafka as the tool of choice. Our team assisted in all facets of the project, from design to delivery. We drove technology decisions, trained their staff on the new technology, easily pivoted to the second phase of data normalization, and set them up to continue to evolve their products.