Background Image

An Enterprise Eventing Platform Using Kafka

Icon - Centene

The Customer

This publicly traded Fortune 25 company is the largest Medicaid and Medicare managed care provider in the country focused on transforming the health of the communities it serves.

The Project

An Enterprise Eventing Platform using Kafka


This healthcare organization has 26.6 million members and over 72,500 employees across 50 states. Through mergers and acquisitions, their aggressive growth strategy has increased revenue by 13% year-over-year to $126 billion in 2021.

The Challenge

This company’s ambitious growth mindset created data integration and data migration challenges. Departments could not discover or share valuable data and often needed permission from database owners, which slowed down business processes. They needed to remove data silos that hindered data discoverability between teams, and they needed to move from batch processing to real-time stream processing to improve data accuracy and accelerate business processes.

They identified Apache Kafka as a solution but lacked the expertise for an enterprise-wide implementation that ensured SOC compliance. Once the Kafka implementation was complete, they needed help migrating and transforming massive amounts of data from multiple newly acquired systems into a standard format. The unprecedented scale of this project included billions and billions of data points from over ten different systems, making each decision crucial to the overall long-term success of the technology implementation.

The Solution

Phase 1: Kafka Implementation

For this project, we refactored monolithic systems into distributed micro-services, developing an enterprise eventing platform to accommodate the transformation to event-driven architecture. Our applications provided a representational state transfer (REST) and remote procedure call (gRPC) interface to allow development teams to publish and consume events. The platform included:

  • Structured schema evolutions with Confluent Schema Registry and AVRO

  • Configurable (self-service) event-routing with K-Tables

  • Dynamic event-aggregation with Kafka Streams

  • Distributed event tracing with Jaeger

  • Event querying against a MongoDB event-store hydrated by Kafka Connect

  • Self-Service Portal UI and API to allow teams to onboard and maintain their configurations

Phase 2: Data Normalization as a Service

Once we implemented Kafka, the platform around Kafka, and trained the development team on the software, we began the normalization of billions of pieces of data across over ten systems. The primary tech stack included Java/Spring Boot apps running in Kubernetes talking to Apache Kafka and databases such as Mongo, Neo4j, and Postgres. In recent months, they’ve adopted Quarkus as an alternative to Boot. A frontend VueJS app was built from scratch to dynamically manage the mappings between data formats. We developed Gitlab CI/CD pipelines for deployments and built out monitoring and metrics through Kibana/Grafana. We ran this project across the entire stack, and we trained their staff on the technologies.

Icon - Kafka
Icon - Confluent
Icon - Jaeger
Icon - MongoDB
Icon - Java
Icon - Spring Boot
Icon - Kubernetes
Icon - Vue.JS
Icon - GitLab CICD
Icon - Grafana

The Business Benefits

This company’s internal teams now share data in real-time without the hindrance of data silos and permissions hurdles, accelerating business operations. The company is able to interact and engage with providers in real-time through event streams, giving providers real-time updates on the status of their claim appeals. Since its launch, the system has processed billions of records. This client is set up to continue to evolve their products—like adding Confluent Cloud capabilities—without incurring technical debt.

Why Improving?

Our consultants had the skills and knowledge to lead this enterprise streaming strategy with Apache Kafka as the tool of choice. Our team assisted in all facets of the project, from design to delivery. We drove technology decisions, trained their staff on the new technology, easily pivoted to the second phase of data normalization, and set them up to continue to evolve their products.

Modern Data

Get Started

Learn more about how Improving can help you get started by contacting us today at or filling out a contact form through the link below.

Most Recent Case Studies

Explore our case studies and get inspired from thought leaders throughout our enterprises.
Thumbnail -Modern Web Application Platform with AWS

Modern Web Application Platform with AWS

Minnesota Public Radio (MPR) collaborated with Improving to create a proof of concept to host their news, streaming music, and podcast sites in AWS.