DHCS Data Platform

The Project

DHCS Data Platform

Overview

The DHCS (California Department of Healthcare Services) Data Platform project is an ongoing initiative aimed at modernizing the state’s outdated technology infrastructure. Partnering with Confluent and leveraging the Databricks platform, Improving has played a critical role in this modernization effort. The project focuses on building a data platform to improve data visibility, governance, and decision-making in the Behavioral Health Transformation (BHT) program, which aims to enhance mental health services for California residents.

The Challenge

DHCS faced significant challenges with its legacy technology, which included fragmented data stored in various outdated systems such as Excel, Microsoft Access, and a legacy Teradata warehouse. The inefficiencies in data quality and timeliness led to difficulties in tracking and measuring health outcomes, particularly in the Behavioral Health Transformation (BHT) program. These issues impeded DHCS’s mission to provide equitable access to quality healthcare and leverage data to improve health outcomes for California residents.

The Solution

Improving approached this challenge by developing a comprehensive data platform centered around Databricks for data warehousing and compute, enhancing data governance, and integrating with existing DHCS infrastructure. The solution involved building data pipelines, ingesting and transforming data, and constructing BI reports and dashboards. The team also implemented Confluent platform to facilitate data movement from legacy systems to Databricks, and employed tools like Collibra for data cataloging. Collaboration with Databricks and Confluent ensured seamless integration and effective implementation of the new platform.

Business Benefits

• Improved Data Reliability: Migrated from fragmented and outdated systems to a centralized Databricks platform, enhancing data accuracy and consistency.

• Enhanced Data Visibility: Enabled real-time data access and comprehensive reporting through BI tools, allowing for better tracking and measurement of health outcomes.

• Strengthened Data Governance: Implemented robust data governance practices, ensuring controlled access, data privacy, and compliance with regulations like HIPAA and FedRAMP.

• Increased Scalability: Leveraged AWS’s private cloud infrastructure for scalable data storage and processing capabilities.

• Efficient Data Movement: Utilized Confluent platform for seamless data transfer from legacy systems to Databricks, improving data pipeline efficiency.

• User-Friendly Platform: Developed a self-service platform for DHCS data engineers, facilitating easy data ingestion, transformation, and product creation.

Technologies And Methodologies Used

• Confluent Platform: Used for data movement and integration from legacy systems.

• Collibra: Data cataloging tool for metadata management and governance.

• Databricks: Core platform for data warehousing, compute, and analytics.

• AWS: Provided the underlying private cloud infrastructure.

• Kubernetes: Employed for deploying and managing applications and data pipelines.

• Power BI: Integrated for building and visualizing business intelligence reports.

Partnerships

Improving collaborated closely with technology partners Confluent and Databricks to ensure the successful implementation of the DHCS Data Platform. Confluent assisted in deploying the Confluent platform within DHCS’s AWS environment, while Databricks provided significant support in integrating their platform with existing DHCS processes. Regular communications and strategic meetings with both partners facilitated smooth project execution and alignment with DHCS’s modernization goals.

Lessons Learned

1. Regulatory Compliance: Essential to understand and adhere to extensive regulations in government and healthcare projects.

2. Documentation and Training: Crucial for successful platform adoption and usability among diverse user groups.

3. Client Collaboration: Maintaining flexibility and open communication with DHCS facilitated problem-solving and project adjustments.

4. Governance Readiness: Implementing robust data governance practices early on ensures long-term data integrity and security.

5. Integration Complexity: Integrating new technologies with existing systems requires careful planning and execution.

6. User-Centric Development: Focusing on the needs of various stakeholders, including non-technical users, enhances platform adoption and effectiveness

Conclusion

The DHCS Data Platform project exemplifies Improving’s ability to manage complex modernization efforts in a highly regulated environment. By leveraging advanced technologies like Databricks and Confluent, and fostering strong partnerships, Improving successfully addressed DHCS’s data challenges, resulting in improved data reliability, visibility, and governance. The project underscores our commitment to delivering scalable, user-friendly solutions that drive meaningful outcomes in healthcare services.