The Customer
The Friedkin Group
The Project
The Friedkin Group
Overview
The Friedkin Group, an international conglomerate based in Houston, Texas, sought to establish a shared Databricks platform to drive AI and ML work across its diverse portfolio of companies. Improving was engaged to architect and implement an innovative approach to managing the Databricks Intelligence Platform as a strategic initiative. The project involved designing an ephemeral Databricks environment on AWS with strict compliance to infrastructure-as-code principles to minimize support and maximize automation.
The Business Challenge
The Friedkin Group needed a scalable and efficient platform to support AI and ML initiatives across its numerous subsidiaries. The goal was to unify data processing, delta lakehouse storage, agentic AI and inference solutions, and analytics functions on a single, shared service platform. Challenges included ensuring compliance, minimizing manual support through end-end automation, and creating a robust environment capable of handling diverse workloads from multiple companies. This required a solution that was both innovative and practical, capable of quick adaptation and rigorous governance.
Our Solution
Improving approached the challenge by working closely with Ajit Singh, the lead architect at The Friedkin Group, to realize his vision for an ephemeral Databricks platform on AWS. The team implemented infrastructure-as-code using Terraform, REST APIs, Databricks CLI, and Python SDK, allowing the environments to rebuild every three months, ensuring compliance and reducing manual interventions. Continuous collaboration with Databricks representatives and other technology partners was crucial in overcoming technical hurdles and achieving a mature implementation.
The Business Benefits
Enhanced Compliance: Regular recycling of the Databricks environment enforced strict adherence to infrastructure-as-code principles.
Reduced Manual Support: Automation minimized the need for manual management, freeing up resources for strategic tasks.
Scalability: The platform supported varied workloads from multiple subsidiaries, enhancing overall operational efficiency.
Innovation Recognition: The project won a global innovation award from Databricks in the manufacturing sector, highlighting its pioneering approach.
Portfolio Integration: Successfully onboarded multiple companies, driving widespread adoption of AI and ML practices.
Data Governance: Improved data governance across the platform with partnerships and advanced tooling.
Technologies and Methodologies Used
Databricks: Core platform for data processing and analytics.
AWS: Infrastructure hosting and management.
Databricks CLI: Command-line interface for managing Databricks resources.
Terraform: Infrastructure-as-code for automating environment setup.
Python SDK: For developing custom solutions and integrations.
GitHub: Version control and collaboration among team members.
Partnerships
The project's success was bolstered by strategic partnerships. Boston Consulting provided initial data science support, while Tiger Analytics contributed a large team of data scientists and engineers. Technology partners included Fivetran for data movement and governance support from Ataccama. Continuous collaboration with Databricks representatives ensured alignment with platform capabilities and innovation.
Lessons Learned
Automation is Key: Automated processes reduced manual workload and enhanced compliance.
Small, Focused Teams: A small, dedicated team proved more effective than larger groups, ensuring better coordination and agility.
Client Trust: Building trust through early deliverables and maintaining open, collaborative relationships was crucial.
Continuous Innovation: Leveraging cutting-edge technologies and methodologies ensured the project stayed ahead of industry trends.
Effective Partnerships: Collaborating with third-party providers added significant value and expertise to the project.
Flexible Architecture: The ephemeral nature of the Databricks environment provided flexibility and scalability to adapt to changing needs.
Conclusion
The Friedkin Group's Databricks project exemplifies the power of innovative thinking and strategic collaboration. By automating and regularly recycling the environment, we achieved a scalable and compliant platform that supports AI and ML across numerous subsidiaries. The project not only met its initial objectives but also set a new benchmark in platform maturity and innovation, earning industry recognition and paving the way for future advancements. Our unique approach and dedicated team were instrumental in delivering a solution that drives significant business benefits and positions The Friedkin Group for continued success.
