DeveloperWeek Management 2024 + AI DevSummit 2024 (+ DW Europe/LATAM 2024)

Rakesh Jain, IBM, Senior Technical Staff Member & Researcher

In this session, we will describe how we built the Data Management Platform based on open table format Apache Iceberg, serving terabytes of data. The primary use case - data preprocessing for foundation models training at large scale. In addition, the Lakehouse was extended to support model checkpoints store and controlled sharing, providing a full Data and Model Factory experience. We will talk about various approaches we tried with focus on data acquisition, governance, preprocessing, leading up to tokenization, maintaining lineage from data to models and back.
In addition, the platform has been extended to support other aspects of foundation models, including fine tuning, evaluation, Retrieval Augmented Generation (RAG) etc. We will also talk about different strategies we adopted to deal with small data and big data, so that we can provide a seamless experience to different user bases of our Data Pl