DeveloperWeek Management 2024 + AI DevSummit 2024 (+ DW Europe/LATAM 2024)

Marisa Smith, Tobiko Data, Developer Advocate

Data. It's everywhere- used for everything. It helps us make intelligent business decisions, makes our teams more efficient and transparent, trains our AI and ML models, or allows users to access info. Yet, our engineering teams face several complex challenges when updates or new features are needed for downstream data assets.

These problems include costly data outages, inaccurate reporting or metrics affecting the business outcomes, and delayed deliverables due to complex communication chains. Because of these problems, engineers and data practitioners often resort to building new pipelines and datasets instead of changing existing ones, escalating technical debt and increasing complexity. These factors drive up development costs, slow iteration speed, and make data less reliable.

At the heart of the open-source project SQLMesh, we have created a novel data pipeline development workflow from our combined experience at companies like Airbnb, Apple, Google, and Netflix. We designed this data transformation tool to handle complexities associated with evolving data pipelines at the internet scale. In this session, we will discuss the innovations and core concepts of SQLMesh and show how it has helped other data teams scale data safely, quickly and reliably.