DeveloperWeek Management 2024 + AI DevSummit 2024 (+ DW...

Attending this event?

DeveloperWeek Management 2024 + AI DevSummit 2024 (+ DW Europe/LATAM 2024)

10:30am PDT

OPEN TALK (AI): Solving Enterprise Use Cases through Agents

Mark Kim-Huang, Gradient, Co-founder and Chief Architect

Have you ever struggled to make an LLM perform the task that you intend? Providing more raw context is still only a stop gap. Even though new releases of models such as Gemini support context lengths of up to 1M tokens, Retrieval Augmented Generation (RAG) is brittle to the retrieval mechanism and often fails to reconcile between multiple sources of context. The same failure points are amplified when introducing Autonomous Agents due to compounding effects of chaining multiple tasks together.

Learn how to build a reliable production AI Agent System and apply it to high stake industries such as financial services and healthcare. We will deep dive into sentiment analysis as an unexpected and useful showcase of how you can go from a naive system to production level performance through a composable AI system to manage an agent’s memory, connect it to the higher-level concept of sentiment, and utilize reasoning traces for continual performance enhancement.

Speakers

Mark Kim-Huang

Co-founder & Chief Architect, Gradient

Mark is a co-founder and Chief Architect at Gradient, a full stack AI platform that enables businesses to build customized agents to power enterprise workloads. Known for his pioneering work in LLMs and fine-tuning, Mark is a frequent contributor to the AI and MLOps community. Prior... Read More →

Wednesday May 29, 2024 10:30am - 10:55am PDT
AI DevSummit Expo Stage

AI DevSummit: Generative AI & LLMs

Talk Type OPEN TALK
Track or Conference AI DevSummit, Generative AI & LLMs
In-Person/Virtual In-Person

11:00am PDT

OPEN TALK (AI): Gemini vs. GPT4? The Science of LLM Benchmarks

Shir Chorev, Deepchecks, Co-founder & CTO

In this talk, we'll delve into the methods, metrics, and insights behind popular LLM Benchmarks, and learn how to review them effectively. We’ll take a close look at some of the notable leaderboards and LLM benchmarks (such as MMLU, HellSwag, TruthfulQA, MT-Bench) and understand their complexities and uniqueness. Ultimately, we’ll answer the question: did Gemini really outperform GPT4?
Finally, we’ll differentiate between evaluating LLM models to evaluating LLM based applications, and connect the discussion back to practical, real-world applications.

Speakers

Shir Chorev

Co-founder & CTO, Deepchecks

Shir is the co-founder and CTO of Deepchecks, an MLOps startup for continuous validation of ML models and data. Previously, Shir worked at the Prime Minister’s Office and at Unit 8200, conducting and leading research in various Machine Learning and Cybersecurity related challenges... Read More →

Wednesday May 29, 2024 11:00am - 11:25am PDT
AI DevSummit Expo Stage

AI DevSummit: Generative AI & LLMs

Talk Type OPEN TALK
Track or Conference Generative AI & LLMs, AI DevSummit, MLOps & AIOps
In-Person/Virtual In-Person

11:30am PDT

PRO TALK (AI): Are You Gen AI Ready? A Guide to Surviving the Bot Horde

Ben Lloyd Pearson, LinearB, Director of Developer Relations

Generative AI tools have rapidly become commonplace in the typical software engineer’s arsenal, and many feel that we’re on the verge of a paradigm shift in how modern software engineering organizations operate. The next generation of Gen AI tools will reduce or eliminate significant time-consuming work for developers. However, that doesn’t necessarily result in overall productivity gains unless your organization has prepared to use the new tooling.

This session will focus on the connection between human creativity and LLM capabilities and show you how to leverage the new tools for growth while mitigating the risks they introduce to your organization. You’ll learn how to:

- Create a resilient team ecosystem that leverages AI as a catalyst for productivity rather than simply increasing the volume of new code.
- Identify and overcome the new bottlenecks that could slow down the software delivery lifecycle in an AI-augmented landscape.
- Implement strategic safeguards to embrace the innovative capabilities of Gen AI while mitigating the risks accompanying its adoption.
- Build a clear, panoramic view of Gen AI's role within your projects, ensuring transparency and control over automated code integration.

Prepare to leave this session ready and excited to harness the next generation of generative AI tools armed with the knowledge to maximize impact and mitigate risks.

Speakers

Ben Lloyd Pearson

Director of Developer Relations, LinearB

Ben Lloyd Pearson is a thought leader in developer experience, advocacy, and relations. He is a technology generalist that focuses his broad range of capabilities on growing and engaging developer audiences through digital media, open source advocacy, experiential activations, and... Read More →

Wednesday May 29, 2024 11:30am - 11:55am PDT
AI DevSummit Main Stage

AI DevSummit: Generative AI & LLMs

Talk Type PRO TALK
Track or Conference Generative AI & LLMs, AI DevSummit
In-Person/Virtual In-Person

1:00pm PDT

PRO TALK (AI): How to Ground LLMs for the Enterprise

Alex Kahng, Aisera, Technical Director of AI/ML

The advent of Generative AI holds the promise of revolutionizing entire enterprises and sectors. Large Language Models (LLMs) have revolutionized natural language understanding and generation, offering a wide array of opportunities for enterprise applications.

However, the unbridled potential of LLMs presents unique challenges in adapting them for practical use within the enterprise.

This session explores the critical considerations and strategies for grounding LLMs in the enterprise from prompt engineering to fine-tuning, encompassing issues such as data privacy, model fine-tuning, domain adaptation, and ethical AI practices. By addressing these factors, organizations can leverage the potential of LLMs to enhance factuality, and accuracy, ensuring responsibility and security in their integration across business applications, ranging from AI Copilots to building domain-specific LLMs and their downstream tasks. In this session you will also gain valuable insights into the deployment of LLMs in enterprise settings, ultimately enabling companies to capitalize on the transformative capabilities of these cutting-edge language models.

Speakers

Alex Kahng

Technical Director of AI/ML, Aisera

Wednesday May 29, 2024 1:00pm - 1:25pm PDT
AI DevSummit Main Stage

AI DevSummit: Generative AI & LLMs

Talk Type PRO TALK
Track or Conference Generative AI & LLMs, AI DevSummit
In-Person/Virtual In-Person

1:30pm PDT

PRO TALK (AI): Measuring Accuracy of Your Rag Based LLM System

Venkata Karthik Penikalapati, Salesforce, Lead Member of Technical Staff

The integration of Retrieval-Augmented Generation (RAG) into Large Language Models (LLMs) represents a significant advancement in artificial intelligence, blending the generative prowess of LLMs with the precision of information retrieval to produce contextually relevant and accurate outputs. This talk aims to dissect the methodologies and challenges involved in measuring the accuracy of RAG-based LLM systems, a crucial aspect for their application across diverse domains including, but not limited to, search engines, conversational agents, and domain-specific question-answering systems. We will explore various evaluation metrics and benchmarks that gauge the performance of these systems, touching upon both the generative and retrieval components. Additionally, the presentation will delve into the challenges of defining and measuring "accuracy" in a field where relevance and truthfulness are paramount yet often subjective. By examining case studies and current best practices, this talk will provide insights into strategies for enhancing model performance, alongside a discussion on the ethical considerations of deploying highly accurate RAG systems in sensitive areas. The ultimate goal is to foster a deeper understanding of the state-of-the-art in RAG technology and inspire continuous innovation and responsibility in its application.

Speakers

Venkata Karthik Penikalapati

Lead Member of Technical Staff, Salesforce

Venkata Karthik Penikalapati is a seasoned software developer with over a decade of expertise in designing and managing intricate distributed systems, data pipelines, and ML Ops. Armed with a Master's degree in Computer Science from the University at Buffalo, his knowledge spans the... Read More →

Wednesday May 29, 2024 1:30pm - 1:55pm PDT
AI DevSummit Main Stage

AI DevSummit: Generative AI & LLMs

Talk Type PRO TALK
Track or Conference AI DevSummit, Generative AI & LLMs, MLOps & AIOps
In-Person/Virtual In-Person

3:00pm PDT

PRO TALK (AI): Beyond Vectors: Evolving GenAI through Transformative Tools and Methods

Alison Cossette, Neo4j, Graph Data Science

Embark on a thought-provoking exploration of GenAI's evolution with "Beyond Vectors: Evolving GenAI through Transformative Tools and Methods." Tailored for engineers seeking fresh perspectives, this session encourages practitioners to step beyond familiar Vector Database practices. It's not just a departure; it's a pragmatic leap forward into precision methodologies for data quality and crafting datasets essential for Retrieval-Augmented Generation (RAG) excellence. We'll navigate the complexities of adding non-semantic context through graph databases, shedding light on the nuanced limitations of distance metrics like Cosine Similarity. Join us for this insightful journey, pushing the boundaries of GenAI evolution with transformative tools and methods.

Speakers

Alison Cossette

Developer Advocate, Neo4j

Alison Cossette is a dynamic Data Science Strategist, Educator, and Podcast Host. As a Developer Advocate at Neo4j specializing in Graph Data Science, she brings a wealth of expertise to the field. With her strong technical background and exceptional communication skills, Alison bridges... Read More →

Wednesday May 29, 2024 3:00pm - 3:25pm PDT
AI DevSummit Main Stage

AI DevSummit: Generative AI & LLMs

Talk Type PRO TALK
Track or Conference AI DevSummit, Generative AI & LLMs, Bots & Language Processing
In-Person/Virtual In-Person

3:30pm PDT

PRO TALK (AI): OWASP Top 10 Risks and Mitigations for Secure and Responsible Generative LLM/AI

Scott Clinton, OWASP, Core Team

In October of 2023, The OWASP (Open Web Application Security Project) Foundation launched a new initiative, the OWASP Top 10 for LLMs, Identifying and cataloging the highest risks and mitigations facing organizations deploying LLMs in their application stack. In this session, I'll cover the risks and mitigations in detail and discuss how responsible AI plays a role in helping organizations and developers create a sustainable strategy for securing and guiding responsible considerations when building their LLM/AI applications. We will walk through real-world use cases. I will also preview what the OWASP team is producing for the 2.0 version of the top ten list and how developers should integrate investigation and mitigation steps into their AIOp and Dev workflows.

Speakers

Scott Clinton

Core Team, OWASP

Scott, an industry veteran, has 25+ years of executive and technology leadership, with more than 14+ years in cybersecurity and 12+ years in data analytics and software development, extensive experience leading open-source products, and a Core Team Lead of the OWASP Top 10 for LLM... Read More →

Wednesday May 29, 2024 3:30pm - 3:55pm PDT
AI DevSummit Main Stage

AI DevSummit: Generative AI & LLMs

Talk Type PRO TALK
Track or Conference AI DevSummit, Generative AI & LLMs
In-Person/Virtual In-Person

11:00am PDT

OPEN TALK (AI): How to Use Your Development Data to Make LLMs Code Like You and Your Team

Tyler Dunn, Continue, Co-founder & CEO

Software development copilots are all the rage these days. Innovation in the space makes it easier to talk to your code, ask for improvements, and get code written for you, all from your IDE. But you can make coding with LLMs even more beneficial by collecting data on what happens between commits. Today, all that data goes to the copilot vendor. But what if it went to you and your development org to improve your models, tools, and processes? Learn how Continue is solving this problem by helping developers and engineering teams use their development data

Speakers

Tyler Dunn

Co-founder & CEO, Continue

While studying the intersection of language and computation at the University of Michigan, Ty built dialogue management systems as a software engineer. Motivated to make them leverage machine learning more, he grew from the first product manager to a Group PM at Rasa, where his open-source... Read More →

Thursday May 30, 2024 11:00am - 11:25am PDT
DeveloperWeek Management Expo Stage

AI DevSummit: Generative AI & LLMs

Talk Type OPEN TALK
Track or Conference AI DevSummit, Generative AI & LLMs
In-Person/Virtual In-Person

11:30am PDT

OPEN TALK (AI): Building and Evaluating Robust Conversational Interfaces Using LLMs

Janvi Palan, Samsung Research America, Tech Lead, Senior ML Research Engineer

Dive into the world of crafting resilient conversational interfaces with Large Language Models (LLMs). In the era of advancing technology, the demand for seamless digital interactions has skyrocketed. This talk explores the nuances of using LLMs to build interfaces that not only grasp user input but also deliver contextually relevant and coherent responses.

Key highlights include constructing robust interfaces, tackling challenges like context retention and user engagement. Discover the significance of fine-tuning LLMs for optimal performance in specific conversational domains. Evaluation aspects will emphasize metrics such as user satisfaction, task completion rates, and response coherence. Attendees will gain a comprehensive understanding of constructing and evaluating conversational interfaces, equipping them to navigate the evolving human-computer interaction landscape. Perfect for developers, designers, and industry professionals, this talk offers valuable insights into leveraging LLMs for user-friendly conversational experiences.

Speakers

Janvi Palan

Tech Lead, Senior ML Research Engineer, Samsung Research America

Janvi is a Senior ML research engineer at Samsung Research America. Previously working in Samsung’s artificial humans division, she leads the realistic behavior and conversation generation component of the virtual humans - while ensuring that the quality and latency of interaction... Read More →

Thursday May 30, 2024 11:30am - 11:55am PDT
AI DevSummit Expo Stage

AI DevSummit: Generative AI & LLMs

Talk Type OPEN TALK
Track or Conference AI DevSummit, Generative AI & LLMs, Bots & Language Processing
In-Person/Virtual In-Person

2:30pm PDT

PRO TALK (AI): Supercharge Your Cloud-Native Applications with Generative AI

Cedric Clyburn, Red Hat, Developer Advocate

Generative AI is quickly revolutionizing how we build modern applications, but for developers, it can be daunting, particularly with evaluating models, building with GenAI, and the path to production. But, it doesn’t have to be worrisome! Join us in this session to be ahead of the curve when it comes to AI-enabled cloud-native application development.

Using open-source tools like Podman Desktop and Hugging Face, we’ll show how to practically integrate GenAI in an existing application from your local development environment and ultimately deploy it onto Kubernetes. Why work with local and open-source models? From reducing cloud computing costs, keeping control of your sensitive data, and alleviating vendor-locking, it’s an increasingly popular way for developers to prototype AI applications quickly. We’ll demonstrate the whole AI journey, starting from assessing models, building applications with LLMs, and deploying/serving AI applications.

Speakers

Cedric Clyburn

Developer Advocate, Red Hat

Cedric Clyburn (@cedricclyburn), Developer Advocate at Red Hat, is an enthusiastic software technologist with a background in Kubernetes, DevOps, and container tools. He has experience speaking at conferences and events including DevNexus, WeAreDevelopers, DevConf, and more. Cedric... Read More →

Thursday May 30, 2024 2:30pm - 2:55pm PDT
AI DevSummit Main Stage

AI DevSummit: Generative AI & LLMs

Talk Type PRO TALK
Track or Conference AI DevSummit, Generative AI & LLMs
In-Person/Virtual In-Person

10:30am PDT

[Virtual] OPEN TALK (AI): Solving Enterprise Use Cases through Agents

Chris Chang, Gradient, CEO & Cofounder

Have you ever struggled to make an LLM perform the task that you intend? Providing more raw context is still only a stop gap. Even though new releases of models such as Gemini support context lengths of up to 1M tokens, Retrieval Augmented Generation (RAG) is brittle to the retrieval mechanism and often fails to reconcile between multiple sources of context. The same failure points are amplified when introducing Autonomous Agents due to compounding effects of chaining multiple tasks together.

Learn how to build a reliable production AI Agent System and apply it to high stake industries such as financial services and healthcare. We will deep dive into sentiment analysis as an unexpected and useful showcase of how you can go from a naive system to production level performance through a composable AI system to manage an agent’s memory, connect it to the higher-level concept of sentiment, and utilize reasoning traces for continual performance enhancement.

Speakers

Chris Chang

CEO & Co-Founder, Gradient

Chris is the Co-Founder and CEO at Gradient, a full stack AI platform that enables businesses to build customized agents to power enterprise workloads. Chris most recently led Studio AI at Netflix and was an architect of early domain expert LLMs for content production and intelligence... Read More →

Wednesday June 5, 2024 10:30am - 10:55am PDT
VIRTUAL AI DevSummit Expo Stage

AI DevSummit: Generative AI & LLMs

Talk Type OPEN TALK
Track or Conference AI DevSummit, Generative AI & LLMs, VIRTUAL
In-Person/Virtual Virtual

11:00am PDT

[Virtual] OPEN TALK (AI): Gemini vs. GPT4? The Science of LLM Benchmarks

Philip Tannor, Deepchecks, Co-Founder & CEO
Shir Chorev, Deepchecks, Co-founder & CTO

In this talk, we'll delve into the methods, metrics, and insights behind popular LLM Benchmarks, and learn how to review them effectively. We’ll take a close look at some of the notable leaderboards and LLM benchmarks (such as MMLU, HellSwag, TruthfulQA, MT-Bench) and understand their complexities and uniqueness. Ultimately, we’ll answer the question: did Gemini really outperform GPT4?
Finally, we’ll differentiate between evaluating LLM models to evaluating LLM based applications, and connect the discussion back to practical, real-world applications.

Speakers

Shir Chorev

Co-founder & CTO, Deepchecks

Shir is the co-founder and CTO of Deepchecks, an MLOps startup for continuous validation of ML models and data. Previously, Shir worked at the Prime Minister’s Office and at Unit 8200, conducting and leading research in various Machine Learning and Cybersecurity related challenges... Read More →

Philip Tannor

Co-Founder & CEO, Deepchecks

Philip is the co-founder and CEO of Deepchecks. Philip is an experienced Data Scientist and in thepast, he led a top-tier ML research group that tackled difficult problems from various disciplines(NLP, Computer Vision, Signal Processing, etc).Philip has a B.Sc. in Physics from the... Read More →

Wednesday June 5, 2024 11:00am - 11:25am PDT
VIRTUAL AI DevSummit Expo Stage

AI DevSummit: Generative AI & LLMs

Talk Type OPEN TALK
Track or Conference Generative AI & LLMs, AI DevSummit, MLOps & AIOps, VIRTUAL
In-Person/Virtual Virtual

11:30am PDT

[Virtual] PRO TALK (AI): Are You Gen AI Ready? A Guide to Surviving the Bot Horde

Ben Lloyd Pearson, LinearB, Director of Developer Relations

Generative AI tools have rapidly become commonplace in the typical software engineer’s arsenal, and many feel that we’re on the verge of a paradigm shift in how modern software engineering organizations operate. The next generation of Gen AI tools will reduce or eliminate significant time-consuming work for developers. However, that doesn’t necessarily result in overall productivity gains unless your organization has prepared to use the new tooling.

This session will focus on the connection between human creativity and LLM capabilities and show you how to leverage the new tools for growth while mitigating the risks they introduce to your organization. You’ll learn how to:

- Create a resilient team ecosystem that leverages AI as a catalyst for productivity rather than simply increasing the volume of new code.
- Identify and overcome the new bottlenecks that could slow down the software delivery lifecycle in an AI-augmented landscape.
- Implement strategic safeguards to embrace the innovative capabilities of Gen AI while mitigating the risks accompanying its adoption.
- Build a clear, panoramic view of Gen AI's role within your projects, ensuring transparency and control over automated code integration.

Prepare to leave this session ready and excited to harness the next generation of generative AI tools armed with the knowledge to maximize impact and mitigate risks.

Speakers

Ben Lloyd Pearson

Director of Developer Relations, LinearB

Ben Lloyd Pearson is a thought leader in developer experience, advocacy, and relations. He is a technology generalist that focuses his broad range of capabilities on growing and engaging developer audiences through digital media, open source advocacy, experiential activations, and... Read More →

Wednesday June 5, 2024 11:30am - 11:55am PDT
VIRTUAL AI DevSummit Main Stage

AI DevSummit: Generative AI & LLMs

Talk Type PRO TALK
Track or Conference Generative AI & LLMs, AI DevSummit, VIRTUAL
In-Person/Virtual Virtual

12:00pm PDT

[Virtual] OPEN TALK (AI): Curing Generative AI Hallucinations with Observability

Jemiah Sius, New Relic, Director of Developer Relations

Generative AI applications are oftentimes known to “hallucinate,” or respond with incorrect and inaccurate information. Hallucinations are one of the biggest setbacks to using this tech and are also infamously difficult to prevent, detect, and repair. However, researchers and engineers are racing to solve this problem as the adoption of Gen AI continues to skyrocket.

In the case of a DevOps team adopting this tech, what’s the best way to identify when hallucinations are present, and most importantly, why?

Engineers, and the company’s bottom line, require a way to trace the path of the response—all the way from the user’s prompt and integration with an LLM, to work by agents, API calls, and back to the end user. By monitoring these steps carefully, engineers can learn where hallucinations are developed and apply a potential fix, like including more context to the input or ensuring the LLM is re-summarizing responses accurately, so they can quickly take action and improve the quality of their output.

In this session, Jemiah Sius, Director of Developer Relations at New Relic will dive into how companies can utilize observability to help prevent, detect, and repair GenAI hallucinations and share ways DevOps teams can improve their LLM integrations with context, agents, chained models, and more—with a focus on how to do it in a cost-efficient manner.

In his role, Jemiah is the leader of a global team focused on solving friction points for engineering teams—keeping them at the heart of New Relic’s innovation strategy—so they can find and fix issues more efficiently before they impact their business and customers. With that, he possesses a deep understanding of the challenges DevOps teams face and how, if deployed strategically, GenAI solutions can help solve those challenges.

Speakers

Jemiah Sius

Director of Developer Relations, New Relic

Jemiah Sius is the Director of Developer Relations at New Relic, focused on building a successful community of developers engaged with New Relic. He is also a full-stack engineer with domain knowledge in JavaScript, ES6, Node, React, PHP, and more. Outside of work, Jemiah is a girl... Read More →

Wednesday June 5, 2024 12:00pm - 12:25pm PDT
VIRTUAL AI DevSummit Expo Stage

AI DevSummit: Generative AI & LLMs

Talk Type OPEN TALK
Track or Conference Generative AI & LLMs, AI DevSummit, VIRTUAL, MLOps & AIOps
In-Person/Virtual Virtual

12:30pm PDT

[Virtual] PRO TALK (AI): LangChain Legends: Crafting the LLM Apps of Tomorrow, Today!

Rishab Kumar, Twilio, Developer Evangelist

In the rapidly evolving landscape of Natural Language Processing, Language Models (LLMs) such as GPT-4 have established themselves as cornerstones of many applications, from chatbots to content generators. Yet, the potential of LLMs is vast and largely untapped, especially when it comes to integrating them into comprehensive applications.
That’s where LangChain comes in: an open-source framework that allows to build apps that deeply integrate LLMs. This presentation will guide attendees through the framework, some of the important concepts within LangChain and the immense potential it offers developers.

Speakers

Rishab Kumar

Staff Developer Evangelist, Twilio

Rishab is a Staff Developer Evangelist at Twilio, part-time cloud professor at St. Lawrence College, co-author of Learn to Cloud, AWS Community Builder, and a Microsoft MVP. He is passionate about helping people get into cloud and sharing his learnings from his time in cloud, DevOps... Read More →

Wednesday June 5, 2024 12:30pm - 12:55pm PDT
VIRTUAL AI DevSummit Main Stage

AI DevSummit: Generative AI & LLMs

Talk Type PRO TALK
Track or Conference AI DevSummit, VIRTUAL, Generative AI & LLMs
In-Person/Virtual Virtual

1:00pm PDT

[Virtual] PRO TALK (AI): How to Ground LLMs for the Enterprise

Alex Kahng, Aisera, Technical Director of AI/ML

The advent of Generative AI holds the promise of revolutionizing entire enterprises and sectors. Large Language Models (LLMs) have revolutionized natural language understanding and generation, offering a wide array of opportunities for enterprise applications.

However, the unbridled potential of LLMs presents unique challenges in adapting them for practical use within the enterprise.

This session explores the critical considerations and strategies for grounding LLMs in the enterprise from prompt engineering to fine-tuning, encompassing issues such as data privacy, model fine-tuning, domain adaptation, and ethical AI practices. By addressing these factors, organizations can leverage the potential of LLMs to enhance factuality, and accuracy, ensuring responsibility and security in their integration across business applications, ranging from AI Copilots to building domain-specific LLMs and their downstream tasks. In this session you will also gain valuable insights into the deployment of LLMs in enterprise settings, ultimately enabling companies to capitalize on the transformative capabilities of these cutting-edge language models.

Speakers

Alex Kahng

Technical Director of AI/ML, Aisera

Wednesday June 5, 2024 1:00pm - 1:25pm PDT
VIRTUAL AI DevSummit Main Stage

AI DevSummit: Generative AI & LLMs

Talk Type PRO TALK
Track or Conference Generative AI & LLMs, AI DevSummit, VIRTUAL
In-Person/Virtual Virtual

1:30pm PDT

[Virtual] PRO TALK (AI): Measuring Accuracy of Your Rag Based LLM System

Venkata Karthik Penikalapati, Salesforce, Lead Member of Technical Staff

The integration of Retrieval-Augmented Generation (RAG) into Large Language Models (LLMs) represents a significant advancement in artificial intelligence, blending the generative prowess of LLMs with the precision of information retrieval to produce contextually relevant and accurate outputs. This talk aims to dissect the methodologies and challenges involved in measuring the accuracy of RAG-based LLM systems, a crucial aspect for their application across diverse domains including, but not limited to, search engines, conversational agents, and domain-specific question-answering systems. We will explore various evaluation metrics and benchmarks that gauge the performance of these systems, touching upon both the generative and retrieval components. Additionally, the presentation will delve into the challenges of defining and measuring "accuracy" in a field where relevance and truthfulness are paramount yet often subjective. By examining case studies and current best practices, this talk will provide insights into strategies for enhancing model performance, alongside a discussion on the ethical considerations of deploying highly accurate RAG systems in sensitive areas. The ultimate goal is to foster a deeper understanding of the state-of-the-art in RAG technology and inspire continuous innovation and responsibility in its application.

Speakers

Venkata Karthik Penikalapati

Lead Member of Technical Staff, Salesforce

Venkata Karthik Penikalapati is a seasoned software developer with over a decade of expertise in designing and managing intricate distributed systems, data pipelines, and ML Ops. Armed with a Master's degree in Computer Science from the University at Buffalo, his knowledge spans the... Read More →

Wednesday June 5, 2024 1:30pm - 1:55pm PDT
VIRTUAL AI DevSummit Main Stage

AI DevSummit: Generative AI & LLMs

Talk Type PRO TALK
Track or Conference AI DevSummit, Generative AI & LLMs, MLOps & AIOps, VIRTUAL
In-Person/Virtual Virtual

2:00pm PDT

[Virtual] PRO TALK (AI): Bridging the Gap: Creating Domain-Specific Synthetic Data using Generative AI

Sachin Gupta, Cisco, Lead Data Scientist, Cisco Research

In today's data-driven world, access to high-quality datasets is essential for training and validating machine learning models. However, acquiring real-world data can be costly, time-consuming, and often limited by privacy concerns. In this session, we explore innovative state-of-the-art approaches to create domain-specific synthetic data using generative AI techniques. The session involves demos creating synthetic data in a telemetry domain, how we evaluate the generated data, as well as applications, use cases, and future directions.

Speakers

Sachin Gupta

Lead Data Scientist, Cisco Research, Cisco

Sachin, a visionary leader in Enterprise AI, specializes in Generative AI. With over a decade of experience developing products for Technical Support Services at Microsoft, ServiceNow, and now Cisco, he possesses deep expertise in advanced model architectures and training strategies... Read More →

Wednesday June 5, 2024 2:00pm - 2:25pm PDT
VIRTUAL AI DevSummit Main Stage

AI DevSummit: Generative AI & LLMs

Talk Type PRO TALK
Track or Conference AI DevSummit, Generative AI & LLMs, Applied Machine Learning, VIRTUAL
In-Person/Virtual Virtual

3:00pm PDT

[Virtual] PRO TALK (AI): Beyond Vectors: Evolving GenAI through Transformative Tools and Methods

Alison Cossette, Neo4j, Graph Data Science

Embark on a thought-provoking exploration of GenAI's evolution with "Beyond Vectors: Evolving GenAI through Transformative Tools and Methods." Tailored for engineers seeking fresh perspectives, this session encourages practitioners to step beyond familiar Vector Database practices. It's not just a departure; it's a pragmatic leap forward into precision methodologies for data quality and crafting datasets essential for Retrieval-Augmented Generation (RAG) excellence. We'll navigate the complexities of adding non-semantic context through graph databases, shedding light on the nuanced limitations of distance metrics like Cosine Similarity. Join us for this insightful journey, pushing the boundaries of GenAI evolution with transformative tools and methods.

Speakers

Alison Cossette

Developer Advocate, Neo4j

Alison Cossette is a dynamic Data Science Strategist, Educator, and Podcast Host. As a Developer Advocate at Neo4j specializing in Graph Data Science, she brings a wealth of expertise to the field. With her strong technical background and exceptional communication skills, Alison bridges... Read More →

Wednesday June 5, 2024 3:00pm - 3:25pm PDT
VIRTUAL AI DevSummit Main Stage

AI DevSummit: Generative AI & LLMs

Talk Type PRO TALK
Track or Conference AI DevSummit, Generative AI & LLMs, Bots & Language Processing, VIRTUAL
In-Person/Virtual Virtual

3:30pm PDT

[Virtual] PRO TALK (AI): OWASP Top 10 Risks and Mitigations for Secure and Responsible Generative LLM/AI

Scott Clinton, OWASP, Core Team

In October of 2023, The OWASP (Open Web Application Security Project) Foundation launched a new initiative, the OWASP Top 10 for LLMs, Identifying and cataloging the highest risks and mitigations facing organizations deploying LLMs in their application stack. In this session, I'll cover the risks and mitigations in detail and discuss how responsible AI plays a role in helping organizations and developers create a sustainable strategy for securing and guiding responsible considerations when building their LLM/AI applications. We will walk through real-world use cases. I will also preview what the OWASP team is producing for the 2.0 version of the top ten list and how developers should integrate investigation and mitigation steps into their AIOp and Dev workflows.

Speakers

Scott Clinton

Core Team, OWASP

Scott, an industry veteran, has 25+ years of executive and technology leadership, with more than 14+ years in cybersecurity and 12+ years in data analytics and software development, extensive experience leading open-source products, and a Core Team Lead of the OWASP Top 10 for LLM... Read More →

Wednesday June 5, 2024 3:30pm - 3:55pm PDT
VIRTUAL AI DevSummit Main Stage

AI DevSummit: Generative AI & LLMs

Talk Type PRO TALK
Track or Conference AI DevSummit, Generative AI & LLMs, VIRTUAL
In-Person/Virtual Virtual

10:30am PDT

[Virtual] PRO TALK (AI): Computing Primitive in LLM Space | Building Your Custom AI models

Ahmad Awais, Langbase.com, CEO

"Did we really just do; what we really just did?" was the feeling when I built the first version of a computing primitive in the LLM space. It's called a pipe. Prompt Instructions Personalization Engine. A pipe is a language and framework-agnostic; it's like a serverless function, and it works anywhere with virtually any LLM, like a small mixture of expert models. Simple, right? Not exactly rocket science, not groundbreaking, but oh-so-powerful.
It scales so well, I couldn't believe it myself. 984 million tokens generated in three weeks. It's an undeniably brilliant full-stack AI developer experience.
In this talk, I'll explain why an LLM Pipe is genuinely a winning bet and how it will change the way we build gen-AI use cases with LLMs. I'll also discuss how you can create your own custom AI operating system on top of Langbase.
By the end of this talk, you'll walk away prepared to build your own generative AI LLM web apps in less than five minutes. You'll feel confident building any AI use case, from a simple chatbot like ChatGPT to a complex RAG engine with stellar developer experience (DX). You'll be decisive in crafting apps with lightning-fast iteration velocity and default high performance. Let's go!

Speakers

Ahmad Awais

CEO, Langbase.com

Thursday June 6, 2024 10:30am - 10:55am PDT
AI DevSummit Main Stage

AI DevSummit: Generative AI & LLMs

Talk Type PRO TALK
Track or Conference AI DevSummit, Generative AI & LLMs, VIRTUAL
In-Person/Virtual Virtual

11:00am PDT

[Virtual] OPEN TALK (AI): How to Use Your Development Data to Make LLMs Code Like You and Your Team

Tyler Dunn, Continue, Co-founder & CEO

Software development copilots are all the rage these days. Innovation in the space makes it easier to talk to your code, ask for improvements, and get code written for you, all from your IDE. But you can make coding with LLMs even more beneficial by collecting data on what happens between commits. Today, all that data goes to the copilot vendor. But what if it went to you and your development org to improve your models, tools, and processes? Learn how Continue is solving this problem by helping developers and engineering teams use their development data

Speakers

Tyler Dunn

Co-founder & CEO, Continue

While studying the intersection of language and computation at the University of Michigan, Ty built dialogue management systems as a software engineer. Motivated to make them leverage machine learning more, he grew from the first product manager to a Group PM at Rasa, where his open-source... Read More →

Thursday June 6, 2024 11:00am - 11:25am PDT
VIRTUAL DeveloperWeek Management Expo Stage

AI DevSummit: Generative AI & LLMs

Talk Type OPEN TALK
Track or Conference AI DevSummit, Generative AI & LLMs, VIRTUAL
In-Person/Virtual Virtual

11:30am PDT

[Virtual] OPEN TALK (AI): Building and Evaluating Robust Conversational Interfaces Using LLMs

Janvi Palan, Samsung Research America, Tech Lead, Senior ML Research Engineer

Dive into the world of crafting resilient conversational interfaces with Large Language Models (LLMs). In the era of advancing technology, the demand for seamless digital interactions has skyrocketed. This talk explores the nuances of using LLMs to build interfaces that not only grasp user input but also deliver contextually relevant and coherent responses.

Key highlights include constructing robust interfaces, tackling challenges like context retention and user engagement. Discover the significance of fine-tuning LLMs for optimal performance in specific conversational domains. Evaluation aspects will emphasize metrics such as user satisfaction, task completion rates, and response coherence. Attendees will gain a comprehensive understanding of constructing and evaluating conversational interfaces, equipping them to navigate the evolving human-computer interaction landscape. Perfect for developers, designers, and industry professionals, this talk offers valuable insights into leveraging LLMs for user-friendly conversational experiences.

Speakers

Janvi Palan

Tech Lead, Senior ML Research Engineer, Samsung Research America

Janvi is a Senior ML research engineer at Samsung Research America. Previously working in Samsung’s artificial humans division, she leads the realistic behavior and conversation generation component of the virtual humans - while ensuring that the quality and latency of interaction... Read More →

Thursday June 6, 2024 11:30am - 11:55am PDT
VIRTUAL AI DevSummit Expo Stage

AI DevSummit: Generative AI & LLMs

Talk Type OPEN TALK
Track or Conference AI DevSummit, Generative AI & LLMs, Bots & Language Processing, VIRTUAL
In-Person/Virtual Virtual

2:30pm PDT

[Virtual] PRO TALK (AI): Supercharge Your Cloud-Native Applications with Generative AI

Cedric Clyburn, Red Hat, Developer Advocate

Generative AI is quickly revolutionizing how we build modern applications, but for developers, it can be daunting, particularly with evaluating models, building with GenAI, and the path to production. But, it doesn’t have to be worrisome! Join us in this session to be ahead of the curve when it comes to AI-enabled cloud-native application development.

Using open-source tools like Podman Desktop and Hugging Face, we’ll show how to practically integrate GenAI in an existing application from your local development environment and ultimately deploy it onto Kubernetes. Why work with local and open-source models? From reducing cloud computing costs, keeping control of your sensitive data, and alleviating vendor-locking, it’s an increasingly popular way for developers to prototype AI applications quickly. We’ll demonstrate the whole AI journey, starting from assessing models, building applications with LLMs, and deploying/serving AI applications.

Speakers

Cedric Clyburn

Developer Advocate, Red Hat

Cedric Clyburn (@cedricclyburn), Developer Advocate at Red Hat, is an enthusiastic software technologist with a background in Kubernetes, DevOps, and container tools. He has experience speaking at conferences and events including DevNexus, WeAreDevelopers, DevConf, and more. Cedric... Read More →

Thursday June 6, 2024 2:30pm - 2:55pm PDT
VIRTUAL AI DevSummit Main Stage

AI DevSummit: Generative AI & LLMs

Talk Type PRO TALK
Track or Conference AI DevSummit, Generative AI & LLMs, VIRTUAL
In-Person/Virtual Virtual