An MVP for Simplifying AI Model Selection

In today’s fast-moving AI landscape, choosing the right large language model (LLM) for your project can feel like navigating a maze. With hundreds of models, each offering different capabilities, the process can be overwhelming. That’s why Mozilla.ai is developing Lumigator, a product designed to help developers confidently select the best LLM for their specific project. It's like having a trusty compass for your AI journey.

The Problem (And Why We’re Tackling It)

As more organizations turn to AI for solutions, they face the challenge of selecting the best model from an ever-growing list of options. The AI landscape is evolving rapidly, with twice as many new models released in 2023 compared to the previous year. Yet, in spite of the wealth of metrics available, there’s still no standard way to compare these models. 

The 2024 AI Index Report highlighted that AI evaluation tools aren’t (yet) keeping up with the pace of development, making it harder for developers and businesses to make informed choices. Without a clear single method for comparing models, many teams end up using suboptimal solutions, or just choosing models based on hype, slowing down product progress and innovation.

Our Mission (And How We’re Getting Started)

With Lumigator MVP, Mozilla.ai aims to make model selection transparent, efficient, and empowering. Lumigator provides a framework for comparing LLMs, using task-specific metrics to evaluate how well a model fits your project’s needs. With Lumigator, we want to ensure that you’re not just picking a model—you’re picking the right model for your use case.

Why We Started with Model Evaluation and Text Summarization

Model selection is only the beginning. While our MVP focuses on making it easier for developers to evaluate and choose models, Lumigator’s long-term vision stretches much further. We’re starting with model evaluation for the specific task of text summarization because it’s a fairly common use-case across industries. Whether you’re working in finance, healthcare, or tech, summarizing data is a critical task but we are betting on making the product extendable to further use cases and metrics. 

What Lumigator aims at Offering

Our product takes the guesswork out of model selection by simplifying every step of the process:

1. Seamless Setup and Integration
   1. Easily install all the required dependencies and set up your local development environment. 
2. API Access for Ultimate Flexibility
   1. Start Lumigator and gain full API access. Perfect for developers who want to plug into their existing workflows and scale model evaluation for summarization.
3. Automatic ground truth generation 
   1. Recognizing the critical role of ground truth generation, Lumigator simplifies the process—whether you're working with existing ground truth data or need Lumigator to generate it automatically. We can provide the ground truth generation.
4. Model Discovery and Experiment Creation
   1. Choose from a curated set of models suited for diverse datasets and goals. Lumigator will allow you to create experiments to evaluate different models and parameters on the same dataset.
5. Real-Time Experiment Monitoring
   1. Track the performance of your experiments live via the Dashboard and get instant access to the status of your jobs and outputs. 
6. Comprehensive Results Evaluation
   1. Retrieve and download detailed experiment results in JSON format. Lumigator automatically chooses the right metrics for your task (e.g. Rouge, Meteor, and BertScore for summarization) and shows how models score in relation to them, providing a comprehensive view of their performance. 
7. Advanced Model Comparison
   1. Run multiple models on the same dataset, compare their performance, and make data-driven decisions. Lumigator simplifies cross-model comparisons so you can find the best model for your needs.
8. Iterative Experimentation
   1. Refine your approach with Lumigator's flexible experimentation tools. Test different models or experiment with new datasets, and track every iteration for maximum optimization.
9. Extensibility
   1. With Lumigator, you will have the flexibility to extend the product by adding your own custom evaluations and tasks, seamlessly integrating them into our API for easy access and deployment.

The application will include a UI and an SDK that simplifies the technical complexity, allowing you to focus on results.

Building in the Open

Lumigator is still in its early days, so we ask for your patience and understanding as we grow. Our GitHub repo is live here, and we’re building the project out in the open. Like any newborn project, we’ve got a lot of growing to do, and we invite you to follow along and contribute by participating in our discovery and research activities by signing-up to our Lumigator updates.

We know building in the open isn’t always easy—it’s like cooking in an open kitchen at a restaurant. Everyone can see the mess and mistakes as they happen. But we believe that with the help of the community, we can shape Lumigator beyond a model selection tool. It will become a comprehensive product that makes AI ethical, trustworthy, and accessible to all. Plus, it’s always more fun when you’re part of the process, right?

Our Vision for the Future

In the future, Lumigator will grow beyond evaluation into a full-blown open-source product for ethical and transparent AI development and fill in gaps in the AI development tooling landscape in the industry. We want to create a space where developers can trust the tools they use, knowing they’re building solutions that align with their values.

Our MVP is just the start. While we’re focused on model selection now, we’re building towards something much bigger. Lumigator’s ultimate goal is to become the go-to open-source platform for developers who want to make sure they’re using AI in a way that is transparent, ethical, and aligned with their values. With the input of the community, we’ll continue to expand beyond evaluation and text summarization into all aspects of AI development. Together, we’ll shape Lumigator into a tool that you can trust.

With Lumigator, we want to democratize AI. What do we mean by this? We want to make advanced technologies available to both developers and to organizations of all sizes. Our mission is to enable people to build solutions that leverage AI to align with their goals and values—whether it’s fostering transparency, driving innovation, or creating a more inclusive future for AI.

Want to Help Make Lumigator Even Better?

We're shaping Lumigator into a product that meets the real needs of developers, and we need your input! We're conducting a survey to gather insights on the infrastructure you're using and the challenges you're facing in AI development. Your feedback will directly influence our work.

👉 Take the anonymous survey here to contribute! It takes less than 5 minutes.

We're planning invite-only Alpha previews starting in November 2024, with general availability launching in January 2025 🚀. To stay in the loop on future releases and get involved early, sign up for updates and join our Discord community.