I work on LLMs and LLM infra at Mozilla.ai.
Lumigator 🐊 is a self-hosted, open-source Python application for evaluating large language models using offline metrics. It targets common machine learning use-cases, starting with summarization, and is extensible at the task and job level.
Like our parent company, Mozilla.ai’s founding story is rooted in open-source principles and community collaboration. Since our start last year, our key focus has been exploring state-of-the-art methods for evaluating and fine-tuning large-language models (LLMs), including continued open-source LLM evaluation experiments and establishing our GPU cluster’s infrastructure.
After a year of breakneck innovation and hype in the AI space, we have now moved sufficiently beyond the peak of the hype cycle to start asking a critical question: are LLMs good enough yet to solve all of the business and societal challenges we are setting them up for?