Collective Intelligence Project (CIP) has been hard at work on bringing collective intelligence into AI evaluation. Over the past few months, we have fleshed out our participatory methodology and the Weval platform to elicit expertise from domain experts that we can translate into criteria for higher-quality evals. Through partnerships with coalitions and civil society, we are focused on publishing social impact evals that measure the performance of AI in fields with large-scale implications for society.
Below, Moira Whelan and Jesper Frant of Our Secure Future (OSF) detail how they used Weval to develop a benchmark on the ability of LLMs and AI tools to follow Women, Peace and Security (WPS) principles in global peacekeeping.
