A framework to evaluate and establish trust in generative AI, in learning and development (L&D) applications

点击翻译按钮获取中文摘要

Today, large generative models can take simple instructions from their users and generate artefacts such as code, text, images and videos that until recently, could only be produced by humans. The success of products that rely on such models, however, requires their users/stakeholders to trust them. This is particularly important in domains such as education, finance, and medicine where the stakes are high.We are building Elandi, a product that uses the power of the latest developments in AI (particularly, generative AI) to bring affordable personalised L&D to everyone:1. Elandi will compare a user's profile/resume with their career goals (e.g., a job description) and identify the gaps.2. Using this gap (plus various user preferences, and more), it generates a prompt for the generative AI.3. The generative AI will then provide the user with the right L&D content.The UX areas we are currently exploring are course planning, bite-sized tutoring, assessment, and content generation.Thanks to our successful courses since 2020, we now have a unique user community (1000s of AI talents from 100+ countries) that can the source of data and advice. Furthermore, we worked with top AI professors/lecturers, and some of the world's largest corporations in real-world AI applications. All these helped us make tremendous progress towards Elandi's product-market fit (PMF); from discovery, to conceptual design and ML prototypes. Overall, the most important learning from our discovery is that fit-for-purpose content, improved UX, personalised recommendation, and expert mentoring are in high demand.While this progress continues, we realised that there is an urgent need for a comprehensive framework for AI assurance (e.g., auditing the AI, measuring users' trust in it, and quantifying its various risks) in the L&D space. Therefore, as part of our broader ambition, through this grant we would like to kickstart an explicit work stream on this topic. Our goal is to build on the latest research (including metrics and measurements) on trustworthy AI, to provide the tools and frameworks necessary for auditing AI models and products in the L&D space. While we will be focused on the L&D use cases, we expect our results to have implications beyond L&D and in other domains..An improved L&D offering will be of profound societal and commercial value; it will play a critical role in improving national and international productivity. It is aligned with strategic government initiatives such as "Levelling up", "Lifelong Learning", and "Skills for Life".