From Shared Tasks to Benchmark Datasets:
Lessons from NTCIR’s 26-Year Journey

This half-day tutorial will introduce the history and experiences of the NTCIR evaluation series (NII Testbeds and Community for Information access Research) from its inception in 1999 to the present. We will focus on how NTCIR pioneered multilingual information access evaluation, the challenges of collecting benchmark datasets (e.g. for Japanese, Chinese, and other languages) and how these challenges were addressed.

The tutorial will be Introductory to adjacent areas – providing an overview from the beginning until now – so no prior knowledge of NTCIR is required. We will cover the evolution of NTCIR’s shared tasks, the methods used to gather and curate evaluation corpora, and lessons learned in organizing international benchmark test collections.

Attendees will gain insight into NTCIR’s contributions to information retrieval (IR) and natural language processing (NLP) research – from cross-lingual search to question answering and beyond – and understand best practices for creating and sharing evaluation resources. The target audience includes researchers and practitioners interested in language resources and evaluation campaigns. By the end of the tutorial, participants will have a clear understanding of NTCIR’s legacy, the practical aspects of running collaborative evaluations, and open challenges for future multilingual evaluation initiatives.