Skip to main content
UploadLanguage (EN)Support
BusinessMobileSocial MediaMarketingTechnologyArt & PhotosCareerDesignEducationPresentations & Public SpeakingGovernment & NonprofitHealthcareInternetLawLeadership & ManagementAutomotiveEngineeringSoftwareRecruiting & HRRetailSalesServicesScienceSmall Business & EntrepreneurshipFoodEnvironmentEconomy & FinanceData & AnalyticsInvestor RelationsSportsSpiritualNews & PoliticsTravelSelf ImprovementReal EstateEntertainment & HumorHealth & MedicineDevices & HardwareLifestyle
MA
Uploaded byMohammad Aliannejadi
PPT, PDF473 views

Asking Clarifying Questions in Open-Domain Information-Seeking Conversations

The document discusses the importance of asking clarifying questions in open-domain information-seeking conversations to improve retrieval performance. It outlines methodologies for offline evaluation of clarifying questions, focusing on precision and recall, while introducing the QULAC framework for evaluating question retrieval and selection. The findings suggest that asking a single effective clarifying question can lead to significant improvements in information retrieval outcomes.

Presentations & Public Speaking◦
Download to read offline
Asking Clarifying Questions in Open-Domain Information- Seeking Conversations Mohammad Aliannejadi(1), Hamed Zamani(2), Fabio Crestani(1), and W. Bruce Croft(2) (1) Università della Svizzera italiana (USI), Switzerland (2) University of Massachusetts Amherst, USA
© @dawnieando; @JeffD
5
6
7
8
Can we ask questions to clarify the user information needs? Johannes Kiesel et al. Toward Voice Query Clarification. SIGIR 2018 Radlinski and Craswell. A Theoretical Framework for Conversational Search. CHIIR 2017
How to evaluate?
ClueWeb Collection • A part of the Lemur Project • A common web crawl (English) with 50M documents • TREC Web Track 2009 – 2012 • Ad-hoc retrieval and diversification tasks
TREC facets
An offline evaluation methodology • We assume that each user is interested in one facet per topic.
An offline evaluation methodology • Let be the set of topics (queries). • A collection of facet sets: • includes all defined facets for topic . • A collection of clarifying question sets: • With including all clarifying questions relevant to topic . • An offline evaluation requires defining . • Borrowed from the ClueWeb Collection
Question Verification and Facet Linking • Two main concerns: • Precision: how is the quality of the collected clarifying questions? • Recall: are all facets addressed by at least one clarifying question? • Two expert annotators: • Marked invalid and duplicate questions. • Linked questions to the facets they found relevant. • Facets with no questions: generated new questions relevant to them.
An offline evaluation methodology • Let be the set of topics (queries). • A collection of facet sets: • includes all defined facets for topic . • A collection of clarifying question sets: • With including all clarifying questions relevant to topic . • An offline evaluation requires defining . • Borrowed from the ClueWeb Collection
Quality Check • Regular quality checks on the collected answers. • Manual checks on 10% of submissions per worker. • If any invalid answer was observed, we then checked all the submissions of the corresponding worker. • Invalid answers were removed and workers banned from future tasks. • Disabled copy/paste feature. • Monitored keystrokes.
Qulac: Questions for Lack of Clarity Qulac has two meanings in Persian: • blizzard • wonderful or masterpiece © HBO
Learning to ask clarifying questions
Question Retrieval
Question Retrieval • Task: Given a topic and a context (question-answer history), retrieve clarifying questions. • Desired objective: high recall • Approaches: • Term matching retrieval models: language models, BM25, RM3 (query expansion) • Learning to rank: LambdaMART, RankNet, neural ranking models (e.g., BERT)
Question Retrieval
Question Retrieval
Question Selection • Task: selecting a clarifying question that leads to retrieval improvement • Objective: high precision (in retrieval) • Approaches: • Query performance prediction (QPP): predicting the retrieval performance after asking each question (without answer) and selecting the one with the highest QPP. • Learning to rank: defining a set of features for ranking questions. The features include QPP, similarity to the topic, similarity to the context, etc. • Neural ranking models: learning to rank with representation learning (e.g., BERT)
Question Selection Asking only one good question improves the performance by over 100%.
Case Study Negative answer; new information. Retrieval model fails.
Case Study Open question; new information.
Future Directions • Utilizing positive and negative feedback for document retrieval. • Joint modeling of question retrieval and selection. • Question generation. • Determining the number of questions to ask based on the system’s confidence. • Explore other ways of evaluating a system: • Conversation turns; • Retrieval performance.
Conclusions • Asking clarifying questions in open-domain information-seeking conversations. • Qulac: a collection for automatic offline evaluation of asking clarifying questions for conversational IR. • A simple yet effective retrieval framework. • Asking only one good question improves the performance by over 100%! • More improvement for: • Shorter queries; • Ambiguous queries.
Questions? Qulac is publicly available at http://bit.ly/QulacData Thanks to SIGIR for the student travel grant!
Ad