Zhen Jia
Rishiraj Saha Roy
ABSTRACT
Answering complex questions is one of the challenges that question-answering (QA) systems face today. While complexity has several facets, question dimensions like temporal and spatial intents necessitate specialized treatment. Methods geared towards such questions need benchmarks that reflect the desired aspects and challenges. Here, we take a key step in this direction, and release a new benchmark, TempQuestions, containing 1,271 questions, that are all temporal in nature, paired with their answers. As a key contribution that enabled the creation of this resource, we provide a crisp definition for temporal questions. Most questions require decomposing them into sub-questions, and the questions are of a kind that they would be best evaluated on a combination of structured data and unstructured text sources. Experiments with two QA systems demonstrate the need for further research on complex questions.
- Abdalghani Abujabal, Mohamed Yahya, Mirek Riedewald, and Gerhard Weikum. 2017. Automated Template Generation for Question Answering over Knowledge Graphs WWW. Google Scholar
Digital Library
- Eugene Agichtein, David Carmel, Dan Pelleg, Yuval Pinter, and Donna Harman. 2015. Overview of the TREC 2015 LiveQA Track. In TREC.Google Scholar
- James F. Allen. 1983. Maintaining Knowledge About Temporal Intervals. Comm. ACM (1983). Google ScholarDigital Library
- Omar Alonso, Michael Gertz, and Ricardo Baeza-Yates. 2007. On the value of temporal information in information retrieval ACM SIGIR Forum. Google ScholarDigital Library
- Junwei Bao, Nan Duan, Zhao Yan, Ming Zhou, and Tiejun Zhao. 2016. Constraint-Based Question Answering with Knowledge Graph COLING.Google Scholar
- Hannah Bast and Elmar Haussmann. 2015. More Accurate Question Answering on Freebase. In CIKM. Google ScholarDigital Library
- Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic Parsing on Freebase from Question-Answer Pairs EMNLP.Google Scholar
- Branimir Boguraev, Siddharth Patwardhan, Aditya Kalyanpur, Jennifer Chu-Carroll, and Adam Lally. 2014. Parallel and nested decomposition for factoid questions. Natural Language Engineering (2014).Google Scholar
- Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks. arXiv (2015).Google Scholar
- Qingqing Cai and Alexander Yates. 2013. Large-scale Semantic Parsing via Schema Matching and Lexicon Extension ACL.Google Scholar
- Angel X. Chang and Christopher D. Manning. 2012. SUTime: A library for recognizing and normalizing time expressions LREC.Google Scholar
- Dennis Diefenbach, Vanessa Lopez, Kamal Singh, and Pierre Maret. 2017. Core techniques of question answering systems over knowledge bases: A survey Knowledge and Information systems. Google ScholarDigital Library
- Aditya Kalyanpur et al. 2012 a. Structured data and inference in DeepQA. IBM Journal of Research and Development (2012). Google ScholarDigital Library
- David A. Ferrucci et al. 2012 b. This is Watson. IBM Journal of Research and Development Vol. 56 (2012). Issue 3/4. Google ScholarDigital Library
- Aditya Kalyanpur, Siddharth Patwardhan, BK Boguraev, Adam Lally, and Jennifer Chu-Carroll. 2012. Fact-based question decomposition in DeepQA. IBM Journal of Research and Development (2012). Google ScholarDigital Library
- Erdal Kuzey, Vinay Setty, Jannik Strötgen, and Gerhard Weikum. 2016. As Time Goes By: Comprehensive Tagging of Textual Phrases with Temporal Scopes WWW. Google ScholarDigital Library
- Ken Litkowski. 2014. Pattern Dictionary of English Prepositions. In ACL.Google Scholar
- Ken Litkowski and Orin Hargraves. 2006. Coverage and inheritance in the preposition project SIGSEM. Google ScholarDigital Library
- Christopher D. Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven J. Bethard, and David McClosky. 2014. The Stanford CoreNLP Natural Language Processing Toolkit ACL.Google Scholar
- Donald Metzler, Rosie Jones, Fuchun Peng, and Ruiqiang Zhang. 2009. Improving Search Relevance for Implicitly Temporal Queries SIGIR. Google ScholarDigital Library
- Anselmo Pe nas, Christina Unger, Georgios Paliouras, and Ioannis Kakadiaris. 2015. Overview of the CLEF Question Answering Track 2015 CLEF.Google Scholar
- James Pustejovsky, Robert Knippen, Jessica Littman, and Roser Saurí. 2005. Temporal and Event Information in Natural Language Text LREC.Google Scholar
- Deepak Ravichandran and Eduard Hovy. 2002. Learning surface text patterns for a question answering system ACL. Google ScholarDigital Library
- Swarnadeep Saha, Harinder Pal, and Mausam. 2017. Bootstrapping for Numerical Open IE. In ACL.Google Scholar
- Andrea Setzer. 2002. Temporal information in newswire articles: An annotation scheme and corpus study. Ph.D. Dissertation. University of Sheffield.Google Scholar
- Jannik Strötgen and Michael Gertz. 2015. A Baseline Temporal Tagger for all Languages. In EMNLP.Google Scholar
- Jannik Strötgen and Michael Gertz. 2016. Domain-sensitive Temporal Tagging. Morgan & Claypool Publishers.Google Scholar
- Priyansh Trivedi, Gaurav Maheshwari, Mohnish Dubey, and Jens Lehmann. 2017. LC-QuAD: A Corpus for Complex Question Answering over Knowledge Graphs ISWC.Google Scholar
- Christina Unger, Corina Forascu, Vanessa López, Axel-Cyrille Ngonga Ngomo, Elena Cabrio, Philipp Cimiano, and Sebastian Walter. 2015. Question Answering over Linked Data (QALD-5). In CLEF.Google Scholar
- Christina Unger, André Freitas, and Philipp Cimiano. 2014. An introduction to question answering over linked data Reasoning Web.Google Scholar
- Ricardo Usbeck, Axel-Cyrille Ngonga Ngomo, Bastian Haarmann, Anastasia Krithara, Michael Röder, and Giulio Napolitano. 2017. 7th Open Challenge on Question Answering over Linked Data (QALD-7) SemWebEval.Google Scholar
- Ellen M. Voorhees. 2010. Reflections on TREC QA. In CLEF.Google Scholar
- Mohamed Yahya, Klaus Berberich, Shady Elbassuoni, Maya Ramanath, Volker Tresp, and Gerhard Weikum. 2012. Natural language questions for the Web of data. In EMNLP. Google ScholarDigital Library
- Wen-tau Yih, Ming-Wei Chang, Xiaodong He, and Jianfeng Gao. 2015. Semantic Parsing via Staged Query Graph Generation: Question Answering with Knowledge Base. In ACL.Google Scholar
- Pengcheng Yin, Nan Duan, Ben Kao, Jun-Wei Bao, and Ming Zhou. 2015. Answering Questions with Complex Semantic Constraints on Open Knowledge Bases CIKM. Google ScholarDigital Library
Index Terms
- TempQuestions: A Benchmark for Temporal Question Answering
Recommendations
TEQUILA: Temporal Question Answering over Knowledge Bases
CIKM '18: Proceedings of the 27th ACM International Conference on Information and Knowledge ManagementQuestion answering over knowledge bases (KB-QA) poses challenges in handling complex questions that need to be decomposed into sub-questions. An important case, addressed here, is that of temporal questions, where cues for temporal relations need to be ...
TIQ: A Benchmark for Temporal Question Answering with Implicit Time Constraints
WWW '24: Companion Proceedings of the ACM on Web Conference 2024Temporal question answering (QA) involves explicit (e.g., "...before 2024") or implicit (e.g., "...during the Cold War period") time constraints. Implicit constraints are more challenging; yet benchmarks for temporal QA largely disregard such questions. ...
Faithful Temporal Question Answering over Heterogeneous Sources
WWW '24: Proceedings of the ACM on Web Conference 2024Temporal question answering (QA) involves time constraints, with phrases such as "... in 2019" or "... before COVID". In the former, time is an explicit condition, in the latter it is implicit. State-of-the-art methods have limitations along three ...
Comments