Abstracts parallel session 5

Question-Answering using a Constraint Relaxation Mechanism

Modeling Phonotactic Cues for Speech Segmentation

Document segmentation for passage retrieval in question answering

Lexical Patterns or Dependency Patterns: Which is better?

Automatic Thesaurus Extraction: Comparing Context Models.

The relevance of Natural Language {website|internet|enterprise} Search

Automatic Acquisition of Lexico-Semantic Information for Question Answering.

Automatic measure of speech rate in spoken Dutch

Named Entities in a Question matching environment

Paragraph Retrieval for why-question answering

Home Call for abstracts Abstract submission Important dates Location Programme Registration Proceedings Local Organization Sponsors FAQ Pictures

CLIN 2007

Friday December 7, 2007
University of Nijmegen

CLIN 2007 is organized by the Language and Speech group of the Radboud University Nijmegen.

Question-Answering using a Constraint Relaxation Mechanism
Modeling Phonotactic Cues for Speech Segmentation
Document segmentation for passage retrieval in question answering
Lexical Patterns or Dependency Patterns: Which is better?
Automatic Thesaurus Extraction: Comparing Context Models.
The relevance of Natural Language {website|internet|enterprise} Search
Automatic Acquisition of Lexico-Semantic Information for Question Answering.
Automatic measure of speech rate in spoken Dutch
Named Entities in a Question matching environment
Paragraph Retrieval for why-question answering

Ana Mendes, Luisa Coheur, Nuno J. Mamede (L2F - INESC-ID)

With the growth of the available digital information, as well as the increasing amount of people accessing it through their cell phones, PDAs and web browsers, the need for Question/Answering (QA) systems has become more acute than ever.
Evaluation forums (such as TREC and CLEF) with QA tasks, aim to improve the research in this field. Participating systems present different methods to pre-process provided corpora as well as in question interpretation: some perform named entities recognition (NER) and deep linguistic analysis; others profit from thesaurus and ontologies or exploit machine learning techniques.
QA@L2F is based on a deep semantic analysis and relies on NER, both in corpora pre-processing and question interpretation. Also the identification of the question type triggers distinct answer extraction processes, namely morpho-syntactic pattern matching, linguistic reordering, named entities matching and brute force plus natural language processing.
Although the system is prepared to apply different strategies to different types of questions, its main feature is to have a constraint relaxation mechanism, which allows it to switch among these strategies in order to return the question's answer: if a precise linguistic analysis on the question is not possible, or if no answer is found in the database, the system relaxes and tries to find an answer in a more flexible and less constrained way.
The paper reports on the results obtained by following the different strategies on the different types of questions.

Frans Adriaans (Utrecht University)

One of the tasks that infants are facing in language acquisition is to extract words from continuous speech. Psycholinguistic studies have shown the relevance of phonotactic cues in speech segmentation. Phonotactic cues tell the learner which phoneme sequences are likely or unlikely to occur in the language, where unlikely sequences indicate the presence of a word boundary. In this definition, the term phonotactic cue denotes both statistical cues (transitional probability, mutual information) and abstract phonotactic constraints.
It is not known exactly how phonotactic cues are used to locate word boundaries. Saffran et al. (1996) show that subjects use transitional probabilities, calculated from continuous speech, to distinguish between words and nonwords. Nevertheless, they do not explicitly describe how segmentation proceeds. Brent (1999) implements phonotactic cues in a local minimum approach, using a 4-phoneme window. However, this window size is rather arbitrary. Rytting (2004) argues for a global segmentation threshold, but provides no solution for selecting the right threshold. Moreover, none of these models have considered the possibility that phonotactic cues might take on a more abstract form. Phonological features, for example, can be used to create abstract phonotactic constraints.
In this talk, I will argue that in the framework of Optimality Theory (OT; Prince and Smolensky, 1993) several modeling issues fall into place, since OT by definition integrates the global threshold and local minimum approaches. More importantly, segmentation simulations indicate that our OT model, which uses induced constraints with different levels of abstraction, outperforms purely statistical phonotactic models.

Jörg Tiedemann and Jori Mur (Rijksuniversiteit Groningen)

Passage retrieval is used in question answering to narrow down the search for answers to a given natural language question. In our research we address the problem of document segmentation into appropriate passages, i.e., text units which are large enough for standard IR techniques and small enough for efficient answer extraction. In our study we compare four different segmentation approaches with a baseline of paragraph retrieval using existing markup: (1) We split documents into fixed-size passages. (2) We used a fixed-size sliding window approach (3). We used automatically detected coreference chains to create passages with some semantic coherence (using our own Dutch coreference resolution system). (4) We applied M. Hearst's TextTiling algorithm for detecting subtopic structures.
In this way we compare segmentation strategies for creating linguistically motivated units (using semantic information such as in 3 and 4 or discourse information as in our baseline) with simple window-based techniques.
In our experiments we used data from the Dutch QA tracks at CLEF from previous years (2003-2005). Altogether our test set includes 777 questions with their answers which we used to measure both, passage retrieval performance and question-answering performance. Surprisingly, simple window-based techniques work very well. Details will be discussed in the presentation.

Katja Hofmann and Erik Tjong Kim Sang (Informatics Institute, University of Amsterdam)

This paper is the result of year-long debate between the authors about the value of lexical patterns versus dependency patterns for natural language processing tasks. Lexical patterns are easy to obtain and can be applied to incomplete utterances. Dependency patterns obtain better coverage at the price of robustness and the introduction of parsing errors. For this study we apply both lexical patterns and dependency patterns to the extraction of hypernyms from newspaper texts and the Dutch Wikipedia. By performing a thorough evaluation of the results, we hope to resolve the debate, at least for this NLP task.

Kris Heylen, Yves Peirsman (K.U.Leuven)

Distributional, corpus based approaches to semantic similarity come in many different flavours. To find the semantically most related words of a target word, some models merely take into account context words to the left and right (so called “bag-of-words” models), whereas others look at the 2nd order context words (the context words of the context words), and still others use the syntactic dependency relations that a target word occurs in. Although the output of these different models diverges widely, not much is known about the linguistic consequences of the choice of context model.
In this paper, we assess the linguistic properties of the “semantic” similarity that is catputred by different context models. What are the differences between bag-of-words and syntactic dependency models? What happens if we combine both? To what extent does context size have an influence? What are the effects of dimensionality reduction?
As a Gold Standard we use EuroWordNet Dutch, a lexical database of Dutch words and their semantic interrelations (synonymy, hypernomy, hyponymy and co-hyponymy). For a test set of 10,000 Dutch nouns, we investigate which semantic relations are found among the 10 most related words retrieved using different context models. Results show that syntactic dependency models retrieve significantly more synonyms than bag-of-words models and that dimensionality reduction leads to a bias towards co-hyponyms.
This research does not only give more insight in the linguistic characteristics of the computational models; it also helps to decide which models are best suited for specific NLP tasks.

Leonoor van der Beeek (Q-go)

Q-go performs deep linguistic analysis on user questions on large corporate websites or intranets, extracts the user intent, and serves a small set of relevant results that match this intent. The core of this application is natural language technology, including language-, branch-, and client-specific dictionaries, ontologies, and grammars, as well as a matching algorithm.
Recently, Q-go has extended the application domain of that technology to include not only website search, but also enterprise and internet search. We have developed two demo systems: the first integrates Q-go Natural Language Search in MOSS 2007 (Sharepoint) search, the second combines Q-go search with the Google Search Application.
The integrated Sharepoint/Q-go search functionality, like our customer interaction management application, extracts the user intent from the question. This user intent is then translated into a Sharepoint search query, using the metatags that are inherent to the system. With this query, answers may be retrieved from all types of content, including relational databases. The combined search offers a better user experience, as users can ask questions in their own words, but it also provides more relevant answers and it answers questions that standard Sharepoint search cannot answer.
The combination of Q-go and Google Search Application is a straightforward one from a technical point of view: the query is sent to both systems, and if Q-go can find one or more matching answers in its own database, they are presented to the user in combined, more relevant set of results.

Lonneke van der Plas (University of Groningen)

Lexico-semantic information, such as synonyms (e.g., autumn-fall) and categorised named entities (e.g., Rafael Nadal is a tennis player) can be very helpful in question answering (QA). For example, when a user asks for 'the tennis player that lost the finals of Wimbledon in 2006', it is helpful to have a list of tennis players. Candidate named entities found by the QA system, that are not in the list of tennis players, such as Miguel Ãngel (Rafael's uncle and retired football player) can be easily filtered out. I want to briefly describe the components of our QA system Joost and show how several types of lexico-semantic information can be successfully applied to several system components. Finally, I will explain how we acquired the lexico-semantic information automatically.

Nivja H. de Jong & Ton Wempe (Amsterdam Centre for Language and Communication, University of Amsterdam)

Learning to become fluent in a second language is one of the most difficult aspects of learning a second language. Correlations between subjective measures of fluency and objective measures of fluency have shown that speech rate is one of the most important measures of temporal fluency (Cucchiarini, Strik, & Boves, 2002). However, counting syllables is a tedious job and is often omitted from data analyses due to time constraints. In the context of a large-scale research project on the correlates of speaking proficiency carried out at the University of Amsterdam (What is Speaking Proficiency: http://www.hum.uva.nl/wisp), we developed a tool to measure speech rate automatically.
In this paper, we describe this method to detect syllables in speech without need of a transcription. A script written in PRAAT measures intensity (dB) and syllables are subsequently detected as peaks in intensity above a certain threshold, surrounded by dips in intensity. These peaks are then identified as syllables if the sound is voiced. Testing this script in two corpora of spoken Dutch, we find high correlations between manually counted speech rate and automatically measured speech rate. We conclude that the automatic detection of syllables suffice to reliably calculate and compare speech rates between participants and tasks. Finally, we compare our speech rate measurer to existing methods that automatically measure speech rate.

René Ouendag (Q-go)

Q-go’s Natural Language Search is based on matching user questions to model questions from a database. All questions in this database cover a part or the whole of a corporate website. But what happens when one of our customers wants to disclose a database full of information about, for example, all European capitals? Or worse: wants to make an application with a free format fill-out form accessible through Q-go’s system? Adding a model question for each item is tedious work or even impossible (especially when having free format content).
To solve this, we have developed an extension called ‘Named Entities’. For the database example, the model question would be: “Where can I find information on CITY?”, where city can be replaced by any city in the database. For the application the question might be “When does flight FLIGHTNUMBER arrive?” where FLIGHTNUMBER is a variable. In both cases only one question is needed to cover the whole range of possible values the named entity can be replaced with. If the user question matches the entity question, the variable is replaced with the string which was used in the query and the URL behind the question is adjusted to point directly to the relevant content. The free format content differs only in that the value itself is also used in the URL.
Currently two customers have implemented this functionality: KiesBeter.nl (from the Dutch governmental health and environment institute RIVM) and KLM.

Computational Linguistics in the Netherlands (CLIN) 2007

Home Call for abstracts Abstract submission Important dates Location Programme Registration Proceedings Local Organization Sponsors FAQ Pictures

CLIN 2007

Friday December 7, 2007
University of Nijmegen

CLIN 2007 is organized by the Language and Speech group of the Radboud University Nijmegen.

Abstracts parallel session 5

Question-Answering using a Constraint Relaxation Mechanism
Modeling Phonotactic Cues for Speech Segmentation
Document segmentation for passage retrieval in question answering
Lexical Patterns or Dependency Patterns: Which is better?
Automatic Thesaurus Extraction: Comparing Context Models.
The relevance of Natural Language {website|internet|enterprise} Search
Automatic Acquisition of Lexico-Semantic Information for Question Answering.
Automatic measure of speech rate in spoken Dutch
Named Entities in a Question matching environment
Paragraph Retrieval for why-question answering

Question-Answering using a Constraint Relaxation Mechanism
Ana Mendes, Luisa Coheur, Nuno J. Mamede (L2F - INESC-ID)

With the growth of the available digital information, as well as the increasing amount of people accessing it through their cell phones, PDAs and web browsers, the need for Question/Answering (QA) systems has become more acute than ever.
Evaluation forums (such as TREC and CLEF) with QA tasks, aim to improve the research in this field. Participating systems present different methods to pre-process provided corpora as well as in question interpretation: some perform named entities recognition (NER) and deep linguistic analysis; others profit from thesaurus and ontologies or exploit machine learning techniques.
QA@L2F is based on a deep semantic analysis and relies on NER, both in corpora pre-processing and question interpretation. Also the identification of the question type triggers distinct answer extraction processes, namely morpho-syntactic pattern matching, linguistic reordering, named entities matching and brute force plus natural language processing.
Although the system is prepared to apply different strategies to different types of questions, its main feature is to have a constraint relaxation mechanism, which allows it to switch among these strategies in order to return the question's answer: if a precise linguistic analysis on the question is not possible, or if no answer is found in the database, the system relaxes and tries to find an answer in a more flexible and less constrained way.
The paper reports on the results obtained by following the different strategies on the different types of questions.

Modeling Phonotactic Cues for Speech Segmentation
Frans Adriaans (Utrecht University)

One of the tasks that infants are facing in language acquisition is to extract words from continuous speech. Psycholinguistic studies have shown the relevance of phonotactic cues in speech segmentation. Phonotactic cues tell the learner which phoneme sequences are likely or unlikely to occur in the language, where unlikely sequences indicate the presence of a word boundary. In this definition, the term phonotactic cue denotes both statistical cues (transitional probability, mutual information) and abstract phonotactic constraints.
It is not known exactly how phonotactic cues are used to locate word boundaries. Saffran et al. (1996) show that subjects use transitional probabilities, calculated from continuous speech, to distinguish between words and nonwords. Nevertheless, they do not explicitly describe how segmentation proceeds. Brent (1999) implements phonotactic cues in a local minimum approach, using a 4-phoneme window. However, this window size is rather arbitrary. Rytting (2004) argues for a global segmentation threshold, but provides no solution for selecting the right threshold. Moreover, none of these models have considered the possibility that phonotactic cues might take on a more abstract form. Phonological features, for example, can be used to create abstract phonotactic constraints.
In this talk, I will argue that in the framework of Optimality Theory (OT; Prince and Smolensky, 1993) several modeling issues fall into place, since OT by definition integrates the global threshold and local minimum approaches. More importantly, segmentation simulations indicate that our OT model, which uses induced constraints with different levels of abstraction, outperforms purely statistical phonotactic models.

Document segmentation for passage retrieval in question answering
Jörg Tiedemann and Jori Mur (Rijksuniversiteit Groningen)

Passage retrieval is used in question answering to narrow down the search for answers to a given natural language question. In our research we address the problem of document segmentation into appropriate passages, i.e., text units which are large enough for standard IR techniques and small enough for efficient answer extraction. In our study we compare four different segmentation approaches with a baseline of paragraph retrieval using existing markup: (1) We split documents into fixed-size passages. (2) We used a fixed-size sliding window approach (3). We used automatically detected coreference chains to create passages with some semantic coherence (using our own Dutch coreference resolution system). (4) We applied M. Hearst's TextTiling algorithm for detecting subtopic structures.
In this way we compare segmentation strategies for creating linguistically motivated units (using semantic information such as in 3 and 4 or discourse information as in our baseline) with simple window-based techniques.
In our experiments we used data from the Dutch QA tracks at CLEF from previous years (2003-2005). Altogether our test set includes 777 questions with their answers which we used to measure both, passage retrieval performance and question-answering performance. Surprisingly, simple window-based techniques work very well. Details will be discussed in the presentation.

Lexical Patterns or Dependency Patterns: Which is better?
Katja Hofmann and Erik Tjong Kim Sang (Informatics Institute, University of Amsterdam)

This paper is the result of year-long debate between the authors about the value of lexical patterns versus dependency patterns for natural language processing tasks. Lexical patterns are easy to obtain and can be applied to incomplete utterances. Dependency patterns obtain better coverage at the price of robustness and the introduction of parsing errors. For this study we apply both lexical patterns and dependency patterns to the extraction of hypernyms from newspaper texts and the Dutch Wikipedia. By performing a thorough evaluation of the results, we hope to resolve the debate, at least for this NLP task.

Automatic Thesaurus Extraction: Comparing Context Models.
Kris Heylen, Yves Peirsman (K.U.Leuven)

Distributional, corpus based approaches to semantic similarity come in many different flavours. To find the semantically most related words of a target word, some models merely take into account context words to the left and right (so called “bag-of-words” models), whereas others look at the 2nd order context words (the context words of the context words), and still others use the syntactic dependency relations that a target word occurs in. Although the output of these different models diverges widely, not much is known about the linguistic consequences of the choice of context model.
In this paper, we assess the linguistic properties of the “semantic” similarity that is catputred by different context models. What are the differences between bag-of-words and syntactic dependency models? What happens if we combine both? To what extent does context size have an influence? What are the effects of dimensionality reduction?
As a Gold Standard we use EuroWordNet Dutch, a lexical database of Dutch words and their semantic interrelations (synonymy, hypernomy, hyponymy and co-hyponymy). For a test set of 10,000 Dutch nouns, we investigate which semantic relations are found among the 10 most related words retrieved using different context models. Results show that syntactic dependency models retrieve significantly more synonyms than bag-of-words models and that dimensionality reduction leads to a bias towards co-hyponyms.
This research does not only give more insight in the linguistic characteristics of the computational models; it also helps to decide which models are best suited for specific NLP tasks.

The relevance of Natural Language {website|internet|enterprise} Search
Leonoor van der Beeek (Q-go)

Q-go performs deep linguistic analysis on user questions on large corporate websites or intranets, extracts the user intent, and serves a small set of relevant results that match this intent. The core of this application is natural language technology, including language-, branch-, and client-specific dictionaries, ontologies, and grammars, as well as a matching algorithm.
Recently, Q-go has extended the application domain of that technology to include not only website search, but also enterprise and internet search. We have developed two demo systems: the first integrates Q-go Natural Language Search in MOSS 2007 (Sharepoint) search, the second combines Q-go search with the Google Search Application.
The integrated Sharepoint/Q-go search functionality, like our customer interaction management application, extracts the user intent from the question. This user intent is then translated into a Sharepoint search query, using the metatags that are inherent to the system. With this query, answers may be retrieved from all types of content, including relational databases. The combined search offers a better user experience, as users can ask questions in their own words, but it also provides more relevant answers and it answers questions that standard Sharepoint search cannot answer.
The combination of Q-go and Google Search Application is a straightforward one from a technical point of view: the query is sent to both systems, and if Q-go can find one or more matching answers in its own database, they are presented to the user in combined, more relevant set of results.

Automatic Acquisition of Lexico-Semantic Information for Question Answering.
Lonneke van der Plas (University of Groningen)

Lexico-semantic information, such as synonyms (e.g., autumn-fall) and categorised named entities (e.g., Rafael Nadal is a tennis player) can be very helpful in question answering (QA). For example, when a user asks for 'the tennis player that lost the finals of Wimbledon in 2006', it is helpful to have a list of tennis players. Candidate named entities found by the QA system, that are not in the list of tennis players, such as Miguel Ãngel (Rafael's uncle and retired football player) can be easily filtered out. I want to briefly describe the components of our QA system Joost and show how several types of lexico-semantic information can be successfully applied to several system components. Finally, I will explain how we acquired the lexico-semantic information automatically.

Automatic measure of speech rate in spoken Dutch
Nivja H. de Jong & Ton Wempe (Amsterdam Centre for Language and Communication, University of Amsterdam)

Learning to become fluent in a second language is one of the most difficult aspects of learning a second language. Correlations between subjective measures of fluency and objective measures of fluency have shown that speech rate is one of the most important measures of temporal fluency (Cucchiarini, Strik, & Boves, 2002). However, counting syllables is a tedious job and is often omitted from data analyses due to time constraints. In the context of a large-scale research project on the correlates of speaking proficiency carried out at the University of Amsterdam (What is Speaking Proficiency: http://www.hum.uva.nl/wisp), we developed a tool to measure speech rate automatically.
In this paper, we describe this method to detect syllables in speech without need of a transcription. A script written in PRAAT measures intensity (dB) and syllables are subsequently detected as peaks in intensity above a certain threshold, surrounded by dips in intensity. These peaks are then identified as syllables if the sound is voiced. Testing this script in two corpora of spoken Dutch, we find high correlations between manually counted speech rate and automatically measured speech rate. We conclude that the automatic detection of syllables suffice to reliably calculate and compare speech rates between participants and tasks. Finally, we compare our speech rate measurer to existing methods that automatically measure speech rate.

Named Entities in a Question matching environment
René Ouendag (Q-go)

Q-go’s Natural Language Search is based on matching user questions to model questions from a database. All questions in this database cover a part or the whole of a corporate website. But what happens when one of our customers wants to disclose a database full of information about, for example, all European capitals? Or worse: wants to make an application with a free format fill-out form accessible through Q-go’s system? Adding a model question for each item is tedious work or even impossible (especially when having free format content).
To solve this, we have developed an extension called ‘Named Entities’. For the database example, the model question would be: “Where can I find information on CITY?”, where city can be replaced by any city in the database. For the application the question might be “When does flight FLIGHTNUMBER arrive?” where FLIGHTNUMBER is a variable. In both cases only one question is needed to cover the whole range of possible values the named entity can be replaced with. If the user question matches the entity question, the variable is replaced with the string which was used in the query and the URL behind the question is adjusted to point directly to the relevant content. The free format content differs only in that the value itself is also used in the URL.
Currently two customers have implemented this functionality: KiesBeter.nl (from the Dutch governmental health and environment institute RIVM) and KLM.

Paragraph Retrieval for why-question answering
Suzan Verberne, Lou Boves, Nelleke Oostdijk, Peter-Arno Coppen (Radboud University Nijmegen)

In the current research project, we aim at developing a system for answering why-questions (why-QA). In the present talk, we present an approach for why-QA that is based on paragraph retrieval. There are two main reasons for aiming at paragraphs as retrieval units for why-questions. Firstly, answers to why-questions often consist of some kind of reasoning that cannot be expressed in a single clause. Secondly, users tend to prefer answers embedded in paragraph-sized chunks.
For retrieval, we use the Wikipedia XML Corpus, which we indexed using the Wumpus search engine. We created a baseline system that performs paragraph retrieval using all question content words as query terms, and a set of general explanatory cue words such as because and since on document side.
We extended this system with a number of features from smart question analysis, the most important of which being answer type (cause, motivation and etymology) and the informational value of the subject (semantically poor, common and semantically rich). We use these features for changing the weights of the retrieved results. For each answer type, a specific set of cue words is weighted more heavily. The informational value of the subject gives information on which part of the question is expected to appear in the title of the answer document; we use this knowledge on weighting these results more heavily.
In our talk, we present the results of our experiments: the performance of the baseline system and the improvement that we can achieve with smart question analysis.

Suzan Verberne, Lou Boves, Nelleke Oostdijk, Peter-Arno Coppen (Radboud University Nijmegen)

In the current research project, we aim at developing a system for answering why-questions (why-QA). In the present talk, we present an approach for why-QA that is based on paragraph retrieval. There are two main reasons for aiming at paragraphs as retrieval units for why-questions. Firstly, answers to why-questions often consist of some kind of reasoning that cannot be expressed in a single clause. Secondly, users tend to prefer answers embedded in paragraph-sized chunks.
For retrieval, we use the Wikipedia XML Corpus, which we indexed using the Wumpus search engine. We created a baseline system that performs paragraph retrieval using all question content words as query terms, and a set of general explanatory cue words such as because and since on document side.
We extended this system with a number of features from smart question analysis, the most important of which being answer type (cause, motivation and etymology) and the informational value of the subject (semantically poor, common and semantically rich). We use these features for changing the weights of the retrieved results. For each answer type, a specific set of cue words is weighted more heavily. The informational value of the subject gives information on which part of the question is expected to appear in the title of the answer document; we use this knowledge on weighting these results more heavily.
In our talk, we present the results of our experiments: the performance of the baseline system and the improvement that we can achieve with smart question analysis.