Multi-lingual Question Answering

Multi-lingual Question Answering Hai Lang Li Tilburg University Communication and Information Sciences Human Aspects of Information Technology Supervisor: dr. Menno van Zaanen

Multi-lingual Question Answering Hai Lang Li HAIT Master Thesis series nr. 09-006

Tilburg Centre for Creative Computing (TiCC)

THESIS SUBMITTED IN PARTIAL FULFILLMENT OF THE REQUIREMENTS FOR THE DEGREE OF

MASTER OF ARTS IN COMMUNICATION AND INFORMATION SCIENCES, MASTER TRACK HUMAN ASPECTS OF INFORMATION TECHNOLOGY, AT THE FACULTY OF HUMANITIES OF TILBURG UNIVERSITY

Thesis committee: Prof.dr. H.J. van den Herik Dr. M. Van Zaanen Drs. T. Bogers

Tilburg University Faculty of Humanities Department of Communication and Information Sciences Tilburg, The Netherlands September 2009

Preface First of all, I would like to express my sincere gratitude to my supervisor Dr. Menno van Zaanen for his guidance and support. In addition, I would like to thank Prof.dr. H.J. van den Herik for reviewing this report. Without their time, effort and critical advise, this report could not possibly be written. Secondly, I wish to thank my family and friends for their trust, patience and support. Writing this thesis has been a difficult, but challenging part of my life. I am proud on the final result and I hope that you will enjoy reading. Hai Lang Li

I

Abstract Background This study focuses on multi-lingual question answering. In practice, this means that a question is posed in a specific language, but the answer is given in another language. Question answering systems can provide answer to specific questions. This applies to sources which the reader does not understand due to language differences. Incorporating machine translation into the process of question answering enables the user to pose his question in his native language and the MT system translates the question into the target language, in which the sources are written. The QA system will then look for an answer and return this answer in the target language. If needed, this answer can be translated through MT to a language that the user understands. Due to sub-standard MT performance, extra information of the source question is added to the QA system. This extra information is related to the question class, which focuses on named entities. Results Experiments with 6 MT systems show that MT has matured into an useful tool for translation purposes. Adding extra information from the source language certainly improves the performance of a QA system, although the performance of the QA system is not satisfactory. Question classification remains problematic for certain questions without a specific named entity, thus without a meaningful question class. Adding information other than question classes, might provide better results. Conclusion Experimental results reveal that adding extra information, in this case question classes, significantly improves the performance of QA systems, this effect is even more apparent than the influence of machine translation. However, adding question classes alone is not sufficient. Further research is required in determining what extra information can be added besides question classification.

II

Table of contents Preface ....................................................................................................................................... I Abstract .................................................................................................................................... II Table of contents .................................................................................................................... III Chapter 1 Introduction ....................................................................................................... - 1 1.1 Introduction to information retrieval ..................................................................... - 1 1.2 Multi-lingual information retrieval ......................................................................... - 3 1.3 Question answering – next step beyond search engines? ..................................... - 4 1.4 Multi-lingual question answering ............................................................................ - 6 1.5 Problem statement and two research questions .................................................... - 7 1.6 Research methodology ............................................................................................. - 8 1.7 Structure of the thesis ............................................................................................... - 9 Chapter 2 Background ...................................................................................................... - 11 2.1 Development of machine translation .................................................................... - 12 2.1.1 Classical models of MT ..................................................................................... - 12 2.1.2 Statistical machine translation........................................................................ - 14 2.1.3 Two problems of MT ........................................................................................ - 16 2.2 A standard architecture of QA ................................................................................ - 18 2.2.1 Question classification ..................................................................................... - 19 Chapter 3 Experimental set up and experiments ........................................................... - 23 3.1 Machine translation experiment ............................................................................ - 24 3.2 Question classification – question analyzer .......................................................... - 25 3.3 Multi-lingual question answering - OpenEphyra ................................................. - 28 Chapter 4 Results and research findings ......................................................................... - 29 4.1 Machine translation results .................................................................................... - 29 4.2 Question classification results ............................................................................... - 31 4.3 Question answering results .................................................................................... - 32 4.4 Answers to the research questions........................................................................ - 35 Chapter 5 Conclusions ...................................................................................................... - 37 Discussion and future research .................................................................................... - 38 Reference list ..................................................................................................................... - 41 Appendix 1 – Expected answer types .............................................................................. - 43 Appendix 2 – CLEF 06 Dutch questions........................................................................... - 50 Appendix 3 – CLEF 06 English questions ........................................................................ - 55 -

III

IV

Chapter 1 Introduction This thesis deals with multi-lingual question answering. Question answering (QA) relates to the ability of computers to answer questions, posed by a user in a natural language. The multi-lingual part means that we focus on questions and answers in different languages. This can be resolved using machine translation in the context of QA. In this Chapter we provide an introduction to information retrieval and question answering. Structure of this Chapter: 

1.1 provides an introduction to information retrieval.



1.2 extends the subject to multi-lingual information retrieval.



1.3 introduces a new type of information retrieval: question answering.



1.4 adds a multi-lingual aspect to question answering.



1.5 presents the problem statement of the thesis and two research questions.



1.6 describes the research methodology of the thesis.

1.1 Introduction to information retrieval New information is created every day and our daily lives are heavily influenced by it. We need information and knowledge in order “to get a lot of things done”. According to Lyman and Varian (2003), roughly 5 exabytes (5 billion GB) of new information was produced in 2002. But how do we know where to find the information needed, and more importantly, is the information found really relevant? To handle these large amounts of data, some sort of information filtering is required. Filtering mechanisms can, for example, be found in libraries, where people work with indexes. Filtering can help the users by removing non-relevant search results. By searching through these indexes for information, we are dealing with a sort of information retrieval. The meaning of the term information retrieval (IR) is relatively broad. Looking for a phone number in a telephone book so that you can find a restaurant is a form of -1-

information retrieval. A second example is looking up a TV-guide to see what movies are on in the evening. For the scope of this study the following working definition is used: Definition 1.1

Information retrieval (IR) is finding material (usually documents) of an unstructured nature (usually text) that satisfies an information need from within large collections (usually stored on computers) (Manning et al., 2008). Information retrieval used to be an activity that only librarians and professional searchers practiced. This started in ancient history with the introduction of writing systems and writing materials. People spent much time to organize a library collection and so, they had to come up with logical and consistent ideas that helped them search through the collection: the creation of filtering mechanisms. Most of the collection was organized by specific categories: author name, year, etc. The organizing and categorizing was often an ambiguous task. It was very difficult for non-expert people to search and obtain relevant results. For example, if the library collection is organized by the names of all authors, where should a user look for information about light bulbs when the user does not know the name of the inventor? The world has changed. Since the rise of computers and Internet in the early 1990s, hundreds of millions of people engage in information retrieval every day when they use a web search engine or search their email. People do not have to understand filtering or searching mechanisms in an extensive way in order to use search engines, unlike in the past. The user just needs to provide a word to the search engine as input, and the search engine will return a result. Due to the massive amounts of information present on the Internet, the user will likely find relevant information among all the results. The underlying search mechanism is usually not relevant to or interesting for the end-user. It has become so much easier for people to search for and retrieve information, that organizations see certain opportunities to exploit this fact. We mention three examples. 

Google ads/AdSense : advertising service, based on website content, run by Google,



spam: undesired electronic messages, like e-mails, text messages (sms) etc.,

-2-



recommendation systems: present potentially interesting items, like books and music to users (for sale).

The access to computers and the rise of Internet have caused an exponential information growth since the end of the previous century. The information is not stored at a limited number of places (libraries) like in the past, but it is scattered throughout the world, since anyone can contribute information on the Internet. A first problem is that the information can hardly be organized at all, because new information is produced every day and it remains unclear how much information is present on the Internet (Manning et al., 2008). For this reason, search engines are crucial tools, as they can automatically search through (a part of) the information on the Internet in a respectable time. A second problem is the trustworthiness of this information in practice. Sometimes, users find contradictions among different sources of information. It is hard to tell which publisher to believe without a trusted central (human) authority who judges what is trustworthy or not. Instead, people tend to listen to opinion leaders/experts in this matter. For this reason, search engines are coping with this issue by incorporating a measurement of trust to each website or web page, from which information is extracted. It is understandable that new tools must be designed in order to manage and search through the Internet efficiently and effectively. The next section elaborates on the multilingual characteristics of the Internet and how machine translation helps people understand information, encoded in text containing words that are not available in the users’ vocabulary.

1.2 Multi-lingual information retrieval Trends such as globalization and internationalization encourage information exchange between companies and people all over the world. We distinguish two trends. First, multinationals are planning to invest in emerging markets, where labour costs are less than in developed countries. Second, some companies are outsourcing part of their processes to emerging markets. For both scenarios, it is crucial that the companies can

-3-

communicate and make clear what exactly their needs are to people in other countries, who might not understand English sufficiently. Information is contributed by users with different linguistic backgrounds. However, as mentioned in Section 1.1, the available information is presented in a wide range of different languages, with only relatively little information understandable to users with different linguistic backgrounds. Not only language differences are present, but variations in grammar, structure, and style are also noticeable. Computers are able to distinguish between these languages with some forms of linguistic and syntactic operations. Users who only know Spanish can hardly publish texts in English, but most of these users can recognize some words in English texts (Kraaij et al., 2003). This is sufficient in some cases, if the user can understand the context (meaning) of an utterance, just by understanding some words. Human translation could be an option, but professional translators (especially for certain language pairs) are expensive and rather scarce. In order to cope with this problem, automated translation mechanisms, also known as machine translation, have been created and are still in development. These mechanisms consist of automated translating processes from one language to another language. Machine translation is a very hard task. Here we mention four obstacles: (1) structure (word order), (2) stylistic expressions, (3) cultural background, and (4) many other differences between languages (such as business language as opposed to daily languages). Translation requires a deep and rich understanding of the source and target language (Jurafsky & Martin, 2008). So, the improvement of machine translation will help communication and information retrieval on a global scale. Therefore it has become increasingly more important.

1.3 Question answering – next step beyond search engines? Users can visit a search engine like Google when looking e.g. for information on light bulbs. After typing the words “light bulbs” in the search engine, Google currently returns 28.5 million links in 0.20 seconds, containing information on light bulbs. Some links refer to the history of light bulbs, how light bulbs work, energy saving light bulbs

-4-

etc. Some users may then become a bit confused and asks themselves the next question: “How in the world am I going to read all this?” This example shows that search engines are efficient in terms of speed, but they are not quite clever at returning results. The user first has to determine whether the results are relevant or not. The search engine has filtered a part of non-relevant information that helps the user in his search for information, but the information overload is still present. How can this problem be solved? A possible solution is to develop more interactive “environments” that can help targeting results, which are related to the user’s specific needs. The user can for example asks specific questions to the system: 

Who invented the light bulb?



What materials is a light bulb made of?



Where can I buy energy saving light bulbs?

Ideally the system returns tailored answers to all the questions. The user does not have to spend time looking for answers in documents, unlike in the case with search engines. This is an example of a question answering system (QA). Its basic principle is quite straightforward: a user sends a question in natural language to a system. After processing, the system provides possible answers.

QA can be seen as a type of

information retrieval, but there are some differences in the input/output format (see Table 1.1). Table 1.1 Differences between QA and IR

QA

IR

Input

Real question

Keywords

Output

Focused answers

Documents

Apart from the advantage of not having to read and analyse all the documents, QA has a practical advantage. Smartphones and 3G communication are becoming increasingly popular, since more people can afford these tools. The screen of smartphones is often not sufficient to read large documents, because then the screen size is just too small. -5-

However, QA can be implemented, because the screen usage is limited. Only the question and the answers (with its source) need to be displayed.

1.4 Multi-lingual question answering Many users have some knowledge of foreign languages and they would like to be able to access foreign resources, just out of interest. For example, consider a user from the Netherlands who is interested in research on health effects of soft drinks. This research is conducted in France and the report is only available in French, but his knowledge of the foreign language (French) may not be sufficient in order to formulate targeted questions. This user would require a system that allows him to pose questions in his native language (Dutch). However, he might be able to recognize some words (like the names of soft drinks) in the French answers. For instance, his question was “Welke frisdrank bevat het minste suiker?” (Which soft drink contains least sugar?) The QA system returns: le sode tonique. The user can guess that le sode tonique is the same as tonic water. The emergence of multi-language QA systems is a relatively recent development, and many of the existing systems are still in an experimental stage. Up to now, most work concentrates on the Western European and Asian languages, while many very different methodologies are being proposed (Braschler & Peters, 2002). The most basic method is illustrated in Figure 1.1 and contains two steps. In the first step, the original question is translated to the target language by machine translation. In the second step, the translated question is processed by a QA system and the system returns an answer in the target language. More advanced methods can incorporate natural language processing tools and text mining.

Machine translation Input (Dutch question)

Translated question

Figure 1.1 A basic multi-lingual QA system

-6-

Question-answer system

Output (French answer)

1.5 Problem statement and two research questions Referring to Figure 1.1, the question answering system depends on the quality of the translated question. According to the phrase “Garbage in, Garbage out”, we assume that the performance of a QA system is influenced negatively by the moderate performance of MT systems. Hence, the problem statement (PS) of this study reads as follows. PS: To what extent will a QA system be influenced by the quality of the translated question? The performance of machine translation is still not optimal due to many obstacles, of which four are mentioned in Section 1.2. The first research question is then defined as follows. RQ1: What is the influence of machine translation on question answering?

In order to reduce the (negative) influence of machine translation, we can look at tools that provide extra information about the translated question submitted to the QA system, such as the type of the question, the linguistic information, etc. One way to add extra information is to analyse the translated question. However, this analysis is directly influenced by the quality of the translated question. If the meaning of the translated question differs from the meaning of the source question, the analysis would be incorrect. Therefore, we propose to perform the analysis of the found information on the source question. This thesis introduces a question analyzer module that adds extra information about the source question in addition to machine translation. This extra information can be described as meta data. The basic idea is that the question analyzer examines the source question for the type of question. For example, the source question in Dutch is: waar staat de Eifel toren? (where is the Eifel tower located?). The question analyzer will return ‘location’ as output, since this question indeed asks for a location. This information restricts the number of possible answers. Other examples include person (who), time (when), distance (how far) etc. The meta data is language independent, as the meaning of the question type is similar for all languages.

-7-

In Figure 1.2 we outline the structure of a basic multi-lingual QA system, together with the question analyzer. The second research question is thus formulated according to the situation in Figure 1.2. RQ2: Will the performance of a multi-lingual question-answer system improve by incorporating meta data, extracted from the question in the source language, along with machine translation?

Meta data

extra information from question In source language

Question analyzer Input (Dutch question)

Question answering system

Output (English answer)

Machine translation Translated question

Figure 1.2 A basic multi-lingual QA system integrated with a question analyzer

1.6 Research methodology In order to answer the research questions, background information about machine translation and question answering in general is needed. This background information is mainly composed of a literature study. With a better understanding of the tools used, a set of experiments can be designed using machine translation programs and a questionanswering system. The input for machine translation and the QA system consists of a set of questions used in the CLEF competitions. CLEF stands for Cross Language Evaluation Forum, which promotes research and evaluation of cross-language information retrieval. The experiments are then performed as illustrated in Figure 1.2. The first set of experiments consists of testing the question analyzer. The results, obtained from machine translation, are compared to the translated questions, provided by CLEF. The answers from the QA system (with or without meta data) are then compared to answers provided by CLEF. A detailed overview of all experiments can be found in Chapter 3. After an evaluation of the comparisons, we are able to formulate the answers to -8-

research question one and research question two. In brief, the research methodology is designed as follows. 

Literature study and background information in Chapter 2



Designing experiments in Chapter 3



Performing experiments and describing results in Chapter 3



Analysing the results in Chapter 4



Evaluating the results in Chapter 5

1.7 Structure of the thesis In Chapter 1 we formulate the problem statement with the research questions and design the research methodology. Chapter 2 consists of background information composed from a literature study. In Chapter 3, the experimental set up that is used to conduct experiments is described. In Chapter 4, the results from the experiments are presented and discussed. The two research questions are answered with the help of the results and the outcomes of the discussion. Eventually, the problem statement is answered and conclusions are drawn in Chapter 5.

-9-

- 10 -

Chapter 2 Background This chapter provides a brief overview of the literature on the subject of multi-lingual question answering, which relies heavily on machine translation technology and question answering, as illustrated in Figure 1.1. Therefore, background information is provided in the context of these two fields. Due to the limited scope of this study, it is not very practical to cover all relevant literature of the two fields. The available literature is too comprehensive; instead, this study only focuses on aspects that are relevant for the experimental set up and our experiments, which will be discussed in Chapter 3. Figure 2.1 shows the intersection of both fields. The experimental set up partly consists of testing online-based machine translation systems and a question answering system, which are called the research tools for convenience. It is quite useful to have a basic idea about how these systems operate, without having to know all the ins-and-outs. With some background information about these systems, it is easier to interpret the results. Moreover, when a system performs below par, most of the times this is related to the system’s model. By understanding the model, the system can be improved.

Figure 2.1 Multi-lingual question answering: sub-field of QA & MT

It is practical to have working definitions of machine translation and question answering for a proper progress of this study. The study only focuses on fully automated machine translation. Definition 2.1

Machine translation explores the use of computer software to automate the process of translating from one natural language to another. Definition 2.2

Question Answering relates to the ability of computers to retrieve answers to questions posed in natural language (Jurafsky & Martin, 2008). - 11 -

2.1 Development of machine translation This section introduces some basic ideas of machine translation (MT) and what models are used by computers for MT. Section 2.1.1 provides a brief overview of the classical models of MT. 2.1.2 describes a recent development of MT: Statistical MT. 2.1.3 provides information on problems in translations of MT. 2.1.1 Classical models of MT In classical machine translation, three architectures have been proposed for machine translation. Many systems are still partly based on (combinations of) elements from these architectures. The architectures are visualized in Figure 2.2, which is typically called the Vauquois triangle. In direct translation, the source language text is processed word-by-word, translating each word that passes the system. Direct translation uses a large bilingual dictionary, in which each of the entries is a small program with the task of translating one word. After the words are translated, the sequence of the words is set by following reordering rules. Direct translation is thus characterized as word-by-word translation with some wordorder adjustment. Due to the lacking of grammatical relationship and syntactic analysis between sentences, the results are often disappointing with mistranslation at the lexical level and inappropriate syntax structures which resembled the source language too closely (Hutchins & Somers, 1992). Ambiguity of words and translation remains the main problem, since words can have several meanings and each meaning might have a different translation in another language. Proper translation (i.e. choosing the right meaning) of words is very difficult without knowing the context in which the words occur. In transfer approaches, the input text is firstly parsed. The parsed text has a typical parse structure and by applying translation rules, the parse structure of the text in the source language is transformed into parse structure in the target language. Then the system generates a target language sentence from the parse structure. In brief, the transfer model involves three stages: (1)analysis, (2)transfer, and (3)generation. Two transfer types can be distinguished in the transfer model: syntactic transfer and - 12 -

semantic transfer. Syntactic transfer is characterized by transferring syntactic structures between the source and target language. Semantic transfer is inspired by methods and techniques of Artificial Intelligence (AI), which focuses on semantic representation (meaning), including semantic parsing, semantic networks, etc. For languages in the same family, e.g. Roman languages, syntactic transfer is more suitable because of syntactic similarity between languages. Semantic transfer is more useful for languages with different syntactic structures (Horwood, 1986). A problem with the transfer model is that it requires a distinct set of transfer rules for each pair of languages. For translation systems employed in many-to-many multilingual environments like the European Union, this is not practical (Jurafsky & Martin, 2008). In interlingua approaches, the source language is analyzed and transformed into some abstract meaning representation, known as an interlingua, a language-independent canonical form. Esperanto and Ido are two examples of artificially constructed languages, that have been used in this context. Next the target language is generated from this interlingual representation. The advantage of using an interlingua machine translation approach is that no transfer component has to be created for each language pair, unlike in the transfer model. If a new source language is transformed into an interlingua, then it can be translated from the interlingua to any language available for translation. However, the interlingua model has its own problems. For example, large amounts of unnecessary disambiguation are required for capturing meanings that are not present in all languages, due to cultural differences, metaphors, proverbs, idioms etc. Furthermore, analysis of the semantics of numerous domains requires extra work, because of differences in the meaning of words for a certain domain. Therefore, interlingual systems are generally used in sublanguage domains (medical, juridical, etc.) (Jurafsky & Martin, 2008).

- 13 -

Figure 2.2 The Vauquois triangle

As we move up the pyramid, the translation process becomes more difficult, but the system generates better results and it is more flexible (extensible in language pairs). 2.1.2 Statistical machine translation The three classic architectures, as seen in Figure 2.2, all provide frameworks describing which representations to use and what steps to take in order to translate. All these architectures have to deal with ambiguity in languages. It would be convenient if we do not have to deal with this ambiguity explicitly. This is possible if the translation only focuses on the result and not on the process. This means that when we have an extensive collection of data (bilingual corpora), we can easily observe the language pairs that are most frequent within a certain context. Bilingual corpora refers to a large collection of texts and documents written in two languages. If these language pairs are used to translate from one to another language, the result should be satisfactory given the context of the texts is known. If we only focus on the result, it would be desirable to obtain results that are both faithful to the source language and natural as an utterance in the target language. This is not always possible, so a compromise is formed. The above description serves as the goal of statistical machine translation (SMT), which builds probabilistic models of faithfulness and fluency, and then combines these models to choose the most probable translation (Jurafsky & Martin, 2008).

- 14 -

The development of SMT can be traced back to 1949, mainly based on research of Weaver and Shannon on cryptography during World War II. Machine translation was at the time conceived as the problem of finding a sentence by decoding a given “encrypted” version of it (Weaver (1949) as cited in Brown et al., (1990)). But due to limited computational power the development of SMT was stopped. Four decades later, the research of SMT was picked up, because of two factors: 

the explosive rise of computational power and storage capacity,



the availability of large bilingual corpora (Mariño et al., 2006).

The first SMT systems were developed in the early nineties. These systems were based on the view that each possible target language text is a potential translation for any given source language text, but some translations are more likely than others. Brown et al. (1990) introduced a noisy-channel model, which converts this view to a statistical distribution P(T|S), where S is a source language text and T is a target language text1. In a probabilistic model, the best target language text T is the one with the highest P(T|S). Following Bayes’ law, P(T|S) can be rewritten as P(T|S) =

𝑃 𝑆 𝑇 (𝑃(𝑇) 𝑃 𝑆

.

Since the goal is to find the best target language text T, P(S) can be ignored, as P(S) remains constant. The equation can then be simplified as follows: Best translation

translation model

/\ = argmax P(T|S) = argmax

/\ P(S|T)

language model ∙

/\ P(T)

The translation model can be derived from bilingual corpora, while the language model can be obtained from monolingual data of the target language. The language model refers to a probability distribution that captures the statistical regularities of the generation of language. For example, the language model can be used to predict the probability of the next word in an ordered sequence (Ponte & Croft, 1998).

This model is referred to as noisy-channel, because it pretends that S is the result of some input signal T’s being corrupted while passing through a noisy channel. Then the goal is to recover the initial input, given the corrupted data. 1

- 15 -

2.1.3 Two problems of MT MT systems are far from perfect. They often produce incorrect translations (Kraaij et al., 2003). Fixed phrases, collocations, idioms, and domain-specific terminology are often translated incorrectly. This is the first problem of MT. Furthermore, domain-specific terms are often not covered in common dictionaries. An important sub problem is that certain classes of words, e.g. proper names, can be challenging to MT systems to translate. The names of entities such as persons or locations are frequently used in text. Translating geographical names can be difficult due to spelling differences in different languages (e.g., London/Londen/Londres), while some person names, like Tiger Woods are not always recognized as named entities. Still, the MT system has to decide whether to translate the entity or not. It is important that London is translated in the target language, while Tiger Wood remains untranslated. MT systems has to recognize proper names correctly for this matter. The second problem of MT refers to the ability of MT systems to deal with translation alternatives. A word or term often has multiple translations, partly due to syntactic differences. For instance, the English utterance “they watch television” can be translated in French as “elles regardent la télévision” or “ils regardent la television”. It is not clear which translation to choose, because the English “they” is gender neutral. Extra information from the context is required in order to decide which translation to keep. An important question is how to keep the appropriate translation while discarding the unsuitable ones. MT systems tend to assign weights to words in documents according to how many times a word occurs in documents. The MT system will then pick the translation that occurs most frequently. Adding more bilingual corpora to MT systems may be an option to reduce this problem. Despite the two problems of MT, the meaning of a sentence is preserved, because the key words are translated correctly, although the sentence is not perfectly translated. Since this study focuses on question translation, the rest of this section and Chapters will discuss questions only. Examples of faulty MT are provided in Table 2.1. The incorrect translated words or phrases are marked in italics.

- 16 -

Table 2.1 Examples of faulty MT

Dutch question

English human translated

English MT

Type of error

Wat is Atlantis?

What is Atlantis?

What are Atlantis?

agreement

Welke boete kreeg

What penalty was

Which fine got John

John Fashanu?

John Fashanu given?

Fashanu?

word choice and syntax

Wanneer viel de

When did the Berlin

When the Berlin wall

Berlijnse muur?

wall fall?

fell?

Wie was Alexander

Who was Alexander

Who Alexander

Graham Bell?

Graham Bell?

Graham Bell was?

Noem de drie

List the three living

levende Beatles

Beatles.

Van welke partij is Wouter Bos lid?

syntax syntax

Would like to mention the three live

word choice (NE)

album.

Which party is

Of which party is

Wouter Bos member

Wouter bunch

of?

member?

word choice (NE)

From Table 2.1, we can see an example of entity translation. Alexander Graham Bell and Wouter Bos are both person names, but Wouter Bos is translated. The MT system should have left Wouter Bos untranslated. All meanings of the source questions are somehow preserved, except the example with the living Beatles. “levende Beatles” is translated as live album. An explanation for this mistake, is that the MT system thinks that Beatles resemble the albums they produced and people are more interested in their albums than in the persons forming Beatles.

- 17 -

2.2 A standard architecture of QA Question answering is a form of information retrieval that deals with questions in natural language. The main goal is to retrieve explicit answers to questions rather than whole documents. This task has been intensively investigated in the Text Retrieval Conference2 (TREC) and the Cross Language Evaluation Forum3 (CLEF). One of the QA systems that participated in both competitions is OpenEphyra, an open-source question answering system (Schlaefer et al., 2006, 2007, van Zaanen, 2008). The system is a modular and extensible framework written in Java. In this study we will use this system as the QA component in our multi-lingual QA set up. The modular design of the system allows us to add our own question analyzer component to the system. In section 1.5 we argued that the question analysis is performed on the question in the source language rather than the translated question in the target language, because the performance of MT systems is still not optimal due to many obstacles and problems, which are respectively mentioned in Section 1.2 and Subsection 2.1.3. Many of today’s question answering systems use a common architecture as illustrated in Figure 2.3. A basis QA system consists of four modules: question analysis, query generation, search, and answer extraction and selection.

Figure 2.3 Basic QA system 2 3

www.trec.nist.gov www.clef-campaign.org

- 18 -

The question analyzer determines the expected answer type of the question and interprets the question to create a more concise representation of the question string, e.g. an aspect of the question. Table 2.2 gives some examples (see Subsection 2.2.1). The query generator transforms the question into queries (keywords) for document retrieval. The QA system will then search for possible answers in external sources, which might include structured (Wikipedia) or unstructured documents (web search engine). Eventually, the search results are processed by a filter, which checks the results for the expected answer type (such as location or time). The QA system will then present a list of ranked answers, if more answers are applicable (Schlaefer et al., 2006). 2.2.1 Question classification One of the crucial steps in answering questions is the classification of the expected answer type. Knowing the expected answer type will help the QA system determine the answer type and thus select the answer. In other words, the QA system “knows” what type of answer is expected. In this study, we develop a question analyzer module, which focuses on the source question. Table 2.2 Expected answer type list

Expected answer type NEacronym NEangle NEbirthstone NEbodyPart NEcauseOfDeath NEcolor NEcreature->NEanimal->NEbird NEcreature->NEplant->NEflower NEcrime NEdate NEdate->NEcentury NEdate->NEday NEdate->NEdayMonth NEdate->NEdecade NEdate->NEmonth NEdate->NEseason NEdate->NEweekday NEdate->NEyear NEdisease NEdramaType->NEshowType NEdramaType->NEfilmType NEdrug->NEmedicinal

NEpathogen->NEvirus NEpercentage NEprofession NEproperName->NEaward NEproperName->NEbook NEproperName->NEdrama->NEfilm NEproperName->NEdrama->NEplay NEproperName->NEdrama->NEplay->NEmusical NEproperName->NEdrama->NEshow NEproperName->NEethnicGroup NEproperName->NEevent->NEcompetition NEproperName->NEevent->NEconflict NEproperName->NEevent->NEfestival NEproperName->NEmusic->NEopera NEproperName->NEmusic->NEsong NEproperName->NEmusic->NEsong->NEanthem NEproperName->NEorganization NEproperName->NEorganization->NEeducational Institution NEproperName->NEorganization->NEministry NEproperName->NEorganization->NEnewspaper NEproperName->NEorganization->NEpoliticalParty

- 19 -

NEdrug->NEnarcotic NEdrug->NEvaccine NEduration NEduration->NEdays NEduration->NEyears NEfood NEfood->NEfruit NEfrequency NElanguage NElegalSentence NElocation NElocation->NEairport NElocation->NEcity NElocation->NEcity->NEcapital NElocation->NEcontinent NElocation->NEcountry NElocation->NEcounty NElocation->NEhemisphere NElocation->NEisland NElocation->NEmountain NElocation->NEmountainRange NElocation->NEnationalPark NElocation->NEpeninsula NElocation->NEplanet NElocation->NEprovince NElocation->NEreef NElocation->NEstate NElocation->Nestreet NElocation->NEwater NElocation->NEwater->NEcanal NElocation->NEwater->NElake NElocation->NEwater->NEriver NElocation->NEwater->NEsea NElocation->NEwater->NEsea-> NEocean NEmaterial->NEchemicalElement NEmaterial->NEmetal NEmaterial->NEmineral NEmaterial->NEstone NEmedicalTreatment NEmedicalTreatment->NEtherapy NEmoney NEmusicalInstrument NEmusicType NEnationality NEnumber NEnumber->NEordinal

NEproperName->NEorganization->NEradioStation NEproperName->NEorganization->NEteam NEproperName->NEorganization->NEtvChannel NEproperName->NEperson NEproperName->NEperson->NEactor NEproperName->NEperson->NEauthor NEproperName->NEperson->NEdirector NEproperName->NEperson->NEfirstName NEproperName->NEperson->NElastName NEproperName->NEperson->NEmathematician NEproperName->NEperson->NEplaywright NEproperName->NEperson->NEscientist NEproperName->NEperson->NEusPresident NEproperName->NEstadium NErange NErate NErate->NEspeed NErate->NEspeed->NEmph NErelation NEreligion NEscore NEsize NEsize->NEarea NEsize->NEarea->NEsquareMiles NEsize->NElength NEsize->NElength->NEfeet NEsize->NElength->NEmiles NEsize->NEvolume NEsize->NEvolume->NEgallons NEsize->NEvolume->NEliters NEsize->NEvolume->NEounces NEsize->NEweight NEsize->NEweight->NEgrams NEsize->NEweight->NEpounds NEsize->NEweight->NEtons NEsocialTitle NEsocialTitle->NEacademicTitle NEsocialTitle->NEmilitaryRank NEsocialTitle->NEnobleTitle NEsocialTitle->NEpoliceRank NEsport NEstyle NEtemperature NEtime NEtime->NEdaytime NEtime->NEhour NEtimezone

- 20 -

NEnumber->NEphoneNumber NEnumber->NEzipcode NEpathogen->NEbacteria

NEunit->NEcurrency NEurl NEzodiacSign

The native question analyzer of OpenEphyra targets the question that is used as input to the system. In a multi-lingual set up, this question is the question in the target language. We argued before in Section 1.5 that this approach (question analysis on question in target language) can be problematic, especially if the question is translated incorrectly, resulting in a loss of meaning of the question. Therefore, we recommend to perform the analysis on the question in the source language, so the meaning of the question remains preserved. This will be researched in further analysis when we compare the effect of translation from several MT systems on QA. Our question analyzer works with a list of expected answer types, which are presented in Table 2.2. This list is extracted from OpenEphyra. If the QA system finds a expected answer type (keyword) in a question in the source language that matches one of the answer types in the list, the system will process this result during further analysis of the question in the target language. Conversely, if the QA system does not find a suitable expected answer type for a question, the system will assign the class “NErest” to the keyword in the question. The class NErest is constructed for this research, so it is therefore not present in the list of expected answer types in OpenEphyra. The choice for a predefined list of expected answer types can be explained by the fact that this list can be customized easily. This is in contrast to lists generated from statistical models. Statistical and machine learning models are time-intensive approaches, because a huge amount of various training data is required in order to produce a representative list, which covers a significant part of subjects. For the scope of this study, the use of statistical of machine learning models is therefore not feasible. Instead, we have customized the keywords in the list of expected answer types in order to analyze Dutch questions. A complete list of expected answer types and regular expressions (keywords) is presented in Appendix 1.

- 21 -

- 22 -

Chapter 3 Experimental set up and experiments Having understood the possibilities and limitations of the research tools from Chapter 2, we can perform a series of experiment with these tools. Figure 3.1 illustrates the experiment overview for this research. The experiments are divided into three parts: Experimental set up: Section 3.1 machine translation with online MT systems. 3.2 question classification with the question analyzer. 3.3 multi-lingual question answering using OpenEphyra. The first experiment focuses on machine translation, using six online MT systems and comparing the output to human translations. The second experiment consists of testing several question analyzers and compare the output to human assigned question classes OpenEphyra (gold standard). The final experiments uses the output of the first and second experiment as input to perform question answering. The issue here is the influence of adding the question classes to the QA system in addition to the translated questions and the influence of the quality of MT output. Comparison to Gold standard

Gold standard Regular expressions

Meta data: question class

NEperson (Majority class) NErest

Question-answer System:

Question classification

Machine translation

Question (Dutch)

OpenEphyra

Translated by person

Question translation

Google translate Other MT systems

Figure 3.1 Experiment overview

- 23 -

Translated question Answer (other language)

3.1 Machine translation experiment The input to the first two experiments consists of a set of 200 Dutch questions. These questions are provided by CLEF 06. The complete list of 200 questions can be found at Appendix 2. The output of the MT systems is compared to the English translation of CLEF 06. (However, the English translations provided by CLEF were not complete, so about 100 questions were translated from Dutch to English by the author of this thesis.) The English translations can be found in Appendix 3. Some examples from the question set with the English translation in italics. 

Wie schreef de trilogie "In de ban van de ring"?



Who wrote the trilogy “Lord of the Rings”?



Hoeveel passagiers vervoert het schip "Canadian Empress"?



How many passengers does the ship “Canadian Empress” carry?



Waar werd het cruiseschip gebouwd?



Where was the cruise ship built?



Hoeveel deelstaten heeft Duitsland?



How many federal states are there in Germany?



Wanneer liep de eerste mens op de Maan?



When did the first human walk on the Moon?

By comparing the output of the MT systems with the human translated questions, we can assess the translation quality (from Dutch to English) of the MT systems. Two conditions had to be met choosing MT systems. 1. The MT system is able to translate from Dutch to English. 2. The MT system is freely available. Firstly, the 200 Dutch questions were uploaded to a web page, facilitating the translation process for MT systems, which typically translates entire web pages automatically when provided with the corresponding web link. Secondly the questions were translated from Dutch to English, using the following six online based MT systems (see Table 3.1).

- 24 -

Table 3.1 List of used MT systems

Machine translation system

translation technology (engine)

SYSTRANet4

SYSTRAN

Yahoo! Babel Fish5

SYSTRAN

Worldlingo6

unknown

Google Translate7

Google language tools

Microsoft Live Search Translator8

Microsoft Translator technology

SDL Free Translation9

SYSTRAN

All six MT systems offered the option to translate web pages and each MT system, except SDL Free translation, were capable of translating the web page with Dutch questions smoothly. SDL Free translation returned a black page. For this MT system, the questions were entered manually. Finally, the translations are compared to human translation using two evaluation metrics: 

BLEU (bilingual evaluation understudy) calculates n-gram precision adding equal weight to each one (Papineni et al., 2001).



NIST calculates how informative a n-gram is. If a correct n-gram is found, the rarer that n-gram is, the more weight it will be given (Doddington, 2002).

3.2 Question classification – question analyzer As argued in Section 2.2.1, question classification is an important step in question answering, enabling the QA system to ‘know’ what type of question is posed and what answer type is expected to the particular question type. The native question analyzer of OpenEphyra targets the question that is used as input to the system. In a multi-lingual set up, this question is the question in the target language. In this study, however, we prefer to perform the analysis on the question in the source language instead, because machine translation can result in meaning loss of the translated question if translated Systranet.com Babelifish.yahoo.com 6 worldlingo.com/en/products_services/worldlingo_translator.html 7 Translate.google.com 8 Translator.live.com 9 Freetranslation.com 4 5

- 25 -

incorrectly, as argued in Section 1.5. For this reason, we built a question analyzer that focuses on the source question in Dutch. The question analyzer was built with the concept of retrieving the question class of a question, using the list of predefined question classes of OpenEphyra. The list of question classes consists of several question class with its corresponding keywords. How can the question class then be retrieved? A possible solution is to compare the question with a list of regular expressions, which include a list of keywords related to a certain question class. If a keyword matches with a word in the question, then the question class of the keyword is presented as the question class of the question. The question analyzer for Dutch questions performs the following actions for each line of the list of Dutch questions: 1. First, the question is “normalized”, which means that commas, question marks and extra spaces are removed as these hinder the next process. For example: Wie is de president van Letland / Who is the president of Latvia 2. The question is then compared to the list of predefined regular expressions (Appendix 1), doing this in a case-insensitive way. If a regular expression is found to match the question, the system assigns the corresponding expected answer type to this question and continues with the next question. In this case the word “wie” (who) is found, which matches with NEproperName->NEperson. 3. If the question does not match with any regular expression, then the system assigns the class NErest to the particular question. NErest is a residual class, in which no suitable expected answer type can be assigned to a question. The question analyzer is tested on Dutch questions provided by CLEF 03 & 04, a total of 400 questions. As output a list of question classes was returned, which is compared to human annotated question classes. The results for CLEF 03 were very promising, the question analyzer and human annotator had a match of 95%. In comparison, the results for CLEF 04 were less promising, but still reasonable, a matching of 78%. The difference in matching scores can be explained by the fact that about 15% of CLEF 04 questions

- 26 -

consist of “why” and “how” questions (asking for reasons and explanations), which are not defined by the used expected answer type list. So the inability to classify “why” and “how” questions is a limitation of our question analyzer, because of the hierarchy of classes used in this research. The native question analyzer of OpenEhpyra shares the same limitation, since the list of expected answer types used for classification is the same. Another limitation relates to the method of matching expected answer types.. The first word in a sentence is picked and matched with the expected answer types list. If a match is found, the system will return the result and continue with the next question. But if the word, that is found, is not the subject of the question, a wrong class is likely to be assigned to the question. This will happen if the question consists of many words, where the first word describes the context of the question. For example, a wrong class will be assigned to this question: Given the distance between the earth and the moon, in which part of the year can a full moon be seen? The question analyzer will “think” that this question asks for a distance, but in fact this question asks for a time period. With promising results from the tests with CLEF 03 & 04, we can start with the experiment for CLEF 06 questions. Beforehand, a gold standard of annotated expected answer types by two humans for CLEF 06 is generated. Questions, where the annotators could not agree on the expected answer type, are assigned as NErest. In theory, the question analyzer can perform better than the gold standard, due to the class NErest assigned to questions, which the annotators could not agree on, given that all other questions are correctly assigned. The analyzer can return a suitable question class, while the question in the gold standard is marked as NErest. We have generated a list of a majority class NEperson (the class that occurs most) and a list of NErest (all questions which are classified as unknown), as a baseline for the experiment in OpenEhpyra. It is interesting to see how the question analyzer for Dutch questions performs in comparison with the majority class and NErest.

- 27 -

3.3 Multi-lingual question answering - OpenEphyra After the experiments with MT systems and the question analyzer, we can start testing OpenEhypra with the output from the previous experiments (see Figure 3.1): 

6 lists of translated questions by 6 MT systems + human translated questions.



5 lists of question classes (including OpenEphyra’s native question analyzer).

The results are then presented in a 7 x 5 matrix: gold

Regular

standard

expressions

NEperson

NErest

OpenEphyra analyzer

SYSTRANet Babel Fish Worldlingo Google translate MS live translate SDL free Human translated By analyzing the matrix, several research findings can be extracted. These findings, marked as Fn, are presented in Section 4.3. Two matrices are presented: one matrix for exact matches between QA results and gold standards; one matrix for partial matches.

- 28 -

Chapter 4 Results and research findings This chapter presents the research findings and subsequently provides answers to the research questions which were formulated in Chapter 1. All experiments are conducted with a set of 200 Dutch questions from CLEF 06. Section 4.1 provides machine translation results. 4.2 presents the results of question classification. 4.3 shows the findings from question answering. 4.4 answers the research questions.

4.1 Machine translation results The results from the machine translation experiments are presented in Table 4.1. The meaning of NIST and BLEU are mentioned in Chapter 3.1. For both NIST and BLEU score, Google translate performs the best of the six MT systems. The extensive data collection and the vast computational resources of Google might have an influence on the MT performance of its system. From a visual inspection of Google translated questions, we can conclude that the majority of the questions are translated perfectly with a fluent sentence structure and a logical choice of words. Some small errors can be found, which do not influence the meaning of the questions though. For the lower rankings, the NIST and BLEU scores are quite close for SYSTRANet, SDL free and Wordlingo. Because SYSTRANet and SLD free are based on the same translation technology e.g. SYSTRAN, the two systems have identical NIST and BLEU scores. The sentences are translated identically and the errors are also the same for both systems. The two systems translate most of the words correctly, but they tend to have some problems translating verbs and person names. In some cases the meaning of the question is lost (a correct translation is provided in italics): 

Who reason Jews in the Second world War? Who saved the Jews in the Second World War?



By which party hears Clenched Clinton? Which party is Bill Clinton member of?



Who has Minerva Rigging set up with Chicken Marina? Who founded Minerva Rigging at Kip Marina? - 29 -

The structure of Worldlingo translated questions are moderately fluent . Wordlingo has problems translating certain Dutch words: 

Freud in 1939, moved Waarheen? Where did Freud move to live in 1939?



Which percentage of all sold milk is used with ontbijtgranen? What percentage of all milk sold is used with cereals?



How many keren Zinedine Zidane have won the US open? How many times has Zinedine Zidane won the U.S. Open?

Microsoft Live translate and Babelfish take the fifth and sixth position in the chart. Although BabelFish is based on the same translation technology of SYSTRANet and SDL free, the system scores considerably lower than its “colleague” systems. A possible explanation for this occurrence, is that BabelFish is optimized for non-question sentences, because question sentences take a very small part in a document collection. By concentrating on sentences with a certain “regular” structure, BabelFish might perform better in non-question competitions. Translation of single words is quite good of Babelfish, but producing coherent sentences is quite problematic: 

How many Oscars have won starwars? How many Oscars has Star Wars won?



In which sport won Europe of America in 1987? In which sports did Europe win from America in 1987?



What are the nasty or the largest prison in San Paolo? What is the name of the largest prison in San Paolo?

Microsoft live translate performs quite below par, regarding the status of Microsoft on computational technology. The structure of the sentences is sometimes illogical, words are translated incorrectly, and some Dutch words are not translated: 

Who wrote the trilogy "in the hands of the ring"? Who wrote the trilogy "Lord of the Rings"?



Anyone who redde Jews in the second world war? Who saved the Jews in the Second World War?



Would like to mention the three live album. List the three living Beatles. - 30 -

Table 4.1 NIST and BLUE scores of used MT systems

NIST score

BLUE score

Google translate

9.2164

Google translate

0.7006

Worldlingo

6.6781

SYSTRANet

0.4038

SYSTRANet

6.6732

SDL free

0.4038

SDL free

6.6732

Worldlingo

0.3927

MS live translate

6.5701

MS live translate

0.3642

Babel Fish

6.5547

Babel Fish

0.3500

4.2 Question classification results By classifying the question type of a particular question, the QA system will “know” in what topic area to search for the corresponding answer. The question classifier in this study extracts the question type of a question from a predefined list of expected answer types, represented in Table 2.2. After the classifier defines the expected answer type and returns a list of expected answer types, this list is compared to the gold standard of expected answer types. The gold standard consists of manually assigned expected answer types of two people, and a matching expected answer type of the two people for a question is then transferred to the gold standard list. Questions which are classified as NErest (no corresponding expected answer type can be assigned) are reviewed as incorrect expected answer type, because only meaningful expected answer types are incorporated to the list of gold standard. A comparison of scores for CLEF 03, CLEF 04 and CLEF 06 are presented in Figure 4.1.

CLEF 03

95

CLEF 04

23

78

15

7

correct quesion types NErest other mistakes

CLEF 06

64

0%

20%

18,5

40%

60%

Figure 4.1 Percentage of correct question types

- 31 -

17,5

80%

100%

Compared to the scores for CLEF 03 (95.5%) and CLEF 04 (78%), the result for CLEF 06 is quite disappointing with a score of 64% correctly assigned classes, but still reasonable. Out of the 200 questions of CLEF 06, 37 questions (18.5%) are classified as NErest by the classifier. 17.5% of the questions are classified incorrectly, meaning an incorrect class is returned other than NErest. The inability of the classifier to handle “what is” and “why” questions remains an obstacle in question classification. In further research, this problem should be addressed.

4.3 Question answering results Table 4.2 Percentage of exact matches

gold

Regular

standard

expressions

NEperson

NErest

OpenEphyra analyzer

Google translate

12

10

6

6.5

14.5

Human translated

10

10

5.5

5.5

16.5

SDL free

8.5

8

3

4

10

SYSTRANet

7.5

8

4

3.5

12.5

Babel Fish

7.5

7.5

2.5

3

12.5

8

8.5

2.5

2.5

9.5

8.5

8.5

3

2.5

12

MS live translate Worldlingo

Table 4.3 Percentage of partial matches

Human translated

gold

Regular

standard

expressions

NEperson

NErest

OpenEphyra analyzer

11.5

11.5

7

6.5

19.5

Google translate

14

12

8

6

17.5

SDL free

11

10.5

3.5

5

17.5

SYSTRANet

10.5

10

5

4.5

12

Babel Fish

9

8.5

3

3.5

14.5

MS live translate

9

9

2.5

3.5

12.5

Worldlingo

10

10.5

4

3

14

- 32 -

By evaluating the results in Table 4.2 and Table 4.3, six research findings are found. F1 The quality of translation and with respect to the quality of question classification, both have positive influence on question answering. The quality of the translation has a direct positive influence on the performance of the QA system to provide correct answers. Google translate performs remarkably better than any other MT system (see Section 3.1), which means that the translation from Google translate perform better than the translation from any other MT system for any given question classification. The effect of question classification is minimized on the performance of the QA system, if the class NErest is used. If every question is classified as NErest (no meaningful class), the QA system would solely depend on the quality of the translated question. Worldlingo stands out in a negative way, because it scores most disappointingly for both tables, if the scores for NErest are considered, while Worldlingo performs relativity well in Machine translation (see Table 4.1). A possible explanation is that Worldlingo performs well on function words, which have little lexical meaning, but it has difficulties in translating content words, which do have lexical meaning, correctly. The BLEU and NIST scores do not assign separate weights for function or content words, resulting in a score that is not really valid (measure what you want to measure). Regarding the influence of question classification on the performance of the QA system, we can conclude that a better classification results in a higher QA score, if the results from the OpenEphyra classifier are neglected (See also F2).

F2 The native classifier in Open Ephyra cannot be compared directly to the other classifiers. Of all classifiers, the native classifier in OpenEphyra scores the best, even better than the gold standard. This performance is even more remarkable, if we consider that all classifiers analyse the “perfect” questions in Dutch, while the classifier of Open Ephyra uses “imperfect” translations of MT systems or human as input. It is not clear why the native classifier performs considerably better than other classifiers. One possible explanation is that the other classifiers perform disappointingly for CLEF 06, compared to CLEF 03 (see Figure 4.1). Only 64% of all 200 questions are classified - 33 -

correctly by the classifier based on regular expressions. Even the gold standard scores a mere 77.5% for question classification. While the gold standard and the regular expression based classifier work separately from OpenEphyra, the OpenEphyra classifier likely works together with other processes in OpenEphyra, which improve the performance of the QA system. Moreover, the native classifier probably uses another list of expected answer types, which is more extensive than the list used in this research, including solutions for ‘how’ and ‘why’ type of questions. Thus, in further analysis and discussion, the scores of the native classifier are neglected.

F3 Combination of Google Translate and gold standard performs the best. The best machine translation system combined with the gold standards returns the best result. This result is in line with our expectations, while Google translate even outperforms human translation in QA.

F4 Exact matches show similar results as partial matches The scores of exact matches are relatively the same as the scores of partial matches. Google translate still performs the best, while BabelFish performs worst overall. The performances of the MT systems in between are very close to each other, while the ranking is still the same.

F5 No considerable difference in results between NEperson and NErest. While 20% of all questions for CLEF 06 can be classified as NEperson, the percentage for questions, classified as NErest, is 18.5%. Due to this small difference, the performance of both classes in QA is comparable. It is possible that NErest classes are converted to the majority class (NEperson), since NErest is not a meaningful class.

F6 Regular expressions perform well with respect to gold standard. This result is in line with our expectations. Still, the overall results are rather disappointing with the best combination of Google translate and gold standard classes scoring only 12% for exact correct answers. - 34 -

4.4 Answers to the research questions Having analyzed the QA results, answers to the research questions are proposed in this section. RQ1: What is the influence of machine translation on question answering? Improving the quality of machine translation has a positive effect on question answering. Better translation results in more correct answers. According to the results in Section 4.1, the performance of Google Translate for Dutch to English is very good, in comparison with human translation. The sentence structure is fluent and the meaning is preserved in most cases. In Chapter 1, MT was seen as a difficult task, but it seems that Google has overcome some of the obstacles and it has delivered an unexpectedly good performance. Improvements can still be made for machine translation, but these will only be marginal improvements. Because the performance of human translated questions on question answering can be seen as the highest possible, although the overall performance for QA is still below par, ‘perfect’ machine translation will only have a small influence on improving QA. In order to improve the performance of QA, we have to look beyond the scope of MT. RQ2: Will the performance of a multi-lingual question-answer system improve by incorporating meta data, extracted from the question in the source language, along with machine translation? Adding extra information from the source question to the QA system will certainly have a positive effect on the performance of the QA system. In this research, this extra information is represented by the expected answer type, making clear which type of answer is required, e.g. location, time, person, etc. By providing meaningful classes to the QA system, the amount of correct answers found almost doubles in comparison when adding non-meaningful classes. However, the overall performance of the QA system is still disappointing. Even the combination of human translated questions and gold standard classes results in poor results. The inability of QA systems to cope with ‘why’ and ‘how’ questions remains problematic. In future research, we can look at other types of meta data: what other information about a question can be relevant for QA?

- 35 -

- 36 -

Chapter 5 Conclusions The study focuses on multi-lingual question answering, in which answers are provided to questions that are posed in another language. Search engines are powerful tools in finding information, but the information is often presented in thousands or millions of links. It is not practical to read all the links due to information overload. By posing specific questions to a QA system, the system will return specific answers to a question. This is especially an attractive application for mobile devices, which feature a limited screen size. Reading large amounts of texts can be difficult, so only the question and the answer are displayed. When a person looks for information that is not available is his native language and this person does not understand the foreign language, it is difficult to satisfy his information need. Machine translation can translate the whole document, but still, reading the whole document can be time intensive. By posing specific questions in the native language, the system returns an answer, which is hopefully understandable to the user. Note that, English words are often recognized by most users, but Asian language words are seldom known to the Western world. Multi-lingual question answering is a relatively new development in IT. The performance is still disappointing, so this study tries to incorporate information about the source question into QA, because MT might not return perfect translated questions. This extra information consists of expected answer types, which make clear what type of answer is expected from the QA system, like location, person, time etc. A question analyzer is used to extract the question class from a question. In this research, the questions are provided in Dutch, while the answers are given in English. All questions are provided by CLEF 06. Three experiments are designed in order to determine the influence of this extra information and the quality of MT on QA. The first experiment focuses on machine translation, using six online MT systems and comparing the output to human translations. The second experiment consists of testing several question analyzers and compare the output to human assigned question classes (gold standard). The final - 37 -

experiments uses the output of the first and second experiment as input to perform question answering. The results show that machine translation and question classification have a positive, but moderate influence on question answering. Google Translate, one of the MT systems used in this study, performs very well. A better translation results in more correct answers to a certain extent. The effect of question classification on QA is present, however the native question classifier of the QA system scores significantly higher. It must be noted though, that an internal question classifier is far more integrated in a QA system than the external classifier used in this study. The internal classifier can interact with other processes in a QA system, resulting in more correct answers. Combining the influence of machine translation and question classification, we can conclude that both are crucial contributors in the process of multi-lingual question answering. Machine translation has matured into useful products. Question classification remains a difficult, but interesting topic for future research. Although the final results are not as good as expected, this study provides a usable foundation for future research.

Discussion and future research An obstacle that remains is the inability of question analyzers to cope with questions, which cannot be classified at all or cannot be assigned to meaningful classes, due to limitations of the analyzers. By reducing the number of questions classified into meaningless classes, better results might be obtained. This study only focuses on extracting expected answer types from named entities in a question, but neglected the corresponding verb in a question. Compare the two questions: 1. Which famous book was written by Roald Dahl in 1982? 2. What did Roahl Dahl wrote in 1982?

- 38 -

Both sentences ask for the name of a book, which is The Big Friendly Giant. In the first sentence, it is clear that ‘book’ refers to the class NEbook. However in the second sentence, it is not clear to the classifier what class can be assigned here. Since no known named entity can be found, this question is classified as NErest. If the question analyzer can perform a more thorough analysis, recognizing verbs and assign a meaningful tag to this question, this question can be classified to a class that is related to literature. By adding other extra information of a question to a QA system, the system might perform better. It is difficult for certain language pairs, (languages spoken by very few people) to apply machine translation, because there is no or very few data (bilingual corpora) available. In this case, it might be practical to implement interlingua as a intermediating language for rare language pairs. This way, it is theoretically possible to translate every language to any other language. It must be noted that this is a very costly and time intensive approach.

- 39 -

- 40 -

Reference list Brown P.F., Cocke J., Della Pietra S.A., Della Pietra V.J., Jelinek F., Lafferty D.F., Mercer R.L., Roosin P.S., (1990) “A statistical approach to machine translation.” Computational linguistics, 16(2), pp 79-85. Doddington G., (2002) “Automatic Evaluation of Machine Translation Quality using Ngram Co-occurrence Statistics.” In Proceedings of the ARPA Workshop on Human Language Technology 2002 Kraaij W., Nie J.Y., Simard M., (2003) “Embedding Web-Based Statistical Translation Models in Cross-language Information Retrieval” Association for Computational Linguistics, pp 381-419. Horwood E., (1986) “Machine Translation: past, present, future” New York Halsted Press, Chapter 9 Hutchins W.J., Somers H.L., (1992) “An introduction to Machine Translation” London academic press limited Jurafsky D., Martin J., (2008) “Speech and Language Processing: An introduction to natural language processing” Prentice-Hall Lyman P., Varian H., (2003) “How much information?” University of California Manning C., Raghavan P., and Schutze H., (2008) “Introduction to Information Retrieval.” Cambridge University Press Mariño J.B., Banchs R.E., Crego J.M., de Gispert A., Lambert P., Fonollosa J.A.R., Costajussè M.R., (2006) “N-gram-based Machine Translation” Association for Computational Linguistics, pp 527-549.

- 41 -

Peters C., Braschler M., (2002) “The Importance of Evaluation for Cross-language System Development: the CLEF Experience” Proceedings of the Third International Conference on Language Resources and Evaluation Ponte J.M., Croft W. B., (1998) “ A language modeling approach to information retrieval” Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval, pp 275-281. Papineni K., Roukos S., Ward T., Zhu W.J., (2001) “Bleu: a Method for Automatic Evaluation of Machine Translation.“ Research Report RC22176, IBM Research Division, Thomas J. Watson Research Center Schlaefer N., Gieselmann P., Schaaf T., Waibel A., (2006) “A pattern learning approach to question answering within the Ephyra Framework” In Proceedings of the Ninth International Conference on TEXT, SPEECH and DIALOGUE (TSD) Schlaefer N., Gieselmann P., Sautter G., (2006) “The Ephyra QA System at TREC 2006” In Proceedings of the Fifteenth Text REtrieval Conference (TREC), 2006 Schlaefer N., Ko J., Betteridge J., Sautter G., Pathak M., Nyberg E., “Semantic Extensions of the Ephyra QA System for TREC 2007” In Proceedings of the Sixteenth Text REtrieval Conference (TREC) van Zaanen M., (2008) “Multi-lingual Question Answering using OpenEphyra” Working Notes for the CLEF 2008 Workshop, Tilburg University

- 42 -

Appendix 1 – Expected answer types NEacronym (wat|welk|welke|noem|geef|vertel) (.*)afkorting NEangle (wat|welk|welke|noem|geef|vertel) (.*)hoek NEbirthstone (wat|welk|welke|noem|geef|vertel) (.*)geboortesteen NEbodyPart (wat|welk|welke|noem|geef|vertel) (.*)(lichaamsdeel|ledemaat|arm|been|orgaan|deel van het lichaam|onderdeel van het lichaam) NEcauseOfDeath (wat|welk|welke|noem|geef|vertel) (.*)(hoe stierf|doodsoorzaak) NEcolor (wat|welk|welke|noem|geef|vertel) (.*)(kleur|tint) NEcreature->NEanimal->NEbird (wat|welk|welke|noem|geef|vertel) (.*)vogel NEcreature->NEplant->NEFlower (wat|welk|welke|noem|geef|vertel) (.*)(bloeien|bloem|bloesem) NEcrime (wat|welk|welke|noem|geef|vertel) (.*)(misdaad|delict|vergrijp|misdrijf|strafbaar feit) NEdate wanneer NEdate->NEday (wanneer|wat|welk|welke|noem|geef|vertel) (.*)(dag|datum|geboortedag|jarig) NEdate->NEcentury (wanneer|wat|welk|welke|noem|geef|vertel) (.*)eeuw NEdate->NEdecade (wanneer|wat|welk|welke|noem|geef|vertel) (.*)decennium NEdate->NEmonth (wanneer|wat|welk|welke|noem|geef|vertel) (.*)maand NEdate->NEseason (wanneer|wat|welk|welke|noem|geef|vertel) (.*)(getijde|seizoen) NEdate->NEweekday (wanneer|wat|welk|welke|noem|geef|vertel) (.*)(dag van de week|werkdag) NEdate->NEyear (wanneer|wat|welk|welke|noem|geef|vertel) (.*)jaar NEdisease (wat|welk|welke|noem|geef|vertel) (.*)(abnormaal|achterlijk|afwijking|allergie|angst|anomalie|arbeidsongeschikt|fobie|ha ndicap|infectie|invalide|kanker|klacht|kwaaltje|kwelling|leukemie|pijn|medische conditie|misvorming|stoornis|syndroom|tumor|verminking|ziekte) NEdrug->NEmedicinal (wat|welk|welke|noem|geef|vertel) (.*)(medicijn|farmacon|geneesmiddel) NEdrug->NEnarcotic (wat|welk|welke|noem|geef|vertel) (.*)(narcose|anesthesie) NEdrug->NEvaccine (wat|welk|welke|noem|geef|vertel) (.*)(vaccin|vaccinatie|immuum) NEduration hoe (lang|oud|kort|jong) NEduration->NEdays hoeveel dagen NEduration->NEyears hoeveel jaren NEduration hoeveel (slaap|tijd) NEduration hoeveel (eeuw|dag|decennium|uur|minuut|maand|.*seconde|week|jaar|jaren) NEduration (wat|welk|welke|noem|geef|vertel) (.*)(leeftijd|duur|levensverwachting|spanne|spanwijdte|periode|winning streak|losing streak|wachttijd) NEfood (wat|welk|welke|noem|geef|vertel) (.*)(voedsel|voeding|eten|maaltijd|brood|granen|aardappelen}rijst|pasta|peulvruchten |zuivel|vlees|vis|ei) - 43 -

NEfood->NEfruit (wat|welk|welke|noem|geef|vertel) (.*)(appel|bes|kers|fruit|druif|limoen|citroen|meloen|sinaasappel|perzik|pruim) NEfrequency hoe (vaak|frequent) NEfrequency hoeveel (hertz|keer|keren) NEfrequency (wat|welk|welke|noem|geef|vertel) (.*)frequentie NElanguage (wat|welk|welke|noem|geef|vertel) (.*)(dialect|taal) NElegalSentence (wat|welk|welke|noem|geef|vertel) (.*)(oordeel|uitspraak|vonnis|uitkomst|proces|zaak|rechtsgeding|geding|verhoor|vero ordeling|straf) NElocation waar NElocation (wat|welk|welke|noem|geef|vertel) (.*)(bestemming|plaats|lokatie|oorsprong|punt|positie|terrein|stip) Nelocation->NEAirport (wat|welk|welke|noem|geef|vertel) (.*)(luchthaven|luchthavens|vliegveld|vliegbasis|helihaven) NElocation->NEcity (wat|welk|welke|noem|geef|vertel) (.*)(stad|metropolis|dorp) NElocation->NEcity->NEcapital (wat|welk|welke|noem|geef|vertel) (.*)hoofdstad NElocation->NEcontinent (wat|welk|welke|noem|geef|vertel) (.*)continent NElocation->NEcountry (wat|welk|welke|noem|geef|vertel) (.*)(kolonie|land|rijk|overheid|natie) NElocation->NEcounty (wat|welk|welke|noem|geef|vertel) (.*)graafschap NElocation->NEhemisphere (wat|welk|welke|noem|geef|vertel) (.*)(halfrond|hemisfeer) NElocation->NEisland (wat|welk|welke|noem|geef|vertel) (.*)eiland NElocation->NEmountain (wat|welk|welke|noem|geef|vertel) (.*)berg NElocation->NEmountainRange (wat|welk|welke|noem|geef|vertel) (.*)(berg|bergketen|bergactige rand) NElocation->NEnationalPark (wat|welk|welke|noem|geef|vertel) (.*)(nationaal park|nationale park) NElocation->NEpeninsula (wat|welk|welke|noem|geef|vertel) (.*)schiereiland NElocation->NEplanet (wat|welk|welke|noem|geef|vertel) (.*)planeet NElocation->NEprovince (wat|welk|welke|noem|geef|vertel) (.*)provincie NElocation->NEreef (wat|welk|welke|noem|geef|vertel) (.*)rif NElocation->NEstate (wat|welk|welke|noem|geef|vertel) (.*)staat NElocation->NEstreet (wat|welk|welke|noem|geef|vertel) (.*)(straat|steeg|laan|baan|weg|avenue|ringweg|boulevard|snelweg|rijweg|route) NElocation->NEwater (wat|welk|welke|noem|geef|vertel) (.*)water NElocation->NEwater->NEcanal (wat|welk|welke|noem|geef|vertel) (.*)(kanaal|kanalen|vaart|gracht) NElocation->NEwater->NElake (wat|welk|welke|noem|geef|vertel) (.*)(meer|oever|zeearm) NElocation->NEwater->NEriver (wat|welk|welke|noem|geef|vertel) (.*)rivier NElocation->NEwater->NEsea (wat|welk|welke|noem|geef|vertel) (.*)zee NElocation->NEwater->NEsea->NEocean (wat|welk|welke|noem|geef|vertel) (.*)oceaan NEmaterial->NEchemicalElement (wat|welk|welke|noem|geef|vertel) (.*)element NEmaterial->NEmetal (wat|welk|welke|noem|geef|vertel) (.*)(aluminium|brons|chroom|koper|goud|lood|magnesium|metaal|ijzer|zilver|staal|tin ) NEmaterial->NEmineral (wat|welk|welke|noem|geef|vertel) (.*)mineraal - 44 -

NEmaterial->NEstone (wat|welk|welke|noem|geef|vertel) (.*)steen NEmedicalTreatment (wat|welk|welke|noem|geef|vertel) (.*)(medische behandeling|behandelingsmethode|behandelmethode) NEmedicalTreatment->NEtherapy (wat|welk|welke|noem|geef|vertel) (.*)(therapie|accupunctuur) NEmoney hoe (arm|rijk) NEmoney (hoe duur|hoeveel geld) NEmoney hoeveel (kost|kosten) NEmoney (hoeveel|in) (.*)(dollar|euro|yen) NEmoney (hoeveel||wat|welk|welke|noem|geef|vertel) (.*)(kost|verdienen|verdient|bedrag|prijs|winst|salaris|verkoop|uitgave|belasting|boet e|tol|waarde|waard) NEmusicalInstrument (wat|welk|welke|noem|geef|vertel) (.*)(harmonika|accordeon|cello|klarinet|contrabas|trommel|drum|drumstel|fluit|gitaar |harp|hoorn|instrument|hobo|piano|saxofoon|trompet|altviool|viool|tuba) NEnationality (wat|welk|welke|noem|geef|vertel) (.*)(nationaliteit|burgerschap) NEnumber hoeveel NEnumber (wat|welk|welke|noem|geef|vertel) (.*)(hoeveelheid|aantal|capaciteit|dodencijfer|nummer|populatie|dosis|kwantiteit|getal |getallen) NEnumber->NEordinal (wat|welk|welke|noem|geef|vertel) (.*)ordinaal getal NEnumber->NEphoneNumber (wat|welk|welke|noem|geef|vertel) (.*)telefoonnummer NEnumber->NEzipcode (wat|welk|welke|noem|geef|vertel) (.*)postcode NEpathogen->NEbacteria (wat|welk|welke|noem|geef|vertel) (.*)bacterie NEpathogen->NEvirus (wat|welk|welke|noem|geef|vertel) (.*)(virus|parasiet) NEpercentage hoeveel procent NEpercentage welk deel van NEpercentage wat|welk|welke|noem|geef|vertel) (.*)(fractie|procent|percentage|portie|belasting) NEproperName->NEaward (wat|welk|welke|noem|geef|vertel) (.*)(accolade|award|beloning|certificatie|eer|eremetaal|ereteken|huldiging|medaille|pl ak) NEproperName->NEbook (wat|welk|welke|noem|geef|vertel) (.*)(gedicht|boek|boeken|boekwerk|literatuur) NEpoperName->NEdrama->NEfilm (wat|welk|welke|noem|geef|vertel) (.*)film NEpoperName->NEdrama->NEplay (wat|welk|welke|noem|geef|vertel) (.*)(toneel|toneelvoorstelling|theater) NEpoperName->NEdrama->NEplay->NEmusical (wat|welk|welke|noem|geef|vertel) (.*)musical NEpoperName->NEdrama->NEshow (wat|welk|welke|noem|geef|vertel) (.*)(show|programma|concert) NEproperName->NEethnicGroup (wat|welk|welke|noem|geef|vertel) (.*)(ethnisch|etnische (achtergrond|groep)|ras|stam) NEproperName->NEevent->NEcompetition (wat|welk|welke|noem|geef|vertel) (.*)competitie NEproperName->NEevent->NEconflict (wat|welk|welke|noem|geef|vertel) (.*)(bevrijding|conflict|crisis|gevecht|kruistocht|oorlog|opstand|slachtpartij|rebel|revo lutie|verovering|verzet) - 45 -

NEproperName->NEevent->NEfestival (wat|welk|welke|noem|geef|vertel) (.*)(.*festival|carnaval) NEproperName->NEmusic->NEopera (wat|welk|welke|noem|geef|vertel) (.*)opera NEproperName->NEmusic->NEsong (wat|welk|welke|noem|geef|vertel) (.*)(ballade|lied|melodie|.*zang|.*muziek|symfonie) NEproperName->NEmusic->NEsong->NEanthem (wat|welk|welke|noem|geef|vertel) (.*)(volkslied|nationaal symbool) NEproperName->NEorganization (wat|welk|welke|noem|geef|vertel) (.*)(luchtvaartmaatschappij|vliegtuigmaatschappij|dienst|alliantie|samenwerkingsverb and|vereniging|bedrijf|organisatie|beheer|gezag|band|bank|raad|merk|brouwerij|broe derschap|bureau|casino|centrum|kerk|kanaal|clinic|club|coalitie|commissie|netwerk|s cheepvaartmaatschappij|afdeling|directie|divisie|werkgever|onderneming|fabriek|fede ratie|stichting|fonds|gilde|ziekenhuis|hotel|instituut|laboratorium|producent|fabrikant |museum|aanbieder|maatschappij|sponsor|eskader|winkel|syndicaat|consortium|unie| vakgroep|vakbond|wijngaard) NEproperName->NEorganization->NEeducationalInstitution (wat|welk|welke|noem|geef|vertel) (.*)(college|.*school|.*onderwijs|universiteit) NEproperName->NEorganization->NEministry (wat|welk|welke|noem|geef|vertel) (.*)(ministerie|kabinet|departement) NEproperName->NEorganization->NEnewspaper (wat|welk|welke|noem|geef|vertel) (.*)(courant|.*krant|dagblad) NEproperName->NEorganization->NEpoliticalParty (wat|welk|welke|noem|geef|vertel) (.*)partij NEproperName->NEorganization->NEradioStation (wat|welk|welke|noem|geef|vertel) (.*)(radiostation|radio|omroep) NEproperName->NEorganization->NEteam (wat|welk|welke|noem|geef|vertel) (.*)(team|groep) NEproperName->NEorganization->NEtvChannel (wat|welk|welke|noem|geef|vertel)(.*)(televisiekanaal|televisiezender) NEproperName->NEperson (wie|van wie|aan wie|voor wie) NEproperName->NEperson (wat|welk|welke|noem|geef|vertel) (.*)(geboortenaam|bijnaam|volledige naam|originele naam|echte naam) NEproperName->NEperson (wat|welk|welke|noem|geef|vertel) (.*)(abolitionist|avonturier|apostel|arhitect|artiest|huurmoordernaar|astronaut|tante| bisschop|aartsbisschop|bokser|broer|bouwer|jongen|kandidaat|kapiteitn|CEO|voorzitt er|kampioen|kanselier|karakter|president|directeur|meester|kind|choreograaf|coach|t rainer|komiek|bevelhebber|commodore|concurrent|componist|congreslid|mededinger |cosmonaut|COO|schepper|maker|danser|dochter|verdediger|defensor|ontwerper|insp ecteur|rechercheur|ontwikkelaar|dictator|volgeling|leerling|ontdekker|docter|keizer|k oning|heerser|werknemer|vijand|ingenieur|entertainer|ontdekkingsreiziger|vader|fem inist|oprichter|vriend|generaal|heer|meisje|meid|god|golfspeler|gouverneur|afgestude erde|grootvader|grootmoeder|oma|opa|man|gast|hoofd|held|heldin|gegijzelde|gijzelaa r|echtgenoot|imam|individueel|infielder|uitvinder|rechter|moordenaar|vrouw|dame|a dvocaat|leider|legionair|verliezer|manager|vriend|burgemeester|lid|minister|moderat or|gespreksleider|monarch|staatshoofd|moeder|musikant|neef|nicht|ambtenaar|tegen stander|opponent|bezitter|eigenaar|schilder|partner|pastoor|uitvoerder|performer|pe rsoon|personeel|filosoof|fysicus|piloot|copiloot|werper|speler|eiser|dichter|politicus|paus|professor|hoogleraar|aanklager|psyc - 46 -

hiater|psycholoog|pop|robot|ontvanger|bewaarder|beschermer|wachter|rector|refere nt|journalist|commentator|vertegenwoordiger|onderzoeker|wetenschapper|secretaris| zender|zanger|zus|slachter|zoon|ster|student|docent|leraar|tenor|terrorist|trainer|oo m|vice-president|slachtoffer|echtgenote|eega|winnaar|getuige|ooggetuige) NEproperName->NEperson->NEactor (wat|welk|welke|noem|geef|vertel) (.*)(.*acteur|.*actrice|hoofdrolspeler|hoofdrolspeelster|hoofdrol) NEproperName->NEperson->NEauthor (wat|welk|welke|noem|geef|vertel) (.*)(schrijver|auteur) NEproperName->NEperson->NEdirector (wat|welk|welke|noem|geef|vertel) (.*)(regisseur|filmregisseur) NEproperName->NEperson->NEmathematician (wat|welk|welke|noem|geef|vertel) (.*)(wiskundige|mathemaat|mathematicus) NEproperName->NEperson->NEplaywright (wat|welk|welke|noem|geef|vertel) (.*)(dramaticus|toneelschrijver|dramaturg|scenarioschrijver|scenarist) NEproperName->NEperson->NEfirstName (wat|welk|welke|noem|geef|vertel) (.*)(eerste naam|voornaam|doopnaam|tussennaam|christelijke naam) NEproperName->NEperson->NElastName (wat|welk|welke|noem|geef|vertel) (.*)(familienaam|achternaam) NEproperName->NEperson->NEusPresident (wat|welk|welke|noem|geef|vertel) (.*)amerikaanse president NEproperName->NEperson->NEscientist (wat|welk|welke|noem|geef|vertel) (.*)(wetenschapper|onderzoeker|geleerde|hoogleraar) NEproperName->NEstadium (wat|welk|welke|noem|geef|vertel) (.*)(stadion|amfitheater) NEprofession (wat|welk|welke|noem|geef|vertel) (.*)(ambacht|aanstelling|baan|beroep|dienstbetrekking) NErank (hoe|waar|wat|welk|welke|noem|geef|vertel) (.*)(|positie|rang|rangorde|rank) NErange (wat|welk|welke|noem|geef|vertel) (.*)bereik NErate (wat|welk|welke|noem|geef|vertel) (.*)(ratio|verhouding) NErate->NEspeed hoe (snel|sloom|langzaam|vlot) NErate->NEspeed hoeveel (.*meter per (dag|uur|minuut|.*seconde|week|jaar|)km per uur|km/u) NErate->NEspeed (wat|welk|welke|noem|geef|vertel) (.*)(snelheid|versnelling|acceleratie) NErate->NEspeed->NEmph (wat|welk|welke|noem|geef|vertel) (.*)(mph|miles per hour|mijl per uur) NErelation (wat|welk|welke|noem|geef|vertel) (.*)(band|broederschap|connectie|familie|relatie|stamverwantschap|verwantschap) NEreligion (wat|welk|welke|noem|geef|vertel) (.*)(geestelijk|geloof|geloofsovertuiging|godsdienst|kerkelijk|religie|religieuze instelling|vroom) NEscore hoeveel (doelpunten|goals|punten) NEscore (wat|welk|welke|noem|geef|vertel) (.*)(eindscore|score|uitslag) NEsize hoe (groot|klein) NEsize (wat|welk|welke|noem|geef|vertel) (.*)grootte - 47 -

NEsize->NEarea (hoe groot in|hoeveel) (.*)(acre|hectare|vierkante .*meter) NEsize->NEarea (hoeveel|wat|welk|welke|noem|geef|vertel) (.*)oppervlakte NEsize->NEarea->NEsquareMiles (hoeveel|wat|welk|welke|noem|geef|vertel) (.*)(square mile|square miles|vierkante mijl) NEsize->NElength hoe (dichtbij|diep|ver|hoog|lang|nauw|kort|wijd) NEsize->NElength (hoe groot in|hoeveel) (.*)(lichtjaren|.*meter) NEsize->NElength (hoeveel|wat|welk|welke|noem|geef|vertel) (.*)(hoogte|diepte|diameter|afstand|vlieghoogte|lengte|radius|breedte) NEsize->NElength->NEfeet (hoeveel|wat|welk|welke|noem|geef|vertel) (.*)(feet|foot|voet|vierkante voet|kubieke voet) NEsize->NElength->NEmiles (hoeveel|wat|welk|welke|noem|geef|vertel) (.*)(mile|miles||Engelse mijl) NEsize->NEvolume->NEliters (hoe groot in|hoeveel) (.*)(kubieke .*meter|liter|liters) NEsize->NEvolume->NEgallons (hoeveel|wat|welk|welke|noem|geef|vertel) (.*)(gallon|gallons|gal) NEsize->NEvolume->NEounces (hoeveel|wat|welk|welke|noem|geef|vertel) (.*)(ounce|ounces) NEsize->NEvolume (hoeveel|wat|welk|welke|noem|geef|vertel) (.*)volume NEsize->NEweight hoe (licht|zwaar) NEsize->NEweight (wat|welk|welke|noem|geef|vertel) (.*)(gewicht|weging|massa) NEsize->NEweight->NEgrams hoeveel (.*gram|.*grammen) NEsize->NEweight->NEpounds hoeveel (.*pond|.*pound) NEsize->NEweight->NEtons hoeveel (.*ton|.*tons) NEsocialTitle (wat|welk|welke|noem|geef|vertel) (.*)(sir|madam) NEsocialTitle->NEmilitaryRank (wat|welk|welke|noem|geef|vertel) (.*)(rank|admiraal|Luitenantadmiraal|generaal|kapitein|majoor|kolonel|luitenant|commandeur|sergeant) NEsocialTitle->nobleTitle (wat|welk|welke|noem|geef|vertel) (.*)(ridder|jonkheer|jonkvrouw|baron|burggraaf|graaf|markies|hertog|prins) NEsocialTitle->policeRank (wat|welk|welke|noem|geef|vertel) (.*)(hoofdcommissaris|commissaris|hoofdinspecteur|inspecteur|brigadier|hoofdagent| agent|surveillant|aspirant|adspirant) NEsport (wat|welk|welke|noem|geef|vertel) (.*)(wedstrijd|olympische discipline|olympisch onderdeel|sport) NEstyle (wat|welk|welke|noem|geef|vertel) (.*)(stijl|vorm|soort|type) van (architectuur|kunst|kunstartiest|ballade|lied|melodie|gezang|zang|zanger|dans|danser |tekenen|tekening|literatuur|film|muziek|musical|opera|schilderen|schilder|schilderij|t oneel|toneelvoorstelling|sculptuur|beeld|show|zingen|symfonie|theater|schrijver|gesc hrift|kalligrafie|schrijfkunst) NEtemperature hoe (koud|warm|heet) NEtemperature hoeveel (graden|celcius|fahrenheit|kelvin) NEtemperature (wat|welk|welke|noem|geef|vertel) (.*)(smeltpunt|kookpunt|temperatuur) NEtime wanneer NEtime hoe (vroeg|laat) - 48 -

NEtime (wat|welk|welke|noem|geef|vertel) (.*)tijd NEtime->NEhour (wat|welk|welke|noem|geef|vertel) (.*)uur NEtime->NEdaytime (wat|welk|welke|noem|geef|vertel) (.*)overdag NEtimezone (wat|welk|welke|noem|geef|vertel) (.*)(tijdzone|GMT|UTC|Greenwich Mean Time|Coordinated Universal Time) NEunit->NEcurrency (wat|welk|welke|noem|geef|vertel) (.*)(valuta|geld|koers|wisselkoers|munteenheid) NEurl (wat|welk|welke|noem|geef|vertel) (.*)(domeinnaam|homepage|URL|internetadres|internetsite|webades|website) NEzodiacSign (wat|welk|welke|noem|geef|vertel) (.*)(sterrenbeeld|dierenriem)

- 49 -

Appendix 2 – CLEF 06 Dutch questions Wat is Atlantis? Wat is Hubble? Welk land werd binnengevallen door Irak in 1990? Waarheen verhuisde Freud in 1939? In welk land ligt Kashmir? Waar is de olie-tanker Prestige gezonken? Welk autobedrijf produceert Thunderbird en Mustang? Welk verdrag hebben Michail Gorbatsjov en Ronald Reagan in 1987 ondertekend? Welke symfonie heeft Beethoven gecomponeerd in 1824? Welke voetballer heeft in 1994 de Gouden Bal gewonnen? Wie speelt de hoofdrol in de film "Seven Years in Tibet"? Wie was de commandant van de Opstand van Warschau? Wie schreef de trilogie "In de ban van de ring"? Wat is de naam van de vrouw die als eerste de Mount Everest heeft beklommen zonder een zuurstofmasker. Wie werd de IJzeren Kanselier genoemd? In welk jaar vond de ramp in Tsjernobyl plaats? In welk jaar is Helmut Kohl geboren? In welk jaar is Titanic gezonken? Wanneer liep de eerste mens op de Maan? In welke stad is Wolfgang Amadeus Mozart geboren? Bij welke partij hoort Bill Clinton? Hoeveel mensen overleden tijdens de Pol Pot terreur? Waarvan werd O.J. Simpson beschuldigd? Wie is Stephen Hawking? Wat is de Yakuza? Wat is Euro-Disney? Wat is Hapkido? Waar werden de Olympische Winterspelen 1994 gehouden? Welke milieuorganisatie werd in 1971 opgericht? Welk land werd onafhankelijk na een 30-jarige burgeroorlog? Hoeveel gouden medailles won Brian Goodel in de Pan-Amerikaanse Spelen van 1979? Welke atleet heeft 18 gouden medailles gewonnen en 21 wereldrecords gevestigd? Hoeveel wereldrecords voor mannen werden in 1995 gevestigd? Wat is het Top Quark? Wat zijn de drie fundamentele deeltjes in het Standaardmodel uit de deeltjesfysica? Waar is het Fermi National Accelerator Laboratory? Hoeveel onderzoekers waren er betrokken bij de 17-jarige zoektocht naar het Top Quark? Hoeveel passagiers vervoert het schip "Canadian Empress"? Waar werd het cruiseschip gebouwd? Wat is het volkslied van Beieren? Wie was de katholieke aartsbisschop van München in 1995? Hoeveel deelstaten heeft Duitsland? Wie heeft Minerva Rigging bij Kip Marina opgericht? Wie is de meest succesvolle zeiler van Groot-Brittannië? In welke amerikaanse staat ligt Everglades National Park? - 50 -

Hoe groot zijn Everglades? Wat is de enige van de zeven wereldwonderen die nog is overgebleven? Hoeveel jaren geleden werd de voortoren van Pharos gebouwd? Hoe hoog was de Pharos Vuurtoren? Noem monumenten op de Werelderfgoedlijst. Wie heeft de Shoemaker-Levy komeet ontdekt? Welke planeet heeft de Shoemaker-Levy komeet geraakt? Hoeveel kometen zijn er volgens wetenschappers? Welk percentage van alle verkochte melk wordt met ontbijtgranen gebruikt? Hoeveel russische steden hebben nu een oplossing voor luchtvervuiling nodig volgens de Wereldbank? Wat is Quinoa? Welke melk organisatie werd opgeheven in november 1994? Geef een lijst van dienstweigeraars. Uit welk land komen de Gurkhas? Van welke organisatie nam Willy Claes ontslag als secretaris-generaal in 1995? Hoeveel leden heeft de NAVO? Welk bedrijf nam Barings over na zijn instorting? Wie is Nick Leeson? Wie was de president van Frankrijk tijdens de testen van atoomwapens in de Zuidelijke Stille Oceaan? Welke landen bezitten officieel atoomwapens? In welk jaar was de landing in Normandië? Wie heeft D-Day "een grote kruistocht" genoemd? Welk legeronderdeel dat deel nam aan de D-Day landing had als motto "No mission too difficult, no sacrifice too great, nothing in hell can stop the 1st Division." Wie was Lin Piao? Waar werd het op een nest zittende Oviraptor fossiel ontdekt? Wat is lepra? Bij welke organisatie is Peter Anderson alcohol adviseur? Wie was Vermeer? Bij welk partij hoort Willy Brandt? Wat is de naam van Helmut Kohls partij? Noem een Brits bedrijf die radioactief afval vervoert. Wanneer was het Verdrag van Bazel inzake de beheersing van de grensoverschrijdende overbrenging van gevaarlijke afvalstoffen ondertekend? Wat werd beschreven als "een fout werktuig" en "de meest schitterend wanorde van de wereld"? Wie is Klaus Pohl? Wat is het langste Duitse woord? Wat is de nieuwe naam van Nicosia volgens de spellingswijzigingen in Cyprus? Wanneer werd Hubble Space Telescope gelanceerd? Waar ligt het Jet Propulsion Laboratory van NASA? Wanneer is de Galileo ruimtemissie voorspeld om te beëindigen? Wie is Kailash Satyarthi? Wanneer heeft Uttar Pradesh het akte van de afschaffing van de kinderarbeid goedgekeured? Wat zijn latifundios? In welke stad werd Vladislav Listyev vermoord? Wie heeft de opera "Het Meisje van Pskov" gecomponeerd? - 51 -

Wat betekent "Les Six" in de muziek? Hoe heet de Letse munt? Wie is de premier van Litouwen? Waarbij hoorde Nice tussen 1814 en 1860? Wie was de koning van Italie van 1900 tot 1946? Wat is de naam van de nationale luchtvaartmaatschappij van Italie? Wat is de afkorting voor de Federal Aviation Administration? Wat is zogturbulentie? Wat is de International Civil Aviation Organization? Wie is het hoofd van de Bank van Tokyo? Wanneer werd Cosmo Securities een deel van de Daiwa Bank? Wat is nintendo? Wat is het Verdrag van Dayton? Wat is GI Joe? Noem een production designer die een Oscar voor zijn 1987 film heeft gekregen. Wie zei "Zet talent in je vak en geest in je leven"? Wat is de OAU? Van welke organisatie is Sam Chisholm directeur? Wat is Reuters? Hoeveel zal de voorgestelde plan tegen watersnoodrampen in Perth kosten? Noem een professor van forensische geneeskunde in de Universiteit van Edinburgh. Waar heeft de "killer smog" van 1952 4000 sterfgevalen veroorzaakt? Wanneer heeft Indonesie East Timor geannexeerd? Wat is de belangrijkste religie van East Timor? Hoeveel huizen gaan worden gebouwd onder het Stirling initiatief tussen 1993 en 1998? Op welke airshow verongelukte een F-86 Mk 6 in 1993? Van welke organisatie is Velupillai Prabakaran het hoofd? Door welke organisatie werd Sri Lanka veroverd tussen 1987 en 1989? De voorgestelde zinking van welk bouwsel leidde tot een onenigheid op de G7 bijeenkomst in Halifax? Wat kwetst 11,000 mensen per jaar volgens Prevent Blindness America? Wat is "Sophie's World"? In welk jaar werd Graeme Obree benoemd tot Scotland's Sports Personality? In welk jaar won Aung San Suu Kyi de Nobelprijs voor de Vrede? Tijdens het orkaanseizoen, waar zijn Dugan and Janet Essick van plan om te blijven op hun jacht "Jeekers"? Wat is nandrolone? In welke stad werd de loper Ben Johnson positief getest op Stanozol tijdens de Olympische Spelen? Waar werkte Francis Cammaerts met de Maquis samen tijdens de Tweede Wereldoorlog? In welke regio van Schotland was de proportie van de opgespoord misdaden in de periode 1991-1993 31%? Op welke dag in de zomer begint de Ventura Community Services Department zijn zomers natuurprogramma? Vanaf welke tijd 's ochtends kun je wildbloemen in bloei zien in Mission Canyon tijdens het Wildbloemen Festival? Wat is de RLPO? Wie was Alexander Graham Bell? - 52 -

Wat is Lufthansa? Wat is Christie's? Wat is Eurovisie? Wie is de president van Letland? In welk jaar was het wereldkampioenschap voetbal in de Verenigde Staten? Op welke datum tekende Jordanië en Israël een vredesverdrag? In welk jaar stierf Bernard Montgomery? In welk jaar was de Russische Revolutie? In welke stad is de Johnson Space Center? In welke stad is het Sea World aquapark? Van welke organisatie is Michel Camdessus directeur? Van welke filmstudio was Cedric Gibbons art director? Hoeveel bewoners heeft Longyearbyen? Hoeveel Oscars hebben Star Wars gewonnen? Van welke dienst was Luis Roldan directeur van 1986 tot 1993? Welke oorlog vond van 1939 tot 1945 plaats? In welke sport won Europa van America in 1987? Welke landen vormen de Noord-Amerikaanse Vrijhandelsovereenkomst? Noem de drie levende Beatles. Wat is de hoofdstad van Tsjetsjenië? Welk land won de WK 1994? Wanneer viel de Berlijnse muur? Welke Zweedse muziekgroup lanceerde zijn eerst album in 1994? Waar was de eindstrijd van de WK 1994 gehouden? Op welke dag was de eerste aanval van IRA op de Heathrow luchthaven? Welke beroemde komeet was laatst gezien in de atmosfeer van de Aarde tussen 1985 en 1986? Noem alle luchthavens van Londen in Engeland. Wie is Boris Becker? Wat is het Internet? Hoeveel keren heeft Zinedine Zidane de US Open gewonnen? In welke film van Kevin Reynolds speelt Kevin Costner? Wat is Knesset? Welke voormalige Iranse president werd vermoord in 1991? Waar waren de Olympische Spelen van 1993 gehouden? Wat soort bommen gebruikte IRA in de aanval op Heathrow in 1994? Wat is een Chinook? Hoe oud was Roland Ratzenberger toen hij overleed? Welk bankbedrijf stortte in na de speculaties van Nick Leeson? Wat is sorghum? Wie is Fernando Henrique Cardoso? Wie was de president van de Deutsche Bank tijdens het faillissement van Juergen Schneider? Wat is de naam van de grootste gevangenis in San Paolo? Wie is Rolf Ekeus? Welke landen zijn leden van de Gulf Cooperation Council? Wat is de gemiddelde temperatuur op de Aarde? What is de chemische formule van kooldioxide? Aan wie was de Nobelprijs voor de Literatuur toegekend in 1995? Wat zijn fatwas? - 53 -

Met wie is Hazen Ames getrouwd? Welke boete kreeg Italie bij overproductie van de melkquotum? Welk filmfestival kent de Gouden Beer toe? Bij welke group hoort het autobedrijf Seat? Wie regisseerde "Caro diario"? Hoeveel oorlogs voerden India en Pakistan om Kashmir? Hoeveel keren werd Jackie Stewart een wereldkampioen? Welke staat heeft een wet goedgekeurd tegen de reclame van sigaretten? Wat is ecu? Hoeveel natuurgebieden heeft Costa Rica? Wat is de naam van de oprichter van Black Sparrow Press? What is APEC? Wat is de bijnaam van Eddy Merckx? Waar is Hermitage? Wie heeft het besturingssysteem OS/2 gecreërd? Wie bestuurde Sudan in 1899? Wie waren de Picts? Welke boete kreeg John Fashanu? Wat is de huidige naam van Ceylon? Welke oorlog duurde van 1865 tot 1870? Wie redde joden in de Tweede Wereldoorlog?

- 54 -

Appendix 3 – CLEF 06 English questions What is Atlantis? What is Hubble? Which country was invaded by Iraq in 1990? Where did Freud move to live in 1939? In which country is Kashmir? Where did the oil tanker "Prestige" sink? Which car manufacturer produces Thunderbird and Mustang? What treaty have Mikhail Gorbachev and Ronald Reagan signed in 1987? What symphony has Beethoven composed in 1824? Which football player was awarded with the "Golden Ball" in 1994? Who plays the leading role in the film "Seven Years in Tibet"? Who was the commander of the Warsaw Uprising? Who wrote the trilogy "Lord of the Rings"? What is the name of the woman who has climbed Mount Everest as first one without an oxygen mask. Who was called the Iron Chancellor? In what year did the Chernobyl disaster take place? In what year was Helmut Kohl born? In what year did Titanic sink? When did the first human walk on the Moon? Which city was Wolfgang Amadeus Mozart born in? Which Party is Bill Clinton member of? How many people have deceased during the Pol Pot terror? What was O.J. Simpson accused of? Who is Stephen Hawking? What is the Yakuza? What is Euro-Disney? What is Hapkido? Where were the Olympic Winter Games held in 1994? Which environmental organisation was founded in 1971? Which country has become independent after 30 years of civil war? How many gold medals has Brian Goodel won in the Pan-American Games of 1979? Which athlete has won 18 gold medals and established 21 world records? How many world records for men were established in 1995? What is the Top Quark? What are the three fundamental particles in the Standard Model of particle physics? Where is the Fermi National Accelerator Laboratory? How many researchers were involved in the 17-year quest for the Top Quark? How many passengers does the ship "Canadian Empress" carry? Where was the cruise ship built? What is the anthem of Bavaria? Who was the Catholic Archbishop of Munich in 1995? How many federal states are there in Germany? Who founded Minerva Rigging at Kip Marina? Who is the most successful sailor in Britain? In which American state is Everglades National Park? How big are Everglades? - 55 -

What is the only one of the seven wonders of the world that is still remaining? How many years ago was the tower of Pharos built? How high was the Pharos Lighthouse? List monuments on the World Heritage Site. Who has discovered the Shoemaker-Levy comet? What planet did the Shoemaker-Levy comet hit? How many comets are there according to scientists? What percentage of all milk sold is used with cereals? How many Russian cities now require a solution for air pollution according to the World Bank? What is Quinoa? Which milk organization was closed down in November 1994? Give a list of conscientious objectors. Which country are the Gurkhas from? Which organization did Willy Claes resign from as Secretary General in 1995? How many members does the NATO have? What company took over Barings after its fall? Who is Nick Leeson? Who was the president of France during the testing of atomic weapons in the South Pacific? Which countries have atomic weapons officially? In what year was the landing in Normandy? Who has called D-Day "a great crusade"? Which army that took part in the D-Day landing has the motto "No mission too difficult, no sacrifice too great, nothing in hell can stop the 1st Division." Who was Lin Piao? Where was the Oviraptor fossil seated on a nest discovered? What is leprosy? At which organization is Peter Anderson alcohol counselor? Who was Vermeer? What party does Willy Brandt belong to? What is the name of Helmut Kohl's party? Name a British company which transports nuclear waste. When was the Basel Convention on the Control of Transboundary Movements of Hazardous Wastes signed? What was described as "a mechanical error" and "the most beautiful disorder of the world"? Who is Klaus Pohl? What is the longest German word? What is the new name for Nicosia following spelling changes in Cyprus? When was the Hubble Space Telescope launched? Where is the Jet Propulsion Laboratory of NASA? When was the Galileo space mission predicted to finish? Who is Kailash Satyarthi? When did Uttar Pradesh approve the Act of abolition of child labor? What are latifundios? In which city was Vladislav Listyev murdered? Who has composed the opera "The Maid of Pskov"? What does "Les Six" mean in the music? What is the Latvian currency? - 56 -

Who is the Prime Minister of Lithuania? Where was Nice part of between 1814 and 1860? Who has been the king of Italy from 1900 to 1946? What is the name of the national airline of Italy? What is the abbreviation for the Federal Aviation Administration? What is wake turbulence? What is the International Civil Aviation Organization? Who is the head of the Bank of Tokyo? When did Cosmo Securities become part of the Daiwa Bank? What is nintendo? What is the Treaty of Dayton? What is GI Joe? Name a production designer who has received an Oscar for his 1987 film. Who said "Put your talent into your work, but your genius into your life"? What is the OAU? Which organization is Sam Chisholm director of? What is Reuters? How much will the proposed plan for flood disasters in Perth cost? Name a professor of forensic medicine at the University of Edinburgh. Where did the "killer smog" of 1952 cause 4000 deaths? When did Indonnesia take over East Timor? What is the main religion of East Timor? How many houses will be built under the initiative Stirling between 1993 and 1998? At what airshow did a F-86 Mk 6 crash in 1993? Which organization is Velupillai Prabakaran the head of? By which organization was Sri Lanka conquered between 1987 and 1989? The proposed sinking of which structure led to a disagreement on the G7 meeting in Halifax? What hurts 11,000 people per year according to Prevent Blindness America? What is "Sophie's World"? In what year was Graeme Obree appointed Scotland's Sports Personality? In what year did ung San Suu Kyi win the Nobel Prize for Peace? During the hurricane season, where were Dugan and Janet Dugan Essick intending to stay in their yaught "Jeekers"? What is Nandrolone? In which city was the runner Ben Johnson tested positive for Stanozol during the Olympic Games? Where did Francis Cammaert worked together with the Maquis during the Second World War? In which region of Scotland was the proportion of detected crimes in the period 19911993 31%? On what day in the summer does Ventura Community Services Department start with its summer nature program? From what time in the morning can you see wild flowers blooming in Mission Canyon at the Wild Flower Festival? What is the RLPO? Who was Alexander Graham Bell? What is Lufthansa? What is Christie's? What is Eurovision? - 57 -

Who is the president of Latvia? In which year was the World Cup in the United States? On which date did Jordan and Israel sign a peace treaty? In which year did Bernard Montgomery die? In which year was the Russian Revolution? In which city is the Johnson Space Center? In which city is the Sea World water park? Which organization is Michel Camdessus director of? Which movie studio was Cedric Gibbons art director of? How many residents are in Longyearbyen? How many Oscars has Star Wars won? Which service was Luis Roldan director of from 1986 to 1993? Which war took place from 1939 to 1945? In which sports did Europe win from America in 1987? Which countries form the North American Free Trade Agreement? List the three living Beatles. What is the capital of Chechnya? Which country won the World Cup 1994? When did the Berlin wall fell? Which Swedish music group launched its first album in 1994? Where was the final battle of World Cup 1994 held? On what day was the first IRA attack on Heathrow airport? Which famous comet was last seen in the atmosphere of the Earth between 1985 and 1986? List all airports of London in England. Who is Boris Becker? What is the Internet? How many times has Zinedine Zidane won the U.S. Open? In which film did Kevin Reynolds Kevin Costner play? What is Knesset? Which former Iranian president was assassinated in 1991? Where were the Olympic Games of 1993 held? What type of bombs did the IRA use for the attack at Heathrow in 1994? What is a Chinook? How old was Roland Ratzenberger when he died? Which bank collapsed after the speculation of Nick Leeson? What is sorghum? Who is Fernando Henrique Cardoso? Who was the president of the Deutsche Bank during the bankruptcy of Juergen Schneider? What is the name of the largest prison in San Paolo? Who is Rolf Ekeus? Which countries are members of the Gulf Cooperation Council? What is the average temperature on Earth? What is the chemical formula of carbon dioxide? To whom was the Nobel Prize for Literature awarded in 1995? What are fatwas? Who is Hazen Ames married to? What fine did Italy get for overproduction of the milk quota? Which film festival awards the Golden Bear? - 58 -

Which group is the car company Seat part of? Who directed "Caro diario"? How many wars have been fought between India and Pakistan for the possession of Kashmir? How many times did Jackie Stewart become world champion? Which state has a law approved against advertising cigarettes? What is ECU? How many nature reserves does Costa Rica have? What is the name of the founder of Black Sparrow Press? What is APEC? What is the nickname of Eddy Merckx? Where is Hermitage? Who has created the operating system OS/2? Who controlled Sudan in 1899? Who were the Picts? What penalty was John Fashanu given? What is the current name of Ceylon? What war has lasted from 1865 to 1870? Who saved the Jews during the Second World War?

- 59 -