Personalisation for researchers Nele Smeets Master Thesis MICC-IKAT 06-03
Thesis submitted in partial fulfilment of the requirements for the degree of Master of Science of Knowledge Engineering in the Faculty of General Sciences of the Universiteit Maastricht
Thesis committee: Prof. dr. H.J. van den Herik Prof. dr. E.O. Postma Dr. H.H.L.M. Donkers Drs. A.W.A. Spek
Universiteit Maastricht Institute for Knowledge and Agent Technology Maastricht ICT Competence Centre Maastricht, The Netherlands
28 March 2006
Preface
“Welcome to my thesis” is a sentence which was for a long time music in my mind. I would like to write a thesis that contributed to that part of performing research which, in my opinion, could gain the greatest benefit from support. The ideas of others gave me the opportunity to generalise my own ideas, and they helped them to be introduced into the field of research. Bringing this thesis to a good end turned out to be a greater challenge than I had expected. The music followed me for a considerable amount of time before I understood that its composition is supposed to be written note by note. The reader will find this principle again in this thesis1 . During the completion of my thesis, I was privileged to cooperate with many persons whom I would like to thank below. First, I would like to express my gratitude to professor Jaap van den Herik. His persistent encouragements inspired me to continue. The regular sessions I had with him equipped me with productive methods to organise my thoughts, and to put them to paper. We seem to share a similar interest in words, such as ‘dilapidated’, which would almost have not been part of this thesis. Next to him, Sander Spek acted as my daily advisor, in particular in the beginning he made the effort to read and comment on my writings. The long list of people with whom I cooperated starts with dr. Floris Wiesman and dr. Kees Jan van Dorp. They set me onto the tract of personalisation for researchers. Since my thesis contains a large amount of fieldwork, I would like to recognise the researchers who participated in the interviews: N.H. Bergboer, L. Braun, F. de Jonge, I.G. Sprinkhuizen-Kuyper, P.H.M. Spronk, and J.W.H.M. Uiterwijk. In particular, I would like to thank L. Braun, and F. de Jonge since they were also willing to take part in the evaluation of the prototype. The large number of names is an indication that I spent part of my time at the Faculty of General Sciences, in particular at IKAT, in a very fruitful way. Outside the academy, I first would like to thank my parents for their unvarnished advice and patience, and for practising their empathy. Finally, I would like to recall all friends and fellow students upon who I could reckon for revision and for spending joyful times. Nele Smeets Maastricht, March 28, 2006 1
The model supports the researcher step by step, iteration by iteration.
iii
Abstract
In academic research, we may distinguish junior researchers and senior researchers. Both spend a large amount of time to find relevant information. However, what is relevant for the junior researchers may not be relevant for the senior researchers. So, if we develop tools for the support of a researcher in finding relevant information we should distinguish between both groups. But, even within each group there are differences since each researcher may have his own personal project-related interests. The general challenge that researchers face is an information overload while searching, and an inadequate filtering while reducing the information overload. This thesis investigates how we can use personalisation in combination with filtering to arrive at a retrieval system that produces relevant and adequate information for the researcher involved. Therefore, the thesis investigates in the following problem statement. How can personalisation support the quest for information, as formulated by researchers? Through a series of interviews with researchers, we found out that they search for information for four reasons (which we call the search goals): (1) to obtain background knowledge, (2) to remain up to date with the field of research, (3) to become acquainted with related areas, and (4) to find specific information. To find out where personalisation might be applied, two main categories were investigated: (1) the literature study, and (2) knowledge sharing. The interviewees showed an explicit preference for personalised support on the literature study. Therefore, we focus on supporting the literature study with personalisation. For our research, we formulated three basic research questions: 1. What is involved in the personalisation of information? 2. Where do researchers desire personalised support in their search for information? 3. How can a personalised quest for information be achieved? To answer these questions, we designed a model and implemented a prototype. Experiments with the prototype lead to promising results. The main conclusion we may draw is that personalisation can support the quest for information (as formulated by researchers) by (1) exploiting sufficient information about a researcher’s project-related interests to suggest his personal set of information items, (2) providing an iterative search, and (3) enriching the description of the researcher’s current interest. Finally, five directions for further research are given. v
List of Figures
3.1
Need for knowledge cycle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
25
5.1 5.2 5.3 5.4
Screen shot: a user model of the prototype. . . . . Updating the memory given an evaluated item set. Calculation of the score for an item. . . . . . . . . Screen shot of a resultant set for the researcher . .
43 45 45 46
vii
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
Contents
List of Figures
vii
Contents
ix
1 Introduction 1.1 Motivation and challenges . . . . . . . . . 1.2 The context . . . . . . . . . . . . . . . . . 1.3 Problem statement and research questions 1.4 Thesis overview . . . . . . . . . . . . . . . 2 Factors on personalisation 2.1 What is personalisation? . . . . . . . . . . 2.1.1 A remedy to information overload 2.1.2 Definitions . . . . . . . . . . . . . 2.1.3 Related areas . . . . . . . . . . . . 2.2 User-related factors . . . . . . . . . . . . . 2.2.1 The user . . . . . . . . . . . . . . . 2.2.2 The system . . . . . . . . . . . . . 2.2.3 The acquisition . . . . . . . . . . . 2.2.4 The user model . . . . . . . . . . . 2.2.5 Related issues . . . . . . . . . . . . 2.3 Treasure-related factors . . . . . . . . . . 2.3.1 Representation . . . . . . . . . . . 2.3.2 Content . . . . . . . . . . . . . . . 2.3.3 Metadata . . . . . . . . . . . . . . 2.4 Personalisation approaches . . . . . . . . 2.4.1 Content-based approach . . . . . . 2.4.2 Collaborative approach . . . . . . 2.4.3 Extension-based approach . . . . . 2.4.4 Hybrid approaches . . . . . . . . . 2.5 Supportive domains . . . . . . . . . . . . 2.5.1 Data mining . . . . . . . . . . . . 2.5.2 Intelligent agents . . . . . . . . . . 2.6 An answer to the first research question . ix
. . . .
. . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . .
. . . .
. . . . . . . . . . . . . . . . . . . . . . .
. . . .
1 1 2 3 3
. . . . . . . . . . . . . . . . . . . . . . .
5 5 5 6 7 7 7 8 9 11 13 13 13 14 15 15 16 17 18 19 20 20 21 21
3 Fieldwork in academic research 3.1 Conducting the interviews . . . . . . . . . 3.1.1 The actors . . . . . . . . . . . . . 3.1.2 The interviews . . . . . . . . . . . 3.2 The quest for information . . . . . . . . . 3.2.1 Why do actors quest? . . . . . . . 3.2.2 Written-word resources . . . . . . 3.2.3 Spoken-word resources . . . . . . . 3.2.4 What happens with the treasure? . 3.3 Issues of personalising academic research . 3.3.1 Focus . . . . . . . . . . . . . . . . 3.3.2 Two requirements . . . . . . . . . 3.3.3 Effort . . . . . . . . . . . . . . . . 3.4 An answer to the second research question 4 Our model and its factors 4.1 Item-related factors . . . . . . . . 4.2 Researcher-related factors . . . . 4.2.1 The user . . . . . . . . . . 4.2.2 The system . . . . . . . . 4.2.3 The acquisition . . . . . . 4.2.4 The user model . . . . . . 4.3 The researcher’s treasures . . . . 4.3.1 Content-based approach . 4.3.2 Collaborative approach . 4.3.3 Extension-based approach 4.3.4 Our model . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
. . . . . . . . . . . . .
23 23 23 24 25 25 27 28 29 30 31 31 31 31
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
. . . . . . . . . . .
33 33 34 34 35 36 36 38 38 38 38 39
5 Testing our model 5.1 The prototype . . . . . . . . . . . . 5.1.1 The items . . . . . . . . . . 5.1.2 The user model . . . . . . . 5.1.3 A personalised resultant set 5.2 Conducting the experiments . . . . 5.3 Opinions and evaluation . . . . . . 5.3.1 Opinions on our model . . . 5.3.2 Evaluation of the prototype 5.3.3 Discussion . . . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
. . . . . . . . .
41 41 42 42 44 46 47 47 47 49
6 Conclusions and further research 6.1 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6.2 Further research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
51 51 52
References
53
A The A.1 A.2 A.3
interviews The interviewees . . . . . . . . . . . . The interview questions . . . . . . . . The interview answers . . . . . . . . . A.3.1 Answers of a junior researcher A.3.2 Answers of a senior researcher .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
61 61 61 65 65 69
B The B.1 B.2 B.3 B.4
evaluations Participants . . . . . . . . . The sets of items for testing Experiments . . . . . . . . . Opinions . . . . . . . . . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
77 77 77 78 80
. . . .
. . . .
. . . .
. . . .
. . . .
. . . .
Chapter 1
Introduction
This thesis deals with the retrieval of information for scientific researchers. More specifically, it aims at the provision of adequate information for each researcher personally. This means that personalisation plays a key role in our research. For finding information we use intelligent agents. The aim of this thesis is to model an intelligent agent that is capable of extracting the relevant information from a researcher and to use this information in order to provide him1 with the desired research information. “Welcome to my thesis” will probably be the only sentence in common that will be communicated between users and their personal assistants in a fully personalised world. The assistants preferably communicate information that is valuable to their users. For instance, when you are in a hurry, you would like to receive the summary of this thesis; when you are interested in a definition of personalisation, the definition is communicated and not the conclusions. Even if this complete thesis is marked as valuable for two users, they will receive considerably different information out of the thesis: for each of the users, the personal assistants will tailor the information relevant to the specific knowledge and preferences of the users involved. However, up to now, we do not live in a fully personalised world. Nowadays, researchers are taking the first steps towards personalised systems: systems that know you, that know what you want, what you need, or what you mean. The aim of the current research (as stated above) is to improve the quality of the interaction between a user and a system as if the system was custom made for the user involved. In this thesis, we investigate what such a research can do to support the quest for information by researchers themselves. The first chapter consists of four sections: motivation and challenges (1.1), the context (1.2), problem statement and research questions (1.3), and thesis overview (1.4).
1.1
Motivation and challenges
During research, a large part of the time is usually spent in the process of finding the requested information. That process ranges from spitting through too much information to finding the 1
In this thesis we use the male pronoun when both pronouns, male and female, are possible.
1
2
CHAPTER 1. INTRODUCTION
requested information just beside the target you are aiming at. Let us take, for a while, a look outside the domain of research, and widen the scope to find the right thing in general. Then we see a retrieval concept that helps users during their search. Huge online stores, for example Amazon2 , explicitly present items that are of interest for a certain user, instead of letting him search through their huge amount of items. Amazon preferably presents products that a particular user might want to buy. Another example is the MovieLens project3 , which recommends movies that are of interest for the requesting user. The retrieval concept also manifests itself in e-mail: the spam filter moves irrelevant messages to the junk mail folder. In other words, the selection of items is tailored to the needs of a specific user. In the framework of the retrieval concept we here meet the concept of personalisation. In our research, too, we encounter a huge amount of items, such as articles, that users – researchers – are searching for. Because of this similarity, we opt for personalisation to help improve the search for information by researchers. The challenges of our research are discussed in 1.3. They stem from the specific subject of this thesis: researchers. They are our users, who are searching for information. The challenging question is, what do we have to take into account to apply personalisation for researchers? Well, it is for sure that researchers are (potential) experts in certain areas; almost each project is oriented in their domains of expertise. Consequently, these domains can be interpreted as the context in which the search is performed during such a project. For example, a document on the collaboration of secret kgb agents in the cold war might be valuable for an historian, but not for a multi-agent expert, because the collaboration is not about “software agents”. Furthermore, researchers share knowledge during informal talks with colleagues, in conferences, and with project members. Hence, they use other researchers as a source of information. This knowledge sharing can benefit, too, from the application of personalisation and vice versa. The challenges are investigated through a series of interviews with six researchers, in which each of them gives his vision on the quest for information (see 1.4, and more in particular, chapter 3).
1.2
The context
As mentioned above, the subject of our research is “researchers who search for information during a project”. In order to perform the investigation accurately, we restrict the context of this thesis in three ways. First, we restrict ‘potential users’ to ‘academic researchers’. Other users, such as people from a research and development department in a company, may behave differently during their quest for information. For instance, assuming that they have to keep their research secret for competitive reasons, their knowledge sharing with others is restrained. The second restriction concerns the search domain of the researchers. Usually most of the time of searching is spent during the literature study. Therefore, our research domain is restricted to information that is valuable during that time. Moreover, we intend to search in 2 3
http://www.amazon.com/exec/obidos/tg/browse/-/508506/ref=hp hp ct 4 2/002-3192784-4898460 http://movielens.umn.edu/html/tour/index.html
1.3. PROBLEM STATEMENT AND RESEARCH QUESTIONS
3
articles and technical reports only. For comparison, movies, presentations, or tools that may be useful for a project, are not included. Furthermore, the information used in a research project has to be based on trustworthy information. Therefore, the information in the search domain has to be trustable too. For example, a random document on a site is usually not certified, while articles in scientific magazines are built upon other certified articles. In accordance to this observation, the scientific article is part of our search domain, whereas the random document is not. The third restriction is brought into existence to keep the focus on personalisation. There is a whole range of issues that should be considered when actually building a ‘personalising’ research information system. Two examples are: where to find the information, and how to design the user interface. But that has nothing to do with the actual personalisation of information. Therefore we only consider issues on what should be done, not how. Consequently, the two previous mentioned examples are regarded as follows: we assume the location of information is already known, and the user interface should contain “these” options. However, building such a personalising system might be the next logical step to take.
1.3
Problem statement and research questions
When analysed more deeply, we may state that the problem referred to in the previous sections is information overload ; obviously, if there was none, researchers would have no troubles finding the right information. This thesis investigates how personalisation can deal with this issue for researchers. Accordingly, we formulate the following problem statement. How can personalisation support the quest for information, as formulated by researchers? This statement can be broken down into three fractions: (1) personalisation, (2) the quest for information, and (3) how can ‘(1)’ support ‘(2)’ ? These fractions play key roles in answering the problem statement. Therefore, we formulate them into three corresponding research questions, which will guide our research. Research question 1: What is involved in the personalisation of information? Research question 2: Where do researchers desire personalised support in their search for information? Research question 3: How can a personalised quest for information be achieved? After those questions have been answered, we can formulate an answer to the problem statement.
1.4
Thesis overview
Below, an overview is provided of the contents of our thesis. In the three following chapters, we are dealing with the three research questions. Thereupon, our model is formulated, and evaluated by means of a prototype. Subsequently, the problem statement is answered in the conclusions.
4
CHAPTER 1. INTRODUCTION
In chapter 2, personalisation is scrutinised, metaphorically speaking. It deals with issues about what is denoted by personalisation, which factors can have an influence, and how it can be performed. The chapter will give an answer to the first research question “What is involved in the personalisation of information?” Chapter 3 investigates how researchers search for information, by using the results of a series of interviews with six researchers. Moreover, the chapter deals with the encountered problems and the possible points of improvement. A brief analysis of this investigation will answer the second research question “Where do researchers desire personalised support in their search for information?” In chapter 4, the acquired knowledge of the previous two chapters is combined. We then explore the third research question “How can a personalised quest for information be achieved?” An answer to this question is supported by our model. We test a proof of concept for our model. A prototype is implemented and evaluated by two junior researchers. The results are given in chapter 5. Chapter 6 contains the conclusions on “personalisation for researchers”. They are accompanied by recommendations for further research.
Chapter 2
Factors on personalisation
The search for information — not any information, but the right information — is like a quest for a treasure. Here, one may go numerous roads filled with obstacles. Some of them may lead to a treasure, but many may lead to a dead end. Besides that it is not necessarily obvious in advance what kind of treasure one is looking for, what it looks like, and what it should contain. An information system that applies personalisation may give a you hand: it may guide its users to select valuable roads and treasures. This chapter provides an answer to the first research question: “What is involved in the personalisation of information?”. Section 2.1 investigates what personalisation is. Next, we consider factors on personalisation that are related to two main components of an information system: the users and the items. User-related factors contain what a user may want. These are explored in section 2.2. Then, section 2.3 investigates treasure-related factors, which express why an item may be relevant for a user. Next, in section 2.4, we explore four approaches on how to personalise. In section 2.5, two supportive (research) domains are described. Section 2.6 concludes this chapter.
2.1
What is personalisation?
In this section, we first find out what personalisation can do, i.e., why it may be a remedy to information overload (section 2.1.1). Next, personalisation is defined in our context (subsection 2.1.2). Finally, in subsection 2.1.3, two related areas are briefly discussed.
2.1.1
A remedy to information overload
There is a tremendous amount of information available. A study by Lyman and Varian (2003) estimates the amount of newly created information 2002 between one and two exabytes1 . In 2003, it increased to approximately five exabytes, of which 92% is stored on magnetic media, mostly hard-disks. Film represents 7%, and paper 0.01%. The size of the Internet was estimated on 532, 897 terabytes. In 1998, it was already obvious that there are too many 1
One exabyte is 260 bytes is 1 trillion bytes, or to be precise 1, 152, 921, 504, 606, 846, 976 bytes.
5
6
CHAPTER 2. FACTORS ON PERSONALISATION
resources available for one person to consult exhaustively (Soltysiak and Crabtree, 1998). Even if a researcher would be fortunate enough to be able to browse through all pages, he would only find a small percentage of any interest or relevance. It is clear that the overload makes finding the right information a difficult task. Decreasing the bulk of information makes that task more easy. This can be done by personalising the bulk: it means providing relevant items from the bulk for the current user. Hence, personalisation may be a remedy for information overload. Consequently, it may improve the efficiency of the quest for information.
2.1.2
Definitions
Many definitions of personalisation exist. The term is used loosely and wildly in accordance with the context in which personalisation is employed. Below, we explore two of them, followed by our own definition. For web mining, Eirinaki and Vazirgiannis (2003) define it as follows. “Personalisation is any action that adapts the information or services provided by a Web site to the needs of a particular user or a set of users, taking advantage of the knowledge gained from the users, their navigational behaviour, and individual interests, in combination with the content and the structure of the Web site.” It is negotiable whether the adaptation caused by any action leads to personalisation. We prefer to evaluate the result of the set of actions. Furthermore, our formulation will be about information systems in general. Let us first consider a second definition. For one-to-one marketing, Kramer et al. (2000) define personalisation as follows. “Personalisation is a toolbox of technologies and application features used in the design of an end-user experience. Features classified as personalisation are wideranging, from simple display of the end-user’s name on a web page, to complex catalogue navigation and product customisation based on deep models of users’ needs and behaviours.” The intended end-user experience in this thesis is that researchers find their treasures. Obviously, the treasures are information items, instead ofproducts and services. In a personalising information system, the interactions and information will suit the internal knowledge about the user and his needs. Metaphorically speaking, personalisation acts as a guide in the quest for information. So, our definition used in this thesis is formulated as follows. “Personalisation is the result of an approach towards information systems that improves the interactions with an individual user, given a context and situation, and tailors information to the needs of the individual user.” The given context in this research is “researchers who are searching for information during a project”. The given situation is dependent on the activities of the user. Two examples are: a researcher who is looking for information on related areas, and a Ph.D. student who is looking for information about an author. The ‘needs’ may be explicit as well as implicit. So, a personalising system will anticipate to what the user might need (Mulvenna et al., 2000).
2.2. USER-RELATED FACTORS
2.1.3
7
Related areas
Two areas that are closely related to personalisation are (1) customisation, and (2) information retrieval. They are briefly described below. Customisation and personalisation are often used indistinctively, but there is a difference. Customisation refers to the adaptation of (1) the content, (2) the structure, and (3) the layout of a system to the preferences of the user (Pierrakos et al., 2003). Those preferences are provided explicitly by the user, and the system delivers exactly what the user requests, whereas personalisation anticipates to implicit needs as well. An example of customisation is My MSN portal2 , a portal that offers the possibility to select (1) content topics, (2) the position of the contents, and (3) the colour scheme. According to Krottmaier (2004), customisation can be interpreted as a simple kind of personalisation, perhaps a useful tool to enable it. The area of information retrieval deals with representation, storage, organisation and access of information items (Baeza-Yates and Ribeiro-Neto, 1999). It rather informs us on the existence of information, whereas personalisation tailors it towards a certain user (Van Rijsbergen, 1979; Belkin and Croft, 1992). Numerous techniques of information retrieval are used, but they are less valuable in the actual process of selecting personal information (Good et al., 1999).
2.2
User-related factors
In this section, four topics related to the user and personalisation are described. The first subsection (2.2.1) explores the user himself. In subsection 2.2.2 the focus is on the system. Then, in 2.2.3 we investigate how to acquire information about a user. Next, subsection 2.2.4 studies the internal representation, i.e., a user model. In the last subsection (2.2.5), three related issues are described.
2.2.1
The user
A personalising system aims at the individual user. Below, we explore four factors; they are concerned with how information about a user may influence personalisation: (1) personal data, (2) preferences, (3) interests, and (4) temporal requirements (adapted from Krottmaier (2004)). Personal data Personal data refers to external knowledge about a user. A few examples are name, education, and information, such as experience with the system and familiarity with a domain (Foltz and Dumais, 1992). A main purpose of personal data is to identify the user. Since this data is rather static, it enables two more options for personalisation. First, a set of statical personalising assumptions may be applied (Paliouras et al., 2002). For instance, users of a certain age may be provided with a specific news category. A second option is to adapt the functionality of a system for a certain group of users, such as providing more guidance for ‘novices’. 2
http://my.msn.com
8
CHAPTER 2. FACTORS ON PERSONALISATION
Preferences Preferences of a user are customisation settings to which the system acts exactly as configured. Some of these settings may influence personalisation. For instance, preferences that are concerned with sharing personal information, which is a delicate subject due to privacy issues (see related issues 2.2.5). They define how much personal information a user is willing to share with whom. An example is to offer privacy sharing to the level of (1) the user only, (2) an anonymous usage, or (3) group sharing. When the second level is selected, statistical personalisation assumptions may be derived. Interests The interests of a user mainly determine whether a certain item is relevant; they tell a system what information to deliver. We consider two types. First, information that a user is interested in, such as his likes and dislikes. Second, information concerning what is needed for a certain user. For example, ‘user modelling’ because he is interested in ‘personalisation’. Two characteristics of interests are the following. First, it is trivial that a user’s interests are not confined to one domain. The second characteristic is known by the name ‘interest drift’, i.e., user’s interests may change over time (Koychev and Schwab, 2000). Temporal requirements Temporal requirements are the current user needs that appear when a user is in a rather exceptional circumstance, i.e., the situation is not explicitly taken into account in the context of the user. This may cause a temporary change to one of the previous mentioned factors. For instance, when a user is working against a deadline, an interesting, though unavailable, article is currently not relevant for him. These requirements may be met by customisation settings, e.g., select only articles that are available.
2.2.2
The system
Van Setten (2001) studied existing personalising information systems. From that study, we adopted and slightly adapted five factors. They are concerned with the user and the system: (1) the task, (2) the importance, (3) the usage process, (4) the frequency of use, and (5) the time to spend. They are described below. Task When a user is achieving a goal, he may use a system to perform a certain task. This task may have an effect on the type of personalisation. For example, a learning task may prescribe content-oriented personalisation. Importance The importance of the task may provide requirements for the result set (Foltz and Dumais, 1992). For example, a system for cancer patients, which provides information about their current health status, requires accurate results. Wrong information may cause unnecessary
2.2. USER-RELATED FACTORS
9
stress (Cawsey et al., 2000). While for entertainment, serendipitous results between inaccurate results may be preferred. Usage process The usage process is here defined as how the user approaches the system. This may influence where and which form of personalisation can be applied. Below, two methods of initiating personalisation are explored. • The user takes the initiative to receive information. • The system takes the initiative to push information towards the user. In the first method, the system searches for relevant information when the user asks for it. For example, the user initiates the quest for information with a query. Here, personalisation may be applied in two ways. • The query may be personalised. For example, improving the query by adding keywords that are relevant for the requesting user. • The retrieved results may be personalised. For example, the system ranks the results according to the predicted relevance for the user. In the second method, the system takes the initiative. For instance, when it finds a user that might appreciate a piece of information, it pushes the information towards that user. This approach is characteristic for the area of recommendation. Frequency of use The factor frequency of use may have an impact on the detection of a change in the user his interests. When the system is used once or at irregular times, a user model is not required. It is not used again, or a change of interests will go unnoticed. However, when the frequency of use is on a regular basis or continuously, user models are preferred and can be maintained. Time to spend When a user has only limited time to spend to separate the good from the bad results, a high accuracy of relevance is preferred. But, when the user is searching for one certain item, the false negative ratio of the results can be minimised at the expense of search time.
2.2.3
The acquisition
Sufficient user information is vital for personalisation to perform optimally. In this subsection, we explore five methods to gather information about a user. Ask the user A first method is to ask the user explicitly for information. Two frequently used technologies are questionnaires and explicit relevance feedback. Questionnaires, such as fill-in forms, are used to receive personal data, and enable the selection of personal rules, such as ‘interested in news category X’. The second technology, explicit relevance feedback, is well-known for asking
10
CHAPTER 2. FACTORS ON PERSONALISATION
a user’s opinion on an item. The user evaluates, by means of a rating, the relevance of an item that is returned to him by a personalising system (Middleton et al., 2004). Unfortunately, both technologies require an effort from the user, which he may not be willing to expend when (1) he has to go into another direction, or (2) it does not provide him with an immediate benefit (Webb et al., 2001). Observe the user Relevance feedback can be detected automatically as well, i.e., implicit relevance feedback. For instance, when the user reads a document, he is probably interested in its content. This brings up a second method: observing the user. The actions and behaviour of a user are analysed, and personalising assumptions are derived. That may range from global actions, such as an entered query, to very detailed actions, like a mouse click. Another clever method for retrieving user information is used by Renda and Straccia (2005). Here, assumptions about a user’s interests and beliefs are based on his private folders. In general, observing the user has the advantage of collecting unobtrusive measurements (Webb et al., 2001). But, the assumptions may not reflect to reality. For instance, not reading a document does not have to imply a lack of interest. A study by Boger et al. (2001) showed that users tend to underestimate certain terms which describe their interests. Consequently, they do not include them in their personal rules either. Therefore, automatic analysis may identify user interests more accurately. Classification A third method considers gathering user information as a classification problem (Montaner et al., 2003). We explore two such methods: (1) stereotype modelling, and (2) learning by means of a training set. A stereotype model stores predefined characteristics with typical assumptions. An example of such an assumption is “an Antarctican woman is supposedly not to interested in lawn mowers.” A user is assigned manually, or matched automatically to one or more stereotypes, from which he inherits all properties. A comparative study between a stereotype model and a model that is filled with personal rules showed that first offers more effective results (Kuflik et al., 2003). Furthermore, the assumptions about a user can be supplemented, or overridden, when additional user information is available (Kobsa, 1995, 2001b). For instance, as a rule, a user from the Netherlands receives his information in Dutch; when English is preferred, the rule is overridden. In the second model, a classifier learns the interests of a user through explicit relevance feedback on a set of predefined, categorised examples, i.e., a training set. It then may classify further items likewise. This approach has the advantage of simplified handling. But, there are two disadvantages. First, when the examples are not representative for the user, the results are less accurate (Montaner et al., 2003). Second, the model built is accurate for the current interests. When no regular updates take place, the model may become outdated (Webb et al., 2001).
2.2. USER-RELATED FACTORS
11
Groups of users The implicit version of stereotype models are user group models. They are based on comparable actions between users and assumptions about those actions (Kobsa, 2001b). There is a continuous interaction going on between the group of users and the model. An assigned user inherits the knowledge contained in the model, and his actions have an impact on that knowledge as well. So, a user-group model changes dynamically in composition and knowledge. Hence, it is constructed by preferences and maintained automatically (Paliouras et al., 2002). Colleagues A last method, exploits explicit relationships between users, whom are identified as colleagues. When a team is present, i.e., a group of people who work together, a single member may benefit from the “knowledge space” of other members (Krottmaier, 2004). For example, course information may include materials provided by a teacher as well as by students (Wang and Shao, 2004).
2.2.4
The user model
The user model is an internal representation of information and assumptions about a user. It is used to improve interaction between the system and that user (Kobsa, 1995; Gonzlez, 2003). It may store as little as a single action, such as a currently selected document, or may be be as broad as a learning model which reasons about the expected interests and needs (Kobsa, 2001b). A user model is often dynamic because user information is subject to changes, such as interest drift (see 2.2.1). When merely static information is captured, such as personal data, we prefer to call it a user profile. The representation of user information is important for personalisation, since it is a source for the accuracy of the results. There have been used numerous structures to represent the information in a user model. Below, we identify four types. • A first type provides a direct link between a user and his items. For instance, a user item matrix is a structure that stores the ratings for evaluated items. • A second type comes from expert systems. User information is expressed by means of rules. An example of such a rule is, IF
CONTAINS "personalisation" THEN <relevance> = 0.9. • In the third type, the structure is based on the representation of the items (see 2.3). For example, when a set of documents is categorised with accompanying keywords and weights, then personal weights may represent the user’s interests. • The last type of structure is (a model of) an algorithm. For instance, a Bayesian classifier may predict a probability of likeliness for unseen items as information for the user. Various machine-learning techniques have been applied for user modelling (Paliouras et al., 2002; Montaner et al., 2003; Fan et al., 2005). They have the ability to address the complex and dynamic attributes of a user in an adequate way (Webb et al., 2001).<br />
<br />
12<br />
<br />
CHAPTER 2. FACTORS ON PERSONALISATION<br />
<br />
Furthermore, a subject which is given a great deal of attention in user modelling is interest drift. Numerous systems include some sort of forgetting, i.e., creating a cause that assigns less importance to older interests. Unfortunately, sporadically encountered interests, that also have less impact, will be forgotten too (Burke, 2002). Therefore, Billsus and Pazzani (1999) have modelled two types of interests: (1) long-term interests, such as the interest in a certain type of music, and (2) short-term interests, such as news about earthquakes. When information from short-term interests is insufficient, there is a shift to the long-term interests model. Besides the representation, the life cycle of a user model may have an influence on personalisation. Below we explore three stages: (1) initialisation, (2) maintenance, and (3) destruction. Initialisation of the user model Each time a personalising system is used, the first step is to identify the current user. There are several methods for this matter, such as asking for a user name and password, and storing a cookie – a short text file that contains, for example, an encrypted name – on the user’s computer. However, the first time a personalising system is used, the user’s model is like a blank sheet. There is insufficient knowledge obtained to personalise decently; results may be inaccurate, or even unavailable. This inconvenience is called the cold-start problem. It may appear during the period when a system is learning to know a user. From the aforementioned acquisition methods, those that provide more user information from the start, may bring alleviation, such as a stereotype model. Maintenance of the user model The maintenance of the user model is the most important stage in its life cycle. This stage is responsible for keeping the model up to date, which prevents performance degradation of the personalisation process (Lam et al., 1996). Below, we first describe three strategies to maintain a user model, and next, three modalities in which it can be updated. Based on the acquisition of user information (see 2.2.1), we identify three strategies of maintaining a user model. In manual maintenance, a user is responsible for keeping his model up to date. For instance, when he is in charge of his personal rules, he should adapt them to his convenience. Opposite to this is automatic maintenance, the system may keep track of changes for the user. For instance, information from observing a user may add new assumptions, or adapt older ones. The third strategy is a combination of the two previous strategies: cooperative maintenance, user and system cooperate to keep the model up to date. Explicit relevance feedback is an example, the user gives feedback which the system uses to adapt the user model to represent current user information. Modalities are plans that decide when to update a user model. In the following, we explore three of them (van Setten, 2001). • Real-time update: a user model processes each action of a user immediately. • Off-line update: information is gathered while the user is using the system. Subsequently, the model is updated. • Batch updates: information is continuously gathered and periodically updated. For example, twenty actions are recorded and then used to up-date the user model.<br />
<br />
2.3. TREASURE-RELATED FACTORS<br />
<br />
13<br />
<br />
A real-time update may cause a change to the presented results. This may be required (1) when the validity of an item is short (see 2.3.2), and (2) when a need is satisfied after a certain action. We remark that there is no need to evaluate the same item twice (Herlocker et al., 2004). The offline update may provide more time to perform computational complex updates. Batch updates are encountered when a set of actions form a unit. For instance, an update occurs when a user has adjusted his personal rules. Destruction of the user model A user model may be destructed either through a physical delete, or by removing the identification towards the user, leaving an anonymous user model. A physical delete may have an impact on personalisation. Then, the benefit for other users, gaining knowledge from that resource, has ended (see subsection 2.2.1).<br />
<br />
2.2.5<br />
<br />
Related issues<br />
<br />
Three related issues of user-related factors in personalisation are ownership, privacy, and trust. Users are concerned with their privacy, they are not willing to share that information for any reason (Cranor et al., 1999). Obviously, there is a trade-off between personalisation and privacy: a user needs to give away some of his personal information, in order to let personalisation function properly (Twidale and Nichols, 1998). The willingness increases when (1) a proper reason is given, and (2) it is explained how the information will be used (Rosenbaum, 2001). So, when a user trusts a system, he may be willing to share private information. The issue of ownership is closely related to how the information will be used. It is concerned with the question “who owns the information about a user?” (van Setten, 2001). That includes, among others, who may change it, and may it be used for other purposes. A single solution for the issues privacy and ownership does not exists, since, apart from laws, they differ from user to user (Kobsa, 2001a). But, it is suggested that the user should be in charge of the management and ownership of his personal data and preferences (Shearin and Maes, 2000).<br />
<br />
2.3<br />
<br />
Treasure-related factors<br />
<br />
Certain information items are to be considered as the treasures a user is looking for. This section investigates treasure-related factors for personalisation. The first factor, representation, describes global modelling of items in a personalising system (subsection 2.3.1). Then, we continue with individual items. Such an item consists of two parts: (1) content, and (2) metadata. These are explored in subsections 2.3.2 and 2.3.3, respectively.<br />
<br />
2.3.1<br />
<br />
Representation<br />
<br />
A treasure may be presented as-is in a system, for instance, <name, picture, description>. Here, personalisation may benefit from modelling the items and representing them appropriately. We describe two strategies below. • Top down: this strategy groups the items into either or not predefined topics, such as classes or clusters.<br />
<br />
14<br />
<br />
CHAPTER 2. FACTORS ON PERSONALISATION<br />
<br />
• Bottom up: this strategy makes relationships between (parts of) items available, such as a graph. The ‘additional’ information provided through such modelling may become part in the personalisation process. For example, a rough separation between documents may occur based on topics. When a user is not interested in topic X, documents in that class are likely to be not relevant either.<br />
<br />
2.3.2<br />
<br />
Content<br />
<br />
Content factors are properties of the actual content of an information item. Below, we explore three of them: (1) type, (2) structure, and (3) validity. Type The type of the content is the format of an information item. A few examples are text, video, an idea, and a physical person. The type appoints applicable methods for analysis of the content. For example, text may be pre-processed, i.e., removing the stop words and stemming the words, before analysing the content; video may be evaluated by humans. It has an impact, since item evaluation is required for personalisation. Structure The second factor is the structure of the information. Below, we distinguish between three types of structure (van Setten, 2001). • Structured information is inserted into a predefined structure. For instance, information that is stored in a database. • Unstructured information has no predefined structure. There is no information available about where which content may be. Nonetheless, there exists methods which may provide more information about an item. For instance, the TF-IDF algorithm may point to frequently used words in a set of documents, which are considered as more important (Montaner et al., 2003). • In semi-structured information a basic structure is present that may store unstructured information. Structure may make it easier to analyse an item. It provides a certain predefined reliability about the content. Thereby, it may improve the results of personalisation. Validity The validity factor concerns the life-span of an item. For instance, the value of a stock exchange quotation has a very short life-span. When information becomes outdated, the relevance of that item decreases. Notice that the life-span may differ depending on the user. For example, the weather of last week is not important for most users; but for a meteorologist, who analyses the weather over one year, it is valuable.<br />
<br />
2.4. PERSONALISATION APPROACHES<br />
<br />
2.3.3<br />
<br />
15<br />
<br />
Metadata<br />
<br />
Metadata is often referred to as data about data, or information about information. According to Hodge (2001), metadata is structured information that describes, explains, locates, or otherwise makes it easier to retrieve, use or manage an information resource. Because metadata is information too, the factors structure and validity also apply here. The factor type is divided into three categories (Hurley et al., 1999). • Descriptive metadata describes a resource, mainly for discovery and identification. For instance, a unique identifier for an item. • Administrative metadata provides information to help manage a resource. For instance, date of creation, and who may access it. • Structural metadata indicates how compound objects are put together. For instance, how pages are ordered to form chapters. Metadata provides targeted information about items, which may not be extractable from the content. So, it can be used to enhance personalisation. It may occur that metadata as the only resource for knowledge about the items, is sufficient for personalisation.<br />
<br />
2.4<br />
<br />
Personalisation approaches<br />
<br />
Up to now, we have explored how user-related factors and information-related factors (we mentioned them as treasure-related factors) may have an impact on personalisation. This section will investigate four approaches on how they achieve the effect aimed at. We assume that there are two major components in a personalising system: (1) the users, and (2) the information items. Each of the four approaches makes an own effort to connect a user, referred to as the active user, to the appropriate information items. The first approach is the content-based approach (subsection 2.4.1). It takes the component of information items as its starting point for personalisation. Informally, the approach would reason as follows “I have some more items that are related to what this active user requested, they may be relevant for him too”. The second approach is the collaborative approach (subsection 2.4.2). It starts from the component of users and will select the personal information items for an active user. Informally, the approach would act as follows “I have noticed that other users who liked this item A also liked that item B, maybe the active user likes that item B too”. The third approach is the extended approach (subsection 2.4.3). Here, the knowledge of how to connect an active user with his items is extended to outside the components. The reasoning of the approach would be as follows “I have come to know that when you are looking for something like item A, you most probably want to find item B”. At last, as the well-known saying goes “two heads are better than one”, the fourth approach considers hybrid approaches (subsection 2.4.4). It investigates how the previously mentioned approaches may be combined.<br />
<br />
16<br />
<br />
CHAPTER 2. FACTORS ON PERSONALISATION<br />
<br />
2.4.1<br />
<br />
Content-based approach<br />
<br />
The main characteristic of the content-based approach is that the information to personalise is mainly retrieved from the contents. It assumes that related items may be relevant for the active user too. The strategy may be formulated as in the following four steps. 1. There is information about what the active user is interested in. 2. Items that are related to that information are selected. 3. Items that are related to the items of step two are selected. 4. The selections may be a personalised result fit for the active user. The first step of the strategy is often user modelling (see section 2.2.4). Information about relevant content for an individual user is modelled, such as from his past behaviour. Consequently, each suitable user interest can be included during the selections. That is a first characteristic of the content-based approach. In the second step, user model – item relations are identified. An example is Syskill & Webert, a web page-recommender-agent (Pazzani et al., 1996). Here, the user model is a Bayesian classifier, trained with personal ratings (hot or cold) for previously seen web pages. A novel web page may then be classified likewise. According to Burke (2002), a classifier is commonly used for these two steps. In the third step, item – item relations are identified. When two items share certain similar features, a relation is established. An example is Citeseer, an online computerscience research paper repository (Lee Giles et al., 1998; Lawrence et al., 1999). It uses citation information to identify related documents. To enable steps two and three, the content-based approach is provided with: (1) definitions of personalising features, and (2) methods to identify relations. Hereafter a second characteristic is established, the content is analysed automatically. That makes the system familiar with the items, so it is not required that the user has domain knowledge. But, the items have to be in machine parse-able format (Shardanand and Maes, 1995). Furthermore, content-based analysis has no method to extract subjective features from items (Montaner et al., 2003). For instance, it cannot distinguish between a well-written and a badly-written article. In the fourth step of the strategy, relevant results may be separated from related results. Since the item – item relation may not be transitive, items that are related to related items may not be relevant for the active user. This brings us a last, rather unexpected disadvantage of the content-based approach: over-specialisation. There is no inherent method to select precisely the relevant items that are needed and not directly related on content (Lee et al., 2001). So, serendipitous results do not occur. In the following and in subsection 2.4.4, we will see how these characteristics and disadvantages may be used within personalising approaches.<br />
<br />
2.4. PERSONALISATION APPROACHES<br />
<br />
2.4.2<br />
<br />
17<br />
<br />
Collaborative approach<br />
<br />
The main characteristic of the collaborative approach is that the information to personalise is retrieved from a different component, viz. from the users instead from the items. It is assumed that human behaviour is correlated (Pennock et al., 2000). An example is the implementation of the “worth to mouth” principle (Shardanand and Maes, 1995). If a set of interests of user x is similar to the set of user y, then other interests of y may be of interest too for x. A strategy for the collaborative approach may be formulated as follows. 1. There are user models of the active user and other users. 2. A group of users with similar user models is found. 3. Novel items that were found interesting by this group are selected. 4. The selections may provide a personalised result fit for the active user. The user model for this approach may contain information slightly different from the information for the content-based approach. For the content-based approach we may think of <user x is interested in Asian travelling>. In the collaborative approach, it contains merely evaluations on items. An example is <user x bought ‘Asian Lonely Planet’>. Such evaluations may capture how the user likes an item. So, it is a first characteristic that the collaborative approach can personalise on features that are not content based. In the second step, this approach searches for related users, rather than for related items, as within the content-based approach. For instance, it searches for users with a similar behaviour. The use of human evaluation functions for the items excludes the requirement of machine parseability. However, “grey sheeps”, i.e., users with a unique taste, may fall outside any group, which may provide them with inaccurate results (Claypool et al., 1999). Then, in the third step, the evaluations provided by the group of similar users may identify relevant items for the active user. This method of selection has the characteristic that relevant items do not have to be content-based related. Via the human evaluation functions, serendipitous results may occur. But, it is a challenge to provide items that just entered the system, with sufficient evaluations. Otherwise, they are not likely to occur in any result sets and cannot receive further evaluation. A purely collaborative approach has no knowledge about the items themselves, but that can be adequately assessed by human evaluation functions. However, when the coverage of evaluations is too sparse, the results may be inaccurate. This is referred to as the sparsity problem. It is required to have sufficient users and evaluations (Montaner et al., 2003). Besides selecting items, there is the option of making predictions. The more similar users that indicate they like item x, the more item x may be relevant for the active user (Sarwar et al., 2001). So, there are several possible utilisations (Herlocker et al., 2004). We describe four of them. • Making annotations in an existing context. On top of links between items, annotations predict a certain likelihood. For instance, a web link recommender, such as<br />
<br />
18<br />
<br />
CHAPTER 2. FACTORS ON PERSONALISATION<br />
<br />
Google’s3 page rank (Page and Brin, 1998), overlay predictions of relevance to the search item on top of existing links, this may help the users in their search. • Find good items. Here, a list of “best-bet” suggestions is composed. For instance, Ringo, a music recommender system, can show its requesting user a ranked list of albums or artists he will like and dislike (Shardanand and Maes, 1995). • Find all good items. In contrast to the previous utilisation, where some good items may be overlooked in order to eliminate many bad ones, some domains require an assurance that the false negative rate can be made sufficiently low and the positive rate approaches hundred percent. For example, lawyers are willing to invest a large amount of time searching for a specific case. • Suggest a sequence. Instead of suggesting one item at a time, this utilisation suggests a sequence of items that are relevant as a whole. For instance, the personalised radio website Launch4 recommends a sequence of songs that is pleasant to listen to jointly. According to Herlocker et al. (2000), the collaborative approach is, mainly due to the sparsity problem, not yet trusted in high-risk information domains. But, it has been proven to be sufficiently accurate for entertainment.<br />
<br />
2.4.3<br />
<br />
Extension-based approach<br />
<br />
Instead of relying on information that is available within the components of the system, i.e., items and users, this approach is provided with an extended information source. A strategy may be defined as follows. 1. There is an extended source of information available. 2. Reason about the active user needs. 3. Select an appropriate result set. For the first step, we distinguish three types, (1) user-based extension, (2) item-based extension, and (3) commonsense knowledge. These are described below. A user-based extension identifies what the personalised items for a particular user are. For instance, the reader of an online newspaper has identified national news, and science as his interests. Methods from acquisition (see 2.2.3) that may apply here provide information in advance, such as the questionnaires can do. A second type of extended source, item-based extension, describes how an item may meet the user needs, without regarding any user in particular (Burke, 2002). A design process for the items may be as follows: (1) identify properties for items that may describe certain user needs, (2) identify what values these properties may contain, and (3) fill each item with the appropriate values. An example is: a book may be extended with <genre, style, price category>. A paperback pocket version of Stephen King’s “The dark tower” may then be filled as follows, <fantasy, plot-based, economic>. 3 4<br />
<br />
http://www.google.com/technology/ http://music.yahoo.com/launchcast/default.asp<br />
<br />
2.4. PERSONALISATION APPROACHES<br />
<br />
19<br />
<br />
A last type of extended source is commonsense knowledge (Pazzani, 2005). It does not provide information on users or items in particular, but it provides trivial information, that a machine is not aware off. For instance, from two almost identical documents, the newest is likely an update. It is a disadvantage that the additional information has to be provided. For instance, it may come from another part of the system where that an item is available. An other option is that the user provides it (see above). Or, a last option, someone may be imposed with this task to provide it. In the worst case, that is a knowledge acquisition task (Burke, 2002). The second step of the strategy, reason about the active user needs, identifies why a certain item may be appropriate for the current user. Hereafter, three such methods are described (Burke, 2002). A first method, rule-based filtering, is often encountered as a technique to achieve personalisation. It reasons with a set of rules, predefined or discovered, about which items may be suitable for a user. A second method, used in “Find-me” systems, is gradual refinement. Here, features of items are weighted up against each other. A user starts with a general query, such as rent a car. Then, the needs are gradually refined. For instance, it weights up gas oil against diesel, coupe against four doors, with or without a roof, . . . . The third method is selection. When a user has provided personal values for the properties of an item, matching items are selected. For instance, when booking a flight, a utility function is constructed, providing for each feature a value, such as leaving on <date, hour, from, to>. Then, items that match the function are selected. The third step of the strategy is selecting an appropriate result set. It is unlikely that a stand alone commonsense extension provides sufficient information to personalise. But, it can improve the general achievement of another approach, for instance, by removing its nonsense results (Lieberman et al., 2005). In this approach, it is not required to build long-time relationships with a user5 . The necessary information is already available, or it is valid for one session only. Hence, typical user modelling problems, such as a cold start are not encountered. This is in contrast with the other two approaches, which have to handle them.<br />
<br />
2.4.4<br />
<br />
Hybrid approaches<br />
<br />
In an hybrid approach, two or more personalising approaches are combined, in order to overcome an inconvenience. That makes sense because the approaches have counterbalancing characteristics. This is shown in table 2.1 for the content-based and collaborative approach. The extension-based approach suggests two general solutions: (1) provide the necessary information, and (2) eliminate the disadvantage. For instance, (ad 1) stereotype user modelling is a user-based extension which alleviates the cold-start problem; and, (ad 2) extending the items with possible user needs, eliminates it. Furthermore, the knowledge acquisition task remains, when automatic analysis or human evaluation functions are not applicable. The specific purposes of an hybrid approach can be divided into three categories. They are briefly described below. 5<br />
<br />
Except when rules are discovered automatically, then it may be user modelling after all.<br />
<br />
20<br />
<br />
CHAPTER 2. FACTORS ON PERSONALISATION<br />
<br />
Content-based approach Disadvantages, i.e., (possible) characteristics Machine parse-ability Lack of subjectivity Over-specialisation Collaborative approach Disadvantages, i.e., (possible) characteristics Grey sheep First-rater Sparsity<br />
<br />
Counterbalancing activities Human evaluation Subjective attributes can be taken into account Serendipitous results can occur Counterbalancing activities Each interest is captured Automatic analysis System is familiar with items<br />
<br />
Table 2.1: Disadvantages, i.e., (possible) characteristics, and counterbalancing activities of content-based and collaborative approach.<br />
<br />
• Improve information from a user model: instead of using standard user models, the models are adapted and use another approach. An example is content-based collaboration: in a collaborative approach, content-based user models are exploited to discover similar users (Pazzani, 1999). • Modify items in the start set: all items are processed before personalisation takes place. An example is Libra, a content-based book recommender (Mooney and Roy, 2000). It includes for each book two additional features, “related authors” and “related titles”, which are generated by a collaborative approach. • Change the volume of the result set: the results of one approach are added to the results of a second approach. An example is a TV program recommender, which uses a content-based approach based on program descriptions, and a collaborative approach based on user preferences (Smyth and Cotter, 2000). The final result is a mix of both result sets. Several implementations have attested an improvement due to hybridisation (Pazzani, 1999; Good et al., 1999). Nevertheless, more research may provide a better insight into the trades-off between different hybridisations (Burke, 2002).<br />
<br />
2.5<br />
<br />
Supportive domains<br />
<br />
Several existing domains in computer science may contribute to the personalisation of information. We already mentioned information retrieval (section 2.1.3) to identify the information and its location, and user modelling (section 2.2.4) to know the user. Two other supportive domains are data mining and software agents. These are briefly explored below.<br />
<br />
2.5.1<br />
<br />
Data mining<br />
<br />
Data mining is the domain which turns data into knowledge. It analysis large data sets for exploration, to find relationships and patterns, to make predicts, to generalise, and to summarise<br />
<br />
2.6. AN ANSWER TO THE FIRST RESEARCH QUESTION<br />
<br />
21<br />
<br />
(Hand et al., 2001). We have already seen a number of its applications in personalisation: clustering groups of similar users, discovering association rules between items, predicting a personal ranking score in a collaborative utilisation, analysing user behaviour, and using a classifier as a user model (Schafer, 2005). The common ground between data mining and personalisation is handling large sets of data, and preferably extract new information from them. So, techniques from data mining can be adapted to support personalisation.<br />
<br />
2.5.2<br />
<br />
Intelligent agents<br />
<br />
An intelligent agent is a computer component in some environment, that can act autonomously in order to meet its design objectives (Weiss, 1999). A personalising system may delegate a certain task to an agent. A few examples are searching for information, providing ranks as a pseudo-user, and communicating with the user. Besides these tasks, a personal agent is designed to play a part in personalisation. It is capable of acting without intervention of users or other agents, it has the ability to learn, and can act on behalf of its user (Weiss, 1999). So, it may reason and anticipate to what a user wants. For instance, a personal agent may learn a user’s interest, and actively search for the requested type of information source. The user should trust the agent and must be willing to delegate some of his tasks. It is required that user and agent co-operate in order to achieve the goal of the user (van Setten, 2001).<br />
<br />
2.6<br />
<br />
An answer to the first research question<br />
<br />
The first research question as formed in 1.3 reads as follows. What is involved in the personalisation of information? In this chapter, we have outlined and classified the most important factors of personalisation, their relations and several methods and strategies. In summary, the most important factors are (1) the user, and (2) the information items (we called the latter treasure-related factors). The relations are described in connection with the factors. We distinguish four personalisation approaches, viz. (1) the content-based approach, (2) the collaborative approach, (3) the extension-based approach, and (4) the hybrid approaches. For each of these approaches, we described a strategy (and some alternatives) to follow for an accurate application of the approach.<br />
<br />
Chapter 3<br />
<br />
Fieldwork in academic research<br />
<br />
In this chapter we focus on the second research question. Where do researchers desire personalised support in their search for information? To answer this question, we went to the field of academic research and investigated two topics: (1) how do researchers currently search for information?, and (2) how would they like to have personalised support during that quest? The researchers to be investigated (called the actors) are scientific researchers who quest for information during academic research. We found six actors who agreed to be interviewed on these topics. Their background and the organisation of the interviews is covered in section 3.1. The investigation is reported in section 3.2 (topic 1) and section 3.3 (topic 2). Finally, section 3.4 gives an answer to the second research question.<br />
<br />
3.1<br />
<br />
Conducting the interviews<br />
<br />
This section discusses the background of the six researchers, the actors, who agreed to be interviewed (3.1.1, and see Appendix A.1). More information about the interviews can be found in 3.1.2 and Appendix A.<br />
<br />
3.1.1<br />
<br />
The actors<br />
<br />
We invited six members of Ikat to collaborate in the research over a part of their research, viz. the search for relevant literature. Ikat is the Institute for Knowledge and Agent Technology at the Universiteit Maastricht. Within Ikat, research and eduction activities of the computer science are accumulated1 . The group of six researchers consists of three Ph.D. students and three senior researchers. The three Ph.D. students were in different stages of their project. The first one was focussing on the composition of her project proposal. The second one was in the middle of her project. The literature study was finished, but a quest for specific information still existed. For the third one, the interview was mainly a retrospective action on his quest for information. The 1<br />
<br />
http://www.cs.unimaas.nl<br />
<br />
23<br />
<br />
24<br />
<br />
CHAPTER 3. FIELDWORK IN ACADEMIC RESEARCH<br />
<br />
three senior members were a university senior lecturer, an assistant professor, and a senior researcher. These six researchers may give a biased view on the quest for information, since they are professionally familiar with computers. It may result in a more dedicated and experienced behaviour. For instance, they would use the ability to tune their user model manually to improve the results. Yet, the investigation is sufficiently accurate for our type research, since the focus is on the personalisation (see subsection 1.2).<br />
<br />
3.1.2<br />
<br />
The interviews<br />
<br />
For the first topic of the investigation (how do researchers currently quest for information?) the following three items were explored. • Why do researchers quest while conducting a project? • How do researchers search for information? • What happens when a possible treasure is found? For the second topic of the investigation (a personalised quest for information during academic research) we focussed on the following three items. • How do researchers would like to have personalised support? • What are the issues considering trust in a personalising system? • What is the effort a researcher is willing to spend? To receive a detailed and sufficiently large amount of information, we decided to conduct one-toone interviews with mainly open questions2 (see appendix A.2). This method has the advantage that an interviewee can answer freely and provide a large amount of details (Schreiber et al., 1999). The interviewer can ask for more information when needed. An example interview is available in Dutch in Appendix A.3. We started with a pilot interview to measure the time of the interview. It took approximately one hour and a quarter. We made a few minor changes, so the results are not included in this project. Then, the actual interviews followed. The researchers were informed about the topics of the interview, their quest for information during academic research, and their preferences on personalised support. They did not need to make any preparations. To focus optimally on the contents of the interviews, each one has been recorded with a video camera. Optionally, the lense could be faced to a wall, if the camera would make the interviewees feel uncomfortable. None, except one, used that option since it was assured that the video was used only for afterwards processing. The actual interviews took place between May and July 2004. 2<br />
<br />
In case of questions with a predefined list of answers, the list included the option ‘otherwise’.<br />
<br />
3.2. THE QUEST FOR INFORMATION<br />
<br />
25<br />
<br />
Figure 3.1: Need for knowledge cycle.<br />
<br />
3.2<br />
<br />
The quest for information<br />
<br />
The quest for information is incorporated in a process of obtaining knowledge (see figure 3.1). It starts from a need to know something (1). Then, a researcher searches for appropriate information (2), i.e., a source that contains the required knowledge (in this thesis, we refer to it as a treasure). When it is found, it may be learnt (3). In case the information is not adequate, a researcher starts searching again. Finally, when the knowledge is obtained, the goal is accomplished (4). Personalisation may play a role in searching for appropriate information, and preferably anticipate to what else might be relevant (see definition 2.1.2), in order to provide a complete source of information. We distinguish between two types of such sources. On the one hand, written-word sources, such as online or printed articles and books. On the other hand spokenword sources, which alludes to knowledge sharing with colleague researchers. The remainder of this section reports on the results of the interviews in the following three subsections. Subsection 3.2.1 covers the current quest for information. Here, we briefly explore how the researchers conduct a project, with special interests in the items when and why they search for information. Subsection 3.2.2 explores how the researchers quest for information. Subsection 3.2.3 discusses spoken-word resources. In subsection 3.2.4, it is described what happens when a possible treasure is found.<br />
<br />
3.2.1<br />
<br />
Why do actors quest?<br />
<br />
This subsection describes why researchers search for information, while conducting a project. During the interviews, it transpired that the quest for information is significantly different for<br />
<br />
26<br />
<br />
CHAPTER 3. FIELDWORK IN ACADEMIC RESEARCH<br />
<br />
senior researchers on the one hand, and Ph.D students (junior researchers) on the other hand. A comparable study showed similar results (King and Hanson, 2002): the Ph.D. students actually search for all types of information, whereas the senior members prefer to keep themselves up to date with scientific news from the field. So, we start with a Ph.D. student’s quest, followed by the quest of a senior researcher. At the end, we will formulate four goals in searching for a treasure. Ph.D. students Each of the Ph.D. students started their project with a limited amount of background knowledge. Their first aim was gaining familiarity with the research field. Appropriate sources for this search are (1) the project proposal, and (2) the supervisor, who may recommend relevant articles. Thereafter, the Ph.D. student continues to search independently. Then, the aim moves to finding an appropriate interpretation of the research project. That involves an adequate formulation of research questions which are sufficiently interesting for the following four years of the investigation. Hereafter, more background knowledge may be searched in order to complete the literature study. During the above mentioned quest for general information (in the direction of the research project), the Ph.D. student learns the typical expressions used in the field, and its major developments. Moreover, he is starting to recognise some important authors. Then, at a point where the project moves to a next stage, that is when the own investigation starts, the background knowledge should be sufficient. When a Ph.D. student searches for further information for his project, the search can be formulated straight to the point, since he knows what he is looking for. An example is a discussion on certain techniques that can be used to implement an algorithm. Besides this quest, one wants to keep himself up to date with new developments from the field of research. These two types of search for information are encountered with the senior researchers as well. This is described below. Senior researchers During each project, senior researchers make efforts to keep themselves up to date. Since a new project usually evolves from a previous one, the background knowledge should be for the most part sufficient: common literature is known, or should be recovered without any trouble. The senior researchers mentioned two other searches for information: (1) when a new project, or a new component in a project, starts; the information about that specific subject is searched in order to be fully up to date, and to assure the project is innovative and new; (2) it may be required to situate a project in related areas; for instance, to complete a project proposal. These related areas do not belong to the field of expertise of the researcher. When senior researchers search for information in the field of expertise during a project, they are able to formulate it in detail (straight to the point), since it is known what they want to find.<br />
<br />
3.2. THE QUEST FOR INFORMATION<br />
<br />
27<br />
<br />
Four search goals From the aims which the researchers expressed, we identify four search goals in questing with respect to information during a project. • Obtaining background knowledge: aims at finding general information, such as major developments, in order to become familiar with the research field. • Remain up to date in the field of research: aims at keeping up with recent developments from the field. The sources of information are similar to the background knowledge, but have a later publication date. • Become acquainted with related areas: aims at finding information with a different background related to the current research. To put it in another perspective, it concerns information that is not directly relevant to the researcher, but it is important for other researchers, from other fields, who may be interested in the project of the searching researcher (see Appendix A.3). • Find specific information: aims at finding specific information on a certain subject. This search can be formulated straight to the point, since the researcher knows what treasure he is looking for. Each of these four goals may lead to a different information source which is relevant for the goal of the research. Using personalisation may result in a sharper distinction between these four goals and to specific information sources.<br />
<br />
3.2.2<br />
<br />
Written-word resources<br />
<br />
A part of the interview questioned how researchers seek their written-word resources. Together, they have pointed out seven methods, which are described below. 1. Search the Internet with a set of appropriate keywords for relevant information. Google is mentioned most frequently. When researchers are not familiar with expressions used in a certain field, finding an appropriate set is a challenge. For example, a new idea might have been explored with another name. 2. Consult the personal directory (see subsection 3.2.4). 3. Evaluate relevant references from an article. CiteSeer is mentioned as a convenient tool, due to a “cited by”-attribute with each article. But, not all researchers were satisfied with the detection of new articles. 4. Scan authoritative magazines to find relevant articles. Though, for some research fields, there are not yet such magazines. And for a beginning Ph.D. student, they may be recommended by his supervisor. 5. Keep up with webpages of important authors. Again, when new to the field, it is a question who is who? One interviewee pointed to a circle of searching authors. For example, when three important authors explore only the other two, they may miss other new developments.<br />
<br />
28<br />
<br />
CHAPTER 3. FIELDWORK IN ACADEMIC RESEARCH<br />
<br />
6. Visit a conference, and present the own research (see also 3.2.3). Here, the state of the art is presented, which makes it an adequate method stay up to date. 7. Check references provided by a referee, when an article is submitted for a magazine or conference. A referee may point to overlooked relevant information. Now, how do these methods relate to the search goals, described in the previous subsection? Each of the abovementioned methods is applicable to find background knowledge and specific information. For the quest for information from related areas, explicit written-word resources are not mentioned. Given that the expertise is not located in the related areas, this quest may be related to the search for background knowledge, constrained to a certain subject. To remain up to date, each method, except consulting the personal directory, can be adopted. Note that when a resource, often a paper, is not fully available online, the library is consulted or the paper is ordered. In section 3.3, it will be explored where the researchers would like to have personalised support.<br />
<br />
3.2.3<br />
<br />
Spoken-word resources<br />
<br />
Questing for spoken-word resources alludes to knowledge sharing between researchers. We interviewed the researchers about which parts could be supported by personalisation. The interviewees immediately informed that one does not share the same knowledge with a colleague as with a researcher from another, perhaps competitive, university. So, this subsection is split up in knowledge sharing with (1) colleagues, and (2) other actors. The colleagues’ spoken words Knowledge sharing with colleagues is a social event. Meetings, paying a visit, and using of existing tools, such as a shared map and e-mail, are adequate methods to share knowledge with colleagues. The researchers pointed to three situations where personalisation may fit in order to benefit more from the knowledge of colleagues within the scope of academic research. • Detection of interests. Knowing that a colleague shares a similar interest, can be a reason for a social event. However, it is not known to what extend researchers do catch the similar interests from the colleagues. • Results of a colleague’s quest. A researcher is interested in what information another researcher, who is investigating a similar or related area, has found: it may contain information which is suitable for him too. This is particularly advantageous when the other researcher has a large “knowledge store”, such as the supervisor for a Ph.D. student. Moreover, if a personal judgement is available, such as an annotation, a researcher may consider this as a valuable result3 3<br />
<br />
We have no information concerning to what extent the annotations are digitally available. Often, tools have to be applied to obtain them. However, annotations in a separate file, such as an online conversation and self-written summaries, are encountered.<br />
<br />
3.2. THE QUEST FOR INFORMATION<br />
<br />
29<br />
<br />
• Preservation. When a colleague leaves the department, it is too bad that his resources are leaving as well. Even if an effort is made to keep them, e.g., the computer with its programs and databases, the databases are frequently not sufficiently accessible to be approached by others. Although questing for colleagues’ spoken words is a social event, applying personalisation on the resources and annotations of a colleague may provide the researcher with support for this quest. It does require that the colleague is willing to sacrifice privacy. The interviewed researchers themselves explicated that they are willing to do so, given control of access rights. The other actors’ spoken words For junior researchers, who are new to a field, it is a question of who is involved in relevant research, and what is that research about. Next, contacting these researchers is a social event. For that matter, they agree to be contacted by other researchers. Senior researchers have already built a network of acquaintances. Sharing knowledge with other researchers mainly occurs on conferences. The attendants are researchers with mutual interests, and there is time for conversation. A telling instance is the feedback on conference presentation. For the matter of being contacted, the least we can say from the interviews, is that the researchers are willing to respond to a direct question. One senior researcher spontaneously mentioned to use mailing lists for knowledge sharing with others. But, we did not further investigate in any type of such electronic networks of practice.<br />
<br />
3.2.4<br />
<br />
What happens with the treasure?<br />
<br />
During the interviews, we asked what the researchers would do when they had found a possibly relevant document. The results are shown below in two parts: (1) methods for preservation, and (2) selection criteria and preferences. Methods for preservation When a possibly relevant document is found, researchers usually want to preserve it. Together, they pointed to three methods that can be applied jointly. • Save the document to the hard disk. To keep an overview over all the saved papers, they are kept (1) in one huge map, labelled in the form of author and title, or (2) in a structure of maps that correspond to their topics. Optionally, BibTeX4 information of a document is added to a personal file. • Print the document. The printed papers are organised in a similar way as the papers on the hard disk, i.e., classified by subject or put on one big heap. Optionally, annotations are added. 4<br />
<br />
A file format to store mainly administrative attributes of a paper.<br />
<br />
30<br />
<br />
CHAPTER 3. FIELDWORK IN ACADEMIC RESEARCH<br />
<br />
• Bookmark the site where the document is found. But, due to the instability of the Internet, bookmarking is considered to be tricky. The method of saving documents in a structure of maps has the disadvantage that topics may overlap. The effort needed to find a document for a second time was satisfactory for three of the interviewees. Though, when asked further, a tool that supports search and selection such as on words in text, authors and abstract, was found more convenient5 . Examination of a document We questioned the researchers about how they examine a document. First, we investigate six criteria, besides the content, that may have an effect on the relevance of a paper, viz., title, abstract, author, yes of publication, type of publication, and magazine. Next, it is described how the researchers read a paper. The most selective criteria of a paper are the title, and the abstract. When these seem irrelevant, the paper is usually not further investigated. The authors of a paper is a criterion that may make the researchers read a paper, even if the title and abstract did not seem relevant at first sight. The year of publication is merely a preference: recent papers are preferred over the older ones. Old papers, before 1990 for computer sciences, are considered useful when it is a “basic” paper, such as a review or the introduction of a technology. The type of publication of a paper, may have an effect on relevancy because of three reasons. First, researchers avoid referring to position papers (not published) and web journals. As a matter of fact, only resources that are retraceable offline are considered. Second, although articles from journals are preferred over articles from proceedings, proceedings in the area of computer science are significantly more up to date. That makes them more suitable to stay up to date. Third, the type of publication may point to a content specification, such as, a technical reports contains a large amount of detail. The magazine in which a paper appears is not a decisive criterion. Only the content of the paper counts. However, when the background of a magazine and a researcher do not match, the articles are not evaluated because both sides have another method of approach. When a paper has catched the eye of a researcher, four actions mostly performed are: (1) read the abstract, (2) read the conclusions, (3) scan the paper, (4) examine the article closely. This sequence can be interrupted at any moment; when considered irrelevant, one continues to the next paper.<br />
<br />
3.3<br />
<br />
Issues of personalising academic research<br />
<br />
This section investigates three issues of personalising academic research. First, we establish the focus of further academic research of personalisation. Second, two requirements for a personalising system are described. Third, the effort which a user is willing to spend in enabling personalisation is examined. 5<br />
<br />
At the time the interviews were conducted, there was not yet free software easy available to search on words in text. Currently, three large companies have released such software. So, these answers may have changed<br />
<br />
3.4. AN ANSWER TO THE SECOND RESEARCH QUESTION<br />
<br />
3.3.1<br />
<br />
31<br />
<br />
Focus<br />
<br />
The interviewed researchers were explicitly asked to rank the three main resource components in accordance with their preference for personalised support. The general list was as follows: (1) support for questing written-word resources, i.e., the literature study, (2) spoken words of colleagues, and (3) other actors’ spoken words (3). The first was identified unanimously as most preferred, the numbers (2) and (3) are both tied on a second place. From these results, we decided that the focus of this thesis would be on personalising the quest for information during the literature study.<br />
<br />
3.3.2<br />
<br />
Two requirements<br />
<br />
The researchers were sceptical on the fact that personalisation might narrow their sight. For instance, relevant but not related results they currently may encounter while questing for information, may be filtered out using personalisation. To provide the researcher with a reason to trust the personalisation, it should clarify its results. Furthermore, the researchers would like to retain the explorative behaviour of the quest for information. Some influence of the process of personalisation, such as effectuated by tweaking the search path and modifying his own user model is important to maintain. We understood this as a request for interaction with the support system. So, we arrived at two requirements, viz. (1) trust the personalisation system, and (2) ensure sufficient interaction with the personalisation system.<br />
<br />
3.3.3<br />
<br />
Effort<br />
<br />
In general, actions to let the system know in what topics the researcher is interested, should not interrupt his task of finding literature. This group of actions consists of all convenient actions that may lead to a direct result, such as providing key words. Also, an optional action that may improve future results, such as a button click to assure that a result is (not) relevant, is a convenient action. However, a significant effort may be required to inform the system about a new user, in order to make a good start. For a senior researcher, it should be equipped with information such as his expertise and current research. The researchers found it acceptable to train the system. A junior researcher, has a “light” profile since he is in the first stage of constructing his background knowledge. So, they can learn gradually together, while he is using the system.<br />
<br />
3.4<br />
<br />
An answer to the second research question<br />
<br />
In this chapter we investigated in the second research question ‘ Where do researchers desire personalised support in their search for information? ’. The answer was based on a series of interviews with six researchers. The first aim was to find out how researchers currently perform their quest for information during academic research. We found four goals why they quest: (1) to obtain background knowledge, (2) to remain up to date, (3) to find related areas, (4) to find specific information. Then we continued with routines to search for written-word resources.<br />
<br />
32<br />
<br />
CHAPTER 3. FIELDWORK IN ACADEMIC RESEARCH<br />
<br />
This is followed by an investigation of how they would like to be supported by in spokenword resources, i.e., knowledge sharing with (a) colleagues, and (b) other actors. Finally, it turned out that a literature study was most preferred. Therefore, we focus on this topic for a personalisation system. The answers by the interviewees provided an insight into what to personalise, how to deal with trust, and what effort a researcher was willing to spend.<br />
<br />
Chapter 4<br />
<br />
Our model and its factors<br />
<br />
This chapter investigates the third research question. How can a personalised quest for information be achieved? An answer to this question is supported by our model. The focus of our model is on personalising the quest for information during the literature study of the researchers. This component is most preferred by the researchers (see section 3.3). The design of our model is based on a personalisation strategy which is formulated below in four steps. 1. Structure the information in order to identify which information is stored at what place. Here we deal with item-related factors 2. Find information about the researcher that indicates the information he needs. Here we deal with researcher-related factors. 3. Select the information that may enrich the current information items of the researcher. Here we deal with the researcher’s treasures. 4. If the researcher is not satisfied with the results, repeat steps two and three. The remainder of this chapter investigates the above-mentioned steps. A general discussion on these factors is in chapter 2). Section 4.1 investigates the item-related factors. Section 4.2 explores the researcher-related factors. Section 4.3 covers the researcher’s treasures. Finally, section 4.4 provides an answer to the research question.<br />
<br />
4.1<br />
<br />
Item-related factors<br />
<br />
The first step of the strategy asks for structuring the information items. In section 2.3, we discussed three topics regarding item-related factors, viz., (1) the content of the information items, (2) the representation, and (3) the metadata. In the context of our model on personalisation they are explored below, followed by a proposal on a structure for the information items in our model. 33<br />
<br />
34<br />
<br />
CHAPTER 4. OUR MODEL AND ITS FACTORS<br />
<br />
The items of a literature study are documents; it is characteristic for the content of a document to cover more than one subject regarding a certain research domain. Below, three content-related factors are explored: (1) type, (2) structure, and (3) validity. The preferred type of document is a scientific research paper, such as an article or a conference paper. Hence, the information is semi-structured : each item contains a title, the authors, an abstract, (other) sections, conclusions, and a list of references. Furthermore, the validity of the content remains unchanged. However, since a research domain advances, the items may become obsolete. It is obvious that the content of an item is significant for the personalisation in a literature study. The characteristic of the content (see above) makes that both representation strategies are applicable: a top-down strategy emphasises the research domain of a document, and a bottom-up strategy emphasises the subjects of a document. We have a preference to retain each subject, which, in case of generalisation, may be neglected. Hence, we follow a bottom-up strategy. In our model, we assume that the items are digitally available. For this purpose, there is an infrastructure which provides metadata. The infrastructure, e.g., a retrieval system, requires a name for a document, and it usually keeps track of administrative metadata. An example of metadatum is the attribute ‘creation date’. Hence, metadata may facilitate the adequate operation of our model. The goal of personalisation is to provide the researcher with the information needed. Regarding the facts that (1) each document found is not tailored to the needs of the researcher, and (2) a document may cover more than one subject, we choose to represent each document as a structure that consists of “groups of words”, e.g., a section, that belong to a document. This makes it possible to compose a resultant set that consists of sections of documents which cover the subject that a researcher is searching for.<br />
<br />
4.2<br />
<br />
Researcher-related factors<br />
<br />
The second step of the strategy asks for finding information about a researcher that indicates the information he needs. In section 2.2, we discussed four topics concerning the user and the personalisation, viz. (1) the user himself, (2) the system, (3) acquisition methods, and (4) the user model. They are investigated below.<br />
<br />
4.2.1<br />
<br />
The user<br />
<br />
This subsection investigates what is important to know about the researcher that may influence personalisation. It regards three factors, which were discussed in 2.2.1: (1) the personal data, (2) the interests, and (3) the preferences. Personal Data Since the focus of our model is on a literature study, and not on knowledge sharing, we may assume that only the current researcher uses a system in which our model is implemented. Therefore, identifying who the current researcher is, is not required. However, a personal data record that identifies the researcher as a junior researcher or senior researcher, will be made for reasons of bookkeeping the research.<br />
<br />
4.2. RESEARCHER-RELATED FACTORS<br />
<br />
35<br />
<br />
Interests During a literature study, the interests of a researcher are in the benefit of the research project. Since a new project evolves from a previous one, the expertise of a researcher will define the context of a search. Besides, the researcher has a number of subjects of his interest that may change frequently while performing the literature study. Hence, there are two streams of interests that have an impact on the relevance of an item: • the expertise, i.e., long-standing interests, and • a current subject of interest, i.e., a short-standing interest. When a certain subject of interest appears frequently, it becomes automatically a part of the researcher’s long-standing interests. On the contrary, when the expertise of a researcher clearly has been changed according to the criteria of our model, the old long-standing interests should be forgotten. Preferences We have identified four search goals, i.e., reasons why a researcher may perform a quest for information (see 3.2.1). The goals may influence the long-standing interests and the shortstanding interests. We start remarking that both are not equally important for predicting the relevance of an item. For example, when a researcher is searching for specific information, a relevant item usually has a context that matches with the expertise of a researcher. But, when the search goal is on related areas, relevant items may be those that have a context which is less related to the expertise of the researcher. Preference settings that support a range of important issues with respect for the interests streams, e.g., by a weighting preference for each interest stream, may lead to a better distinction between the search goals. However, for actual support of the search goals and their specific information resources, further investigation is needed. Then, preference settings for the layout of the resultant set based on examination criteria (see 3.2.2) may improve personalisation, but layout issues are not further investigated in this thesis. In summary, regarding the second step of the strategy followed by our model, we establish that researcher’s information concerned with (1) long-standing interests, and (2) short-standing interests, indicate which items may be relevant for him. The acquisition and modelling of this information, will be discussed in subsections 4.2.3 and 4.2.4.<br />
<br />
4.2.2<br />
<br />
The system<br />
<br />
The task of a personalising system is to support the researcher in his quest for literature. Hence, it is content-oriented. In general, a researcher will use a support system if it takes him less time to search for the required information than when searching on his own. This implies that the factors time to spend and importance are dependent on the search task. For instance, a researcher is willing to spend time to separate the good from the bad results if the information is difficult to find. If the results are satisfactory, a researcher will use the system regularly during a literature study (frequency of use). Furthermore, regarding the process of<br />
<br />
36<br />
<br />
CHAPTER 4. OUR MODEL AND ITS FACTORS<br />
<br />
usage, we assume that a researcher takes the initiative to request information. However, if the information items that are available for personalisation are automatically updated, then, the system may present by itself new relevant information, without an explicit user request.<br />
<br />
4.2.3<br />
<br />
The acquisition<br />
<br />
In subsection 2.2.3, five acquisition methods to obtain information about a researcher are explored, viz. (1) groups of users, (2) colleagues, (3) classification, (4) ask the user, and (5) observation. In the context of our model, these are discussed below again. Since we assume that only the current researcher uses the personalising system, the acquisition methods group of users, and colleagues are not applicable for this scenario. The method of classification for acquisition requires a distinctive feature to identify the expertise of a certain researcher. Besides a rough separation, e.g., classification by research domain, the expertise of a certain researcher is too peculiar to be efficiently derived from a predefined set of (a) papers, for a training set, and (b) characteristics and assumptions, for a stereotype model. Hence, these methods are inadequate for finding a researcher’s interests. The above elimination of three methods leads to the acceptance and incorporation of two acquisition methods in our model, viz. ask the user and observation. The first method may interrupt the researcher in his task; this may be a severe drawback, unless the required effort is minimal and the method is optional. So, our model requests explicit information that requires a greater response effort from a researcher before the start of the task. Next, during the task, the researcher may provide explicitly relevance feedback on items in the resultant set. The second method, observing the user, applies when a researcher does not provide explicit relevance feedback on a result item in which he shows interest. Such interests not evaluated adequately may influence the information of a user model, but it should be easily overvoted by the rated items.<br />
<br />
4.2.4<br />
<br />
The user model<br />
<br />
In this subsection, we investigate the topics that were discussed in 2.2.4, i.e., the representation of a user model, and the three stages in its life cycle, viz., initialisation, maintenance, and destruction. Representation The user model has to be capable of handling the two streams of interests. Long-standing interests will ask for representing the most significant information, and the short-standing interests will ask for the user model to be sufficiently flexible. Hence, of the four representation types discussed in 2.2.4, an algorithm is most appropriate for our model. Initialisation The goal of initialising the user model is to obtain user information regarding the long-standing interests. In subsection 3.2.1, we have seen that junior researchers and senior researchers have different information sources available at the start of their project. So, in order to fill in the<br />
<br />
4.2. RESEARCHER-RELATED FACTORS<br />
<br />
37<br />
<br />
user model with adequate user information, we start the initialisation with the researcher’s available resources. Since the initialisation takes place at the beginning of the literature study, we may ask for information that requires some response effort of a researcher. • For a junior researcher, the project proposal, supplemented with recommendations from the supervisor, are available. We note that the long-standing interests are constructed by the junior researcher, in cooperation with the user model. • The expertises of a senior researcher are derived from (1) his own written articles, and (2) his personal literature database. When digitally available, a personal library stores information that is likely relevant according to the researcher. Although the stored information may be not completely known by the researcher (that classifies it again as possibly relevant treasures, if a certain subject occurs regularly) it is likely that the researcher is familiar with it. Maintenance In our model, the maintenance of a user model has to support the short-standing interests and the long-standing interests. For each of these, a maintenance program is described below. For maintaining the short-standing interests, the user provides (1) information about his current subject of interest, in order to initiate the request, (2) explicit relevance feedback on resultant items, and (3) information through his behaviour. The user model employs this information in real-time modality, in order to represent adequately the quick changing shortstanding interests. This type of maintenance is actually cooperative maintenance. Automatic maintenance is applied for the long-standing interests: with each change of current subject of interest, an inventory is made of the most significant information. If the inventory matches with a long-standing interest, the long-standing interest is confirmed again. If the inventory does not matches, but occurs frequently given a series of requests, it becomes part of the long-standing interests. The modality of this maintenance is a batch-update. However, if a batch update requires too much time, an offline update may be applied. Furthermore, the inventory and the long-standing interest need a gradual ‘forget’-mechanism, so that new user information is treated as more importantly than older user information (Koychev and Schwab, 2000). For example, the interests in the inventory are gradually forgotten over a number of rated documents, and the long-standing interests are gradually forgotten over a period of time. However, since certain projects may rely more on certain parts of the long-staning interests, it may be appropriate to include a ‘temporary forgotten’ memory for the long-standing interests: these interests are preserved but have no influence in personalisation. In case they are confirmed by new user information, they can find their way more easily back into the long-standing interests. Further research may prove whether this is necessary. Destruction The destruction of the user model is a stage in the life cycle that might influence the process of personalisation if we focussed on knowledge sharing as well. Since that is not the case, the destruction implies that if a researcher starts using the system again, the user model has to be built from scratch on.<br />
<br />
38<br />
<br />
CHAPTER 4. OUR MODEL AND ITS FACTORS<br />
<br />
4.3<br />
<br />
The researcher’s treasures<br />
<br />
The third step of the strategy is “select the information that may enrich the current information items of the researcher”. Recall that section 2.4 discussed three different approaches for personalisation, viz. (1) the content-based approach, (2) the collaborative approach, and (3) the extension-based approach. Below, we consider them in the context of personalisation for researchers. Next, a description of our model is provided.<br />
<br />
4.3.1<br />
<br />
Content-based approach<br />
<br />
In the previous sections we made the following observations. The context of a document is in a certain research domain, and a “group of words” (further referred to as a section) handles about a subject. The expertise of a researcher reflects the context of a search, and for a quest, a researcher investigates a current subject of interest. So, we may assume that a document with the following properties is possibly a treasure for a certain researcher: • the context of a document is related to his expertise, and • certain sections are related to his current subject of interest. This match suggests a content-based approach for personalising a researcher’s quest for information. Applying this approach means that we have to deal with over-specialisation. Since the user model is capable of extracting the short-standing interests, we may enrich the current subject of interest using that information, which leads to other sections that are related as well. This may alleviate the disadvantages of over-specialisation.<br />
<br />
4.3.2<br />
<br />
Collaborative approach<br />
<br />
The collaborative approach requires user information of other users for personalisation. In the context of personalisation for researchers, the group of ‘other users’ may be (1) the colleagues, and (2) the other researchers. However, the number of colleagues is too small to overcome sparsity, and privacy issues may prevent the sharing of personal information with any other researchers. Nonetheless, a collaborative approach is possible for personalising a researcher’s information items: since scientific papers are written by authors, his (artificial) user model can be constructed automatically, e.g., based on citations (applied in experiments by McNee et al. (2002)). Although this approach has the benefit of serendipitous results, the interests of a researcher that are included in personalisation is dependent on the mutual interests with other user models. So, we assume that the collaborative approach may facilitate the quests for background and new information, but it is less appropriate for quests on related areas and for specific information.<br />
<br />
4.3.3<br />
<br />
Extension-based approach<br />
<br />
The extension-based approach uses additional information, instead of information from the components, the users, and the items. A user-based extension needs information regarding what the personalised items for a user are. For a researcher, the personalised information needed changes continuously, so this method is inefficient. An item-based extension may meet the user needs regarding an item, not a user in particular. Consequently, the items typically have distinctive attributes besides the content. The examination criteria for a document proved<br />
<br />
4.3. THE RESEARCHER’S TREASURES<br />
<br />
39<br />
<br />
that the content is most important, and that the other criteria are not sufficiently distinctive for researchers (see 3.2.4). Hence, the items do not possess the necessary qualities that enable this method for personalisation.<br />
<br />
4.3.4<br />
<br />
Our model<br />
<br />
In our model, the items structure a document as sections, and the user model distinguishes the long-standing interests and the short-standing interests of a researcher. To initialise the user model, the researcher is asked for information to fill his long-standing interests, since it may take a longer period of time to identify them automatically. A quest starts with information about the current subject of interest. Then, our model suggests a personalised resultant set with items that proves to be related with short-standing interests, and that belong to a document that is related to the long-standing interests. The order of the items in the resultant set are preferably evaluated in advance in order to provide the researcher with a diverse set: the match between a result item and the short-standing interests is relatively high, and the match between the mutual result items is relatively low. For a resultant set, the researcher may provide feedback by (1) relevance feedback, and (2) his behaviour. This feedback is processed by the user model in the short-standing interests, and should lead to a more accurate representation. If preferred by the researcher, the description of the current subject of interest is enriched. Through these changes, an improved resultant set can be selected. When the researcher has found his treasures, the researcher may start a new quest. But before this, an inventory of the most significant information of the short-standing interests is constructed. This inventory may invoke a maintenance update for interests that prove to be long-standing interests. The remainder of the inventory is left for further consideration. We suggest that a researcher should be offered control regarding the total matching score. This score consists of the two normalised scores for the interest streams. For each of these, a weight can be applied in order to change their importance. Logically, the maintenance mechanism should apply these factors for updating the interests as well.<br />
<br />
Chapter 5<br />
<br />
Testing our model<br />
<br />
By testing our model, we want to find out whether our model might provide personalising support for a researcher during his quest for information. In general, testing personalisation requires that a great number of participants perform evaluations since the results are subjective. So, for this investigation, a proof of concept for our model is provided. To experiment with our model, we developed a prototype that supports the first steps of personalising information for a junior researcher. Section 5.1 describes the prototype in detail. Section 5.2 describes how the experiments are conducted. Section 5.3 provides opinions on the experiments and an evaluation.<br />
<br />
5.1<br />
<br />
The prototype<br />
<br />
In order to personalise the first few searches of a researcher, the prototype consists of three components: (1) the items, (2) the search object with a description of what the researcher is searching for, and (3) the memory that models the short-standing interests. The initial state of the prototype is a set of items and two empty components for user information. The strategy of the prototype to personalise the items for a researcher is described below. It consists of eleven steps. 1. A description of the search interest is added to the search object. 2. A resultant set is selected based on the search object. 3. For each item in the resultant set a matching score with the memory is calculated. 4. The items in the resultant set are arranged in descending order, according to their matching score. 5. The resultant set is shown to the user. 6. The user may provide explicit feedback. 7. The explicit feedback is used to update the memory. 41<br />
<br />
42<br />
<br />
CHAPTER 5. TESTING OUR MODEL<br />
<br />
8. An enrichment for the search object is selected from the memory. 9. The enrichment accepted by the user is added to the search object. 10. The user may adapt his description of interest. 11. A new resultant set may be selected again. The remainder of this section is as follows. Subsection 5.1.1 describes how the items make their way into the prototype. Subsection 5.1.2 regards how the prototype handles the search object and the memory. That are step 1, and steps 6 to 9 of the strategy above. Subsection 5.1.3 explores how the prototype personalises a resultant set, i.e., steps 2 to 5, and step 11. Notice that we focus on the first few searches. Therefore, the short-standing interests are regarded, and not the long-standing interests.<br />
<br />
5.1.1<br />
<br />
The items<br />
<br />
The items start as a set of documents in portable document format. This set is first converted to a set of text files. Then, each document is divided into sections based on new line breaks. A section that is shorter than five words is usually a section title or a caption. So, it is appended to the next section. After that, the stop words are removed, and a porter stemming algorithm reduces each word to its root form (Porter, 1980). Now, the items are ready to be stored in a database. Remark that the conversion of a document in portable document format to text files may not succeed, or it may produce unusable documents. Usually, these documents are removed automatically. Each section is stored in the database of the prototype. It is represented as a collection of word pairs that belongs to a certain document. The word pairs form an undirected weighted graph: a word is a vertex, and the weight of an edge is the number of occurrences of a word pair in the section. For example, the title of this section ‘testing our model’, is stored as [test, model] = 1.<br />
<br />
5.1.2<br />
<br />
The user model<br />
<br />
This subsection describes how the prototype implements step 1, and steps 6 to 11 of the personalisation strategy. In figure 5.1, a screen shot of the user model in the prototype is shown; it consists of the search object (the upper side of the screen shot), and the memory (the under side of the screen shot). Step 1 of the strategy is implemented as follows. At the start of a quest, the researcher initialises the user model with a description of what he is searching for. This is recorded in the search object in a similar way as a section is stored: the stop words are removed, each word is reduced to its root, and the representation is an undirected graph of word pairs. According to step 6 of the strategy, a researcher may provide feedback on the items in the resultant set; an item may be evaluated as follows: (1) relevant, (2) not relevant, or (3) no evaluation. This information is used to update the memory as follows (step 7). For each word of the item, the weight of the word in the memory is changed with +1 (increase) or −1<br />
<br />
5.1. THE PROTOTYPE<br />
<br />
43<br />
<br />
Figure 5.1: Screen shot: a user model of the prototype.<br />
<br />
(decrease), corresponding to the relevancy. An item that does not receive feedback has no influence on the memory. A more detailed pseudocode can be found in figure 5.2. The memory component contains information about which words occur frequently. These words are probably important for this quest, so they may be an enrichment available for the current search object (step 8 of the strategy). In the prototype the enrichment contains at most twenty words in total. The following procedure is applied to enrich the search object (see figure 5.1 upper right corner). First, the words of previous iterations that were accepted by the user as an enrichment remain in the current enrichment. Next, the enrichment is supplemented with words that have the following characteristics: they have the highest positive weight according to the memory, and they are not yet recorded anywhere in the search object. Step 9 of the strategy prescribes that the enrichment that is accepted by the user is added to the search object. Subsequently, step 10 is introduced to give the researcher the opportunity to adapt his description of his interest for a next iteration. When the researcher asks for a new resultant set (step 11), the strategy starts a new iteration, and the adapted description is processed in step 1. The influence of the above described actions should not last forever, so we equipped the prototype with the following mechanism to forget when necessary. • In the memory: the weight of a word of one iteration is forgotten after four iterations, depending on the word frequency. After each iteration, the memory is revalued and<br />
<br />
44<br />
<br />
CHAPTER 5. TESTING OUR MODEL<br />
<br />
restructured, i.e., the weight of each word is halved, and words with a value between [−0.1, 0.1] are removed. • Sections that are evaluated do not have to occur in the next resultant set, they are known by the researcher, and they are already named relevant. But, it may be necessary to reconfirm the relevancy after a number of iterations (we have chosen 1, 2, and 3). A certain section is evaluated once, but is forgotten after four iterations, if no reconfirmation has taken place. Of course, the section may be selected in the resultant set again in a new iteration. • A word that is excluded from the extension is forgotten after one iteration. The memory evolves quickly enough and the word may take a different position in the memory in a next iteration. We expect that the additional rules provides the prototype with a user model that represent the current subject of interest that is sufficiently accurate to personalise the first searches for information of a researcher. Based on the information in this user model, a personalised resultant set is composed. That is described in the following subsection.<br />
<br />
5.1.3<br />
<br />
A personalised resultant set<br />
<br />
This subsection describes how the prototype implements steps 2 to 5 of the personalisation strategy. The goal is to provide the researcher with a personalised resultant set, based on the information from the user model. Step 2 of the strategy prescribes to select a resultant set based on the search object (recall that the search object consists a representation of the researcher’s description of the current interest, and a set of words the researcher accepted from the enrichment). The prototype aims at 200 items for the resultant set. The selection of items is based on the following two criteria: • The first criterion is the set word pairs that are contained in the search object. For instance, the word pair ‘personalisation’ and ‘researchers’1 (see figure 5.1 left upper corner). • The second criterion is the set of single words (not word pairs) that are contained in the search object, i.e., the words in the description and the words in the accepted enrichment. The items for the resultant set is composed as follows. First, a set of 100 items is retrieved from the database by comparing the first criterion with the contents of the database. The selected items contain the largest number of occurrence of word pairs that match with the search object. It may happen that fewer than 100 items are available. Next, a number of items to retrieve 200 items in total is selected based on the second criterion. These items contain words that are found in search object. There are various procedures for such a selection. We have taken a straightforward procedure, by frequency only. Then, the two sets are merged to one resultant set. We remark that the resultant set does not contain any item that is evaluated, with respect to forgotten items. The first criterion (for selecting the first 100 items) is introduced because 1<br />
<br />
In the search object, the stop words are removed, and the remaining words are reduced to their root. So, technically, the word pair is ‘personalis’ and ‘research’<br />
<br />
5.1. THE PROTOTYPE<br />
<br />
Algorithm 5.1.1: ProcessFeedback(evaluatedItems) for each item ∈ evaluatedItems if item.IsRelevant = true for each word ∈ item do then W eightmemory (word) ← W eightmemory (word) + 1.0 = false if item.IsRelevant for each word ∈ item do do if word ∈ searchObject newweight ← W eightmemory (word) − 1.0 then if newweight < 0 then then W eightmemory (word) ← 1.0 else W eightmemory (word) ← W eightmemory (word) − 1.0 Figure 5.2: Updating the memory given an evaluated item set.<br />
<br />
Algorithm 5.1.2: CalculateScore(item) score ← 1.0 for each wordi ∈ item if wordi ∈ M emory then score ← score + W eightmemory (wordi ) do if wi ∈ SearchObject then score ← score + 1.0 for each wordpairi ∈ item if wordpairi ∈ searchObject do then score ← score + 1.0 score score ← count[item.words] return (score) Figure 5.3: Calculation of the score for an item.<br />
<br />
45<br />
<br />
46<br />
<br />
CHAPTER 5. TESTING OUR MODEL<br />
<br />
Figure 5.4: Screen shot of a resultant set for the researcher<br />
<br />
such items are relevant, but may not be contained in the top 100 of results of the set that is based on the second criterion. According to step 3 of the strategy, for each item in the resultant set, a score is calculated that is based on the memory. The algorithm regards the frequency of words in the item that occur in the memory and in the search object. A pseudo code for the calculation is provided in figure 5.3. The division of the score by the total number of words prevents that long sections receive a high score by default. We note that in this algorithm the score is dependent on the short-standing interests. Figure 5.4 provides a screen shot of a resultant set as it is shown to the researcher2 .<br />
<br />
5.2<br />
<br />
Conducting the experiments<br />
<br />
To conduct experiments, we invited two researchers to evaluate our model and the prototype. Besides an easy item set to personalise, the two experiments are similar. Therefore we include both experiments in the evaluation. The two researchers are both junior researchers who participated in the interviews as well. They were invited by e-mail to evaluate the prototype. Furthermore, they were asked to provide a set of search strings that are relevant for their expertise (see 5.2.2). The evaluation of the prototype opened with a description of our model. This information was provided as a presentation for one person. The researcher was asked to interrupt when he had a comment. After that, the researcher was asked to give his opinion regarding the distinction between long-standing interests and short-standing interests, with regards to the search goals. Next, a description of the prototype was provided, similar to the description in the previous section, followed by a browsing procedure by the prototype. In this way, the evaluating researcher was familiar with the functioning of the prototype before he evaluated 2<br />
<br />
Instead of two items, the researcher is presented five items at a time.<br />
<br />
5.3. OPINIONS AND EVALUATION<br />
<br />
47<br />
<br />
the prototype with his own quest. The comments and results of that evaluation provide the information to test a proof of concept for our model. Details are included in Appendix B. To fill our database with items for the experiments, we aimed at a set of 100 documents that are related to the expertise of a researcher. Since it takes some time to fill our database with items, that process is executed in advance as follows. For each search string which the researcher provided (at least five search strings were provided) we made an effort to download 75 documents from CiteSeer. When the total number of documents was not sufficient, the remaining number of documents were downloaded with a search string that contained the title of the project of the researcher (more information on these sets can found in Appendix B.2) We showed each evaluating researcher an overview of the documents in the database before they started to evaluate the prototype. The researchers recognised a few authors and a couple of document titles (some as relevant, and some as not relevant). For example, for item set B, the evaluating researcher regarded approximately five documents relevant for her research. So, even though they are familiar with the search strings (they have been investigating such subjects), the documents for the item set is not particularly representative for their longstanding interests.<br />
<br />
5.3<br />
<br />
Opinions and evaluation<br />
<br />
In this section, first the opinions of the researchers regarding our model are provided (subsection 5.3.1). More information can be found in appendix B.4. Subsection 5.3.2 provides the evaluation of the prototype. In appendix B.3 the experiments are described in detail; in appendix B.4, the evaluations of the prototype are given. Finally, subsection 5.3.3, discusses several topics, and concludes this chapter.<br />
<br />
5.3.1<br />
<br />
Opinions on our model<br />
<br />
The researcher responded positively on the distinction between long-standing interests and short-standing interests, in combination with the proposed four search goals. The argument that the context of a document is in a certain research domain was less convincing. The definition of research domain is subject to different interpretations. Regarding the personalising link, i.e., the context of a document is in accordance with the long-standing interests of a researcher (which may contain several topics of interest as well), the argument was found plausible. During the experiments, the researcher remarked that they search frequently on certain subjects. In our model, we did not provide an explicit method for keeping multiple shortstanding interests. Since, the researcher found this support valuable, we must concede that this a drawback of our model.<br />
<br />
5.3.2<br />
<br />
Evaluation of the prototype<br />
<br />
The evaluation of the prototype regards the following subjects: (1) the functioning of the prototype, (2) the memory, (3) the enrichment, and (4) the sections as items. These subjects are investigated below.<br />
<br />
48<br />
<br />
CHAPTER 5. TESTING OUR MODEL<br />
<br />
Functioning of the prototype Below, we provide five remarks. First, according to the researchers, the functioning of the prototype is rather intuitive. The reason is that distinction between the memory and the search object was not immediate clear to them. Second, there is adequate interaction between the researcher and the prototype due to the iterative behaviour of the prototype (see the two requirements in 3.3.2). Third, a researcher mentioned that the title of a document, which was situated above each section, led to providing mostly similar feedback for different sections. Another layout, for instance, a more distinctively marked section title, may prevent that. Fourth, it would have been an advantage if it was visible which words and word pairs in a section matched with the search object. Fifth, for the representation of the search object, historical information regarding how the search query changed is considered convenient. The memory The words and weights that appeared in the memory were representative with respect to the feedback. For example, while searching for ‘retrieval results ranking’, the memory presented for the word ‘answer’ a high positive weight, and for the word ‘image’ a high negative weight. So, the weights of significant words responds in relation to the short-standing interests. The researchers asked what happens with the short-standing interests, i.e., the memory component, when another quest starts. The prototype has no method implemented to handle that since it regards only the first few search iterations. According to the researcher, a solution is to preserve multiple memories. In that way, they may choose it again and continue their search. (We conducted an experiment that regards this issue. It can be found in appendix B.3, under the title ‘experiments on set B’, the second quest.) The enrichment The enrichment of the search object for the first few searches showed promising results for further research. The proposed enrichment contains useful additions for the search object. In the example above, the useful addition contained, amongst others, the word ‘answer’. But, in the first two search iterations, it also contained terms as ‘fig’ (the root of figure), and ‘journal’ (when a relevant section contains the reference list). The researcher regards the suggested relevant terms valuable since they improve the selection criteria, and they may help a researcher to think of other terms. Sections as items We asked the researchers about their opinion regarding our choice for sections as items, instead of documents. They gave three advantages. First, it is a small effort to read a small part, that supports providing feedback. Second, since the relevance of the sections of a certain document may differ, it is an advantage that a single section can be in the resultant set. And third, only those relevant sections can be indicated as relevant.<br />
<br />
5.3. OPINIONS AND EVALUATION<br />
<br />
5.3.3<br />
<br />
49<br />
<br />
Discussion<br />
<br />
In this subsection, we go deeper into several topics, viz. (1) our model, (2) functioning of the prototype, (3) the memory, (4) the enrichment, and (5) sections as items. Subsequently, a conclusion is provided. Our model In our model, frequently appearing short-standing interests are noticed, and when they meet other conditions, they added to the long-standing interests. So, the model may be supportive for the appearance of short-standing interests in the list of long-standing interests. As an example we mention, an implementation that applies a higher factor for calculating the weight of a word in the memory when it is present in the long-standing interests. However, in using this method, a quest suffers from a cold-start period, i.e., it takes some time to notice that a quest is related to certain long-standing interests. So, a method for multiple short-standing interests would provide a more efficient support during a researcher’s quest for information. We note that the long-standing interests are still important since they have a purpose in the personalising link, viz. a relation with the context of a document. The functioning of the prototype The prototype asks, due to its implementation, that each word is stored in the database, and that a large amount of words are considered during the selection procedure. Therefore, relatively high requirements are needed regarding the processor speed and data storage for a prototype. Furthermore, we noticed that the sections of a document that is relevant as a whole, appear frequently after each other in the resultant set. If in this case the whole document is presented instead of many sections in a disjunct form, there is more room to set the spotlight on other possibly relevant sections. This may lead to an improvement in efficiency for the researcher. The memory The objective of the memory was to provide the researcher with a reason to trust the personalised results (see the two requirements in 3.3.2). However, we had to tell the researchers that they were able to check the memory. One researcher explained she would like to have the memory available after a short action, such as a button click. We believe that the memory is not necessary for personalisation after such short quests. More importantly, such a memory should fulfil the task of being a reliable source, but for this task it may become more complex. For instance, when it handles also the long-standing interests, and when more iterations are executed. Then, if the resultant set does not match with the expectations of a researcher, it may provide insight information on the cause assume a significant word has a high negative weight. For that matter, the solution is to add the word to the own description. The enrichment The prototype uses a straightforward procedure to suggest an enrichment, i.e., on term frequency. Applying a more specific algorithm, for instance, latent semantic indexing, which tries to associate terms with concepts, may improve the quality of the suggested enrichment.<br />
<br />
50<br />
<br />
CHAPTER 5. TESTING OUR MODEL<br />
<br />
Sections as items We expect that further investigation regarding sections that have particular features, may provide information that can be employed for an improvement in the selection procedure, and therefore also in the resultant set. For instance, instead of only providing the section containing the list of references, sections of documents in that reference list are absorbed as well in the resultant set, even when the match with the selection criteria is low. Another possible point for improvement regarding the contents of sections, might be to substitute the link to a reference with the reference itself. This is logical since the information is actually part of that section. The expected benefits are that then authors and publication attributes are observable for a researcher, and these words may find their way more easily to a proposed enrichment. Conclusion A potential proof for how well personalisation performs its task requires a large set of experiments because the results are subjective. However, regarding (1) that the prototype provided useful enrichments, (2) that the memory represented significant words correctly, and (3) that both researchers responded that the prototype indeed personalised their quest, we may conclude that the prototype succeeded the test of a proof of concept for our model. But, a method to keep track of multiple short-standing interests is necessary to support more than one quest.<br />
<br />
Chapter 6<br />
<br />
Conclusions and further research<br />
<br />
This last chapter returns to the problem statement. The accompanying research questions are discussed in section 6.1, and answers are formulated. Section 6.2 provides recommendations for further research.<br />
<br />
6.1<br />
<br />
Conclusions<br />
<br />
The research question, introduced in section 1.3, stated the following. How can personalisation support the quest for information, as formulated by researchers? For this statement, we have formulated three corresponding research questions. They are answered below, followed by a conclusion. Research question 1: What is involved in the personalisation of information? There are two main components involved, i.e., the users and the items. The goal of personalisation is to equip a current researcher with the items he needs. To achieve this, we discussed four personalisation approaches. A content-based approach uses information from the content of the items to personalise. A collaborative approach regards the users as a source for personalisation. An extension-based approach relies on information from an extended source to provide the user with his personal set of items. The fourth approach, a hybrid approach, explored hybridisations of the other approaches. Furthermore, the domains of data mining and intelligent agents can be utilised for personalisation. Research question 2: Where do researchers desire personalised support in their search for information? From the interviews we identified four search goals on why a researcher may quest for information: (1) to obtain background knowledge, (2) to remain up to date with the field of research, (3) to become acquainted with related areas, and (4) to find specific information. Next, we explored three categories for support by personalisation: (1) the literature study, (2) knowledge sharing between colleagues, and (3) knowledge sharing between other researchers. The 51<br />
<br />
52<br />
<br />
CHAPTER 6. CONCLUSIONS AND FURTHER RESEARCH<br />
<br />
interviewees showed an explicit preference for personalised support on the literature study. That answers the second research question. Research question 3: How can a personalised quest for information be achieved? The answer is presented by our model, which is characterised by (1) the items are sections that belong to a document, and (2) a user model represents the researcher’s long-standing interests and short-standing interests. To personalise the items for a researcher, a contentbased approach is applied. The personalising link between a researcher and his relevant items is as follows. The context of a document is related to his long-standing interest, and a relevant item (a section that belongs to the document) is related to his short-standing interest. To test a proof of concept for our model, a prototype is implemented with (1) a mechanism for iterative searching on a description, (2) a user model that (2a) maintains the shortstanding interests and (2b) provides a personalised resultant set. Experiments with two junior researchers as evaluators show that the assumptions of the relation between the short-standing interests and a section were reliable. Furthermore, the opinions of two researchers were that the prototype personalised their searches. So, we may draw the final conclusion that personalisation can support the quest for information (as formulated by researchers) by (1) exploiting sufficient information about a researcher’s project-related interests to suggest his personal set of information items, (2) providing an iterative search, and (3) enriching the description of the researcher’s current interest.<br />
<br />
6.2<br />
<br />
Further research<br />
<br />
The current research is only the start for further research in personalisation and information filtering. These topics are very challenging and can be broadened along a variety of dimensions. We restrict ourselves to five points only. And even these five points are not elaborated upon below. The idea is to show the reader that the future is bright for researchers on this topic and for researchers using future tools that are a result of the research to be performed. • The long-standing interests. • Investigation to a better understanding of the four search goals. • How to find information items. • Investigation into a cooperation with the two other categories (knowledge sharing with colleagues, and knowledge sharing with other researchers). • Application by intelligent agents.<br />
<br />
References<br />
<br />
T. J. Allen. Managing the Flow of Technology. MIT Press, 1984. ISBN 0-262-51027-8. Technology Transfer and the Dissemination of Technological Information Within the Research and Development Organization. R. A. Baeza-Yates and B. Ribeiro-Neto. Modern Information Retrieval. Addison-Wesley Longman Publishing Co., Inc., 1999. N. J. Belkin and W. B. Croft. Information filtering and information retrieval: Two sides of the same coin? Communications of the ACM, 35(12):29–38, December 1992. D. Billsus and M. J. Pazzani. A personal news agent that talks, learns and explains. proceedings of autonomous agents. In Proceedings of the third annual conference on Autonomous Agents, pages 268–275. ACM Press, 1999. Z. Boger, T. Kuflik, P. Shoval, and B. Shapira. Automatic keyword identification by artificial neural networks compared to manual identification by users of filtering systems. Information Processing and Management, 37(2):187–198, 2001. R. Burke. Hybrid recommender systems: Survey and experiments. User Modeling and User-Adapted Interaction, 12(4):331–370, 2002. A. J. Cawsey, R. B. Jones, and J. Pearson. The evaluation of a personalised information system for patients with cancer. User Modeling and User-Adapted Interaction, 10(1):47–72, 2000. M. Claypool, A. Gokhale, T. Miranda, P. Murnikov, D. Netes, and M. Sartin. Combining contentbased and collaborative filters in an online newspaper. In Proceedings of ACM SIGIR Workshop on Recommender Systems, 1999. L. F. Cranor, J. Reagle, and M. S. Ackerman. Beyond concern: Understanding net users’ attitudes about online privacy. enquete study, AT&T Labs-Research, 1999. S. Deorowicz and A. Sk´ orczy´ nski. LEd documentation. 2004. M. Eirinaki and M. Vazirgiannis. Web mining for web personalization. ACM Transactions on Internet Technology, 3(1):1–27, 2003. W. Fan, M. D. Gordon, and P. Pathak. Effective profiling of consumer information retrieval needs: a unified framework and empirical comparison. Decision Support Systems, 40(2):213–233, August 2005. P. W. Foltz and S. T. Dumais. Personalized information delivery: an analysis of information filtering methods. Communications of the ACM, 35(12):51–60, 1992.<br />
<br />
53<br />
<br />
54<br />
<br />
REFERENCES<br />
<br />
G. Gonzlez. Towards smart user models in open environments. PhD thesis, Departement of electronics, computer science and utomatic control of the university of Girona, July 2003. N. Good, J. B. Schafer, J. A. Konstan, A. Borchers, B. Sarwar, J. Herlocker, and J. Riedl. Combining collaborative filtering with personal agents for better recommendations. In Proceedings of the sixteenth national conference on Artificial intelligence, and the eleventh Innovative applications of artificial intelligence conference innovative applications of artificial intelligence, pages 439–446, 1999. D. J. Hand, P. Smyth, and H. Mannila. Principles of data mining. MIT Press, 2001. J. L. Herlocker, J. A. Konstan, and J. Riedl. Explaining collaborative filtering recommendations. In Proceedings of the 2000 ACM conference on Computer supported cooperative work, pages 241–250, Philadelphia, Pennsylvania, United States, 2000. ACM Press. J. L. Herlocker, J. A. Konstan, L. G. Terveen, and J. T. Riedl. Evaluating collaborative filtering recommender systems. ACM Transactions on Information Systems, 22(1):5–53, 2004. G. Hodge. Metadata made simpler, 2001. B. J. Hurley, J. Price-Wilkin, M. Proffitt, and H. Besser. The making of america ii testbed project: a digital library service model. Technical Report CLIR report 87, Council on Library and Information Resources, 1999. D. W. King and C. Hanson. After migration to an electronic journal collection, the impact on faculty and doctoral students. D-lib magazine, 8(12), December 2002. D-Lib Magazine is a solely electronic publication http://www.dlib.org. A. Kobsa. Supporting user interfaces for all through user modeling. In Proceedings HCI International 1995, pages 155–157, Yokohama, Japan, 1995. A. Kobsa. Tailoring privacy to users’ needs. In Proceedings of the 8th International Conference on User Modeling 2001, pages 303–313, Londen, UK, 2001a. Springer-Verlag. A. Kobsa. Generic user modeling systems. User Modeling and User-Adapted Interaction, 11(1-2):49–63, 2001b. I. Koychev and I. Schwab. Adaptation to drifting user’s interests. In Proceedings of European Conference on Machine Learning 2000, 2000. J. Kramer, S. Noronha, and J. Vergo. A user-centered design approach to personalization. Communications of the ACM, 73(8):44–48, 2000. H. Krottmaier. The need for sharing user-profiles in digital libraries. In Building Digital Bridges: Linking Cultures, Commerce and Science: Proceedings of the 8th ICCC/IFIP International Conference on Electronic Publishing, volume 23, pages 323–330, June 2004. T. Kuflik, B. Shapira, and P. Shoval. Stereotype-based versus personal-based filtering rules in information filtering systems. Journal of the American Society for Information Science and Technology, 54 (3):243–250, 2003. W. Lam, S. Mukhopadhyay, J. Mostafa, and M. Palakal. Detection of shifts in user interests for personalized information filtering. In Proceedings of the 19th annual international ACM SIGIR conference on Research and development in information retrieval, pages 317–325, 1996. S. Lawrence, K. Bollacker, and C. Lee Giles. Indexing and retrieval of scientific literature. In Proceedings of the eighth international conference on Information and knowledge management, pages 139–146, 1999.<br />
<br />
55<br />
<br />
C.-H. Lee, Y.-H. Kim, and P.-K. Rhee. Web personalization expert with combining collaborative filtering and association rule mining technique. Expert Systems with Applications, 21(3):131–137, October 2001. C. Lee Giles, K. D. Bollacker, and S. Lawrence. Citeseer: an automatic citation indexing system. In Proceedings of the third ACM conference on Digital libraries, pages 88–98, 1998. H. Lieberman, H. Liu, P. Singh, and B. Barry. Beating common sense into interactive applications. AI Magazine, 25(4):63–76, 2005. P. Lyman and H. R. Varian. How much information 2003? Study, 2003. S. M. McNee, I. Albert, D. Cosley, P. Gopalkrishnan, S. K. Lam, A. M. Rashid, J. A. Konstan, and J. Riedl. On the recommending of citations for research papers. In CSCW ’02: Proceedings of the 2002 ACM conference on Computer supported cooperative work, pages 116 – 125, New York, NY, USA, 2002. ACM Press. S. E. Middleton, N. R. Shadbolt, and D. C. De Roure. Ontological user profiling in recommender systems. ACM Transactions on Information and System Security (TISSEC), 22(1):54–88, 2004. M. Montaner, B. Lpez, and J. L. De La Rosa. A taxonomy of recommender agents on the internet. Artificial Intelligence Review, 19(4):285–330, 2003. R. J. Mooney and L. Roy. Content-based book recommending using learning for text categorization. In Proceedings of the fifth ACM conference on Digital Libraries, pages 195–204, New York, USA, 2000. ACM Press. M. D. Mulvenna, S. S. Anand, and A. G. Bchner. Personalization on the net using web mining: introduction. Communications of the ACM, 43(8):122–125, August 2000. L. Page and S. Brin. Pagerank, an eigenvector based ranking approach for hypertext. In 2lst Annual ACM/SIGIR International Conference on Research and Development in Information Retrieval, 1998. G. Paliouras, C. Papatheodorou, V. Karkaletsis, and C. D. Spyropoulos. Discovering user communities on the internet using unsupervised machine learning techniques. Interacting with Computers, 14(6): 761–791, 2002. M. Pazzani. Beyond idiot savants: Recommendations and common sense. Workshop: Beyond Personalization 2005, January 2005. M. J. Pazzani. A framework for collaborative, content-based and demographic filtering. Artificial Intelligence Review, 13(5-6):393–408, December 1999. M. J. Pazzani, J. Muramatsu, and D. Billsus. Syskill and webert: Identifying interesting web sites. In Proceedings of the Thirteenth National Conference on Artificial Intelligence, pages 54–61, 1996. D. M. Pennock, E. Horvitz, and C. Lee Giles. Social choice theory and recommender systems: Analysis of the axiomatic foundations of collaborative filtering. In Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence, 2000. D. Pierrakos, G. Paliouras, C. Papatheodorou, and C. D. Spyropoulos. Web usage mining as a tool for personalization: A survey. User Modeling and User-Adapted Interaction, 13(4):311–372, 2003. M. Polanyi. Personal Knowledge: Towards a Post Critical Epistemology. University of Chicago Press, 1958. M. F. Porter. An algorithm for suffix stripping. Program, 14(3):130 – 137, 1980.<br />
<br />
56<br />
<br />
REFERENCES<br />
<br />
M. E. Renda and U. Straccia. A personalized collaborative digital library environment: a model and an application. Information Processing and Management, 41(1):5–21, 2005. A. Rosenbaum. New survey shows consumers are more likely to purchase at web sites that offer personalization. consumers willing to provide personal information in exchange for improved service and benefits. Survey, May 2001. B. Sarwar, G. Karypis, J. Konstan, and J. Reidl. Item-based collaborative filtering recommendation algorithms. In Proceedings of the tenth international conference on World Wide Web, pages 285–598, 2001. J. B. Schafer. Encyclopedia of data warehousing and mining, chapter The Application of Data Mining to Recommender Systems. Idea group Reference, Hershey, PA, 2005. G. Schreiber, H. Akkermans, A. Anjewierden, R. de Hoog, N. Shadbolt, W. Van de Velde, and B. Wielinga. Knowledge Engineering and Management. The MIT Press, 1999. I. Schwab, A. Kobsa, and I. Koychev. Learning about users from observation. In Proceedings AAAI 2000 Spring Symposium: Adaptive User Interface, pages 241–247, 2000. U. Shardanand and P. Maes. Social information filtering: Algorithms for automating “word of mouth”. In Proceedings of ACM CHI’95 Conference on Human Factors in Computing Systems, volume 1, pages 210–217, 1995. S. Shearin and P. Maes. Representation and ownership of electronic profiles. In CHI 2000 Workshop Proceedings: Designing Interactive Systems for 1-to-1 E-commerce., 2000. http://www.zurich.ibm.com/ mrs/chi2000/submissions/shearin.doc. B. Smyth and P. Cotter. A personalized television listings service. Communications of ACM, 43(8): 107–111, 2000. B. Smyth, P. Cunningham, R. Bergmann, R. Traphner, S. Schmitt, and S. Breen. Websell: Intelligent sales assistants for the world wide web. In E-business: Issues, Applications and Developments. Proceedings of eBusiness and eWork 2000 Conference and Exhibition, 2000. S. Soltysiak and B. Crabtree. Knowing me, knowing you: Practical issues in the personalization of agent technology. In Proceedings of the third international conference on the practical applications of agents and multi-agent systems. PAAM98, 1998. D. Stenmark. Information vs. knowledge: The role of intranets in knowledge management. In In Proceedings of HICSS-35. IEEE Press, 2002. H. Tsoukas. The firm as a distributed knowledge system: A constructionist approach. Strategic Management Journal, 17:11–25, December 1996. M. B. Twidale and D. M. Nichols. A survey of applications of cscw for digital libraries. technical report CSEG/4/98, Graduate School of Library and Information Science, University of Illinois at Urbana-Champaign and Computing Department, Lancaster University, 1998. C. J. Van Rijsbergen. Information Retrieval, second edition. Butterworth-Heinemann, 1979. M. van Setten. Personalised information systems. Published online by Telematica Instituut, June 2001. https://extranet.telin.nl/docuserver/dscgi/ds.py/ViewProps/File-16467. F. Wang and H. Shao. Effective personalized recommendation based on time-framed navigation clustering and association mining. Expert Systems with Applications, 27(3):365377, 2004.<br />
<br />
57<br />
<br />
G. Webb, M. J. Pazzani, and D. Billsus. Machine learning for user modeling. User Modeling and User-adapted Interaction, pages 19–29, 2001. G. Weiss. Multiagent systems: a modern approach to distributed artificial intelligence. MIT Press, 1999. ISBN 0-262-23203-0.<br />
<br />
Appendices<br />
<br />
59<br />
<br />
Appendix A<br />
<br />
The interviews<br />
<br />
A.1<br />
<br />
The interviewees<br />
<br />
The researchers who participated in the interviews are listed below in alphabetical order. • Ir. N.H. Bergboer • Drs. L. Braun • Drs. F. de Jonge • Dr. I.G. Sprinkhuizen-Kuyper • Dr. ir. P.H.M. Spronk • Dr. ir. J.W.H.M. Uiterwijk<br />
<br />
A.2<br />
<br />
The interview questions<br />
<br />
Below, the proposed questions of the interviews are shown in Dutch. DOEL 0: Introduction Opnemen op video? Mijn onderzoek beschrijven. Doel van het interview. DOEL 1: De weg van idee tot het einde van het project blootleggen, vooral op welke momenten een literatuurstudie voorkomt, welk doel deze heeft en welke informatie gezocht wordt. Vraag 1a: Je hebt een idee voor een project wat doe je dan, wat is dan je “plan van aanpak” om het project te realiseren? (Je zoekt mensen die ook zouden willen deelnemen, begint met het schrijven van een voorstel,...) 61<br />
<br />
62<br />
<br />
APPENDIX A. THE INTERVIEWS<br />
<br />
Vraag 1b: Je wordt gevraagd of besluit mee te doen aan een project, wat gebeurt er vervolgens? Wat is dan je “plan van aanpak” om het project te realiseren? Vraag 2: Wanneer wordt er naar informatie gezocht? (bijvoorbeeld voor de opstelling van het projectvoorstel, de haalbaarheid van het project, een overzichtje van de kennis (en mensen?) die nodig zijn voor het project, de literatuurstudie) Literatuurstudie naar aanleiding van?<br />
<br />
Doel van de literatuurstudie<br />
<br />
Zoeken naar algemene informatie<br />
<br />
Uitgebreid Zoeken naar informatie<br />
<br />
Zoeken naar Specifieke kennis<br />
<br />
Project voorstel Haalbaarheid People resources Andere informatie Vraag 3a: Indien gezocht wordt naar literatuur voor een projectvoorstel. • Naar aanleiding van wat en wat is het doel hiervan? • In welke mate wordt er naar literatuur gezocht? • Hoe accuraat blijft deze informatie? Vraag 3b: Tijdens de literatuur studie? • Naar aanleiding van wat en wat is het doel hiervan? • In welke mate wordt er naar literatuur gezocht? • Waarvoor wordt de information hiervan gebruikt? DOEL 2: Erachter komen hoe wordt gezocht naar literatuur en wat er gebeurt met de gevonden literatuur. Vraag 4: Weet je bij de project aanvang wat je wil weten (dus op welke gebieden je literatuur wil vinden)? (ja/nee) Indien ”nee”: Wat doe je dan, hoe pak je het aan? Vraag 5: Waar ga je het eerste zoeken (online, bibliotheek,...)? En hoe vervolg je je zoektocht? (wat is het zoekpad?) Vraag 6: Wat doe je als je niet vind waar je naar op zoek bent? Bijvoorbeeld: zoeken op synoniem, vragen aan anderen (projectleden? Derden?), andere bronnen raadplegen (welke?) Vraag 7: Welke zijn de criteria waar je op let? (zie onderstaande tabel) Hoe bepalend zijn deze criteria (bijvoorbeeld, als het voor 2000 gepubliceerd is lees ik het niet)? Veranderen de belangrijkheid van deze criteria of de criteria zelf?<br />
<br />
63<br />
<br />
A.2. THE INTERVIEW QUESTIONS<br />
<br />
Inhoud<br />
<br />
Jaar<br />
<br />
Soort<br />
<br />
Tijdschrift<br />
<br />
Auteur<br />
<br />
Andere<br />
<br />
Belangrijk Minder belangrijk Onbelangrijk Veranderd in 1 project Veranderd in +1 project<br />
<br />
Vraag 8: Je hebt een mogelijk relevant artikel gevonden, wat doe je hiermee? • Abstract lezen • Artikel ’scannen’ • Conclusies lezen • Volledig lezen • Opslaan • Uitprinten • Samenvatting maken • Anders, nl... Vraag 9a: Het blijkt daadwerkelijk een relevant artikel te zijn. Wat doe je daar dan mee? • Opslaan (Ook een strategie? Bijvoorbeeld opslaan in map ”relevante artikels”) • Bijhouden in een document management tool? (welke?) • Verwerken in eigen artikel • Anders, nl... Vraag 9b: Zoek je zo een document erna nog terug en vind je het gemakkelijk terug? Indien het niet altijd even gemakkelijk wordt teruggevonden: • Wat is de oorzaak hiervan? • Wat doe je (zou kunnen doen) om dit te vermijden? • Wat zou een tool moeten bevatten die dit wil ondersteunen? • Zoeken op attributen of tekst (zoals in windows longhorn, bijvoorbeeld, kunnen zoeken op artikels van 2002, van auteur x, waar het woord xyz in voorkomt,...) • De mogelijkheid tot zelf een abstract of keywords erbij zetten • Bijhouden of het al gelezen/verwerkt/... is • Anders, nl...<br />
<br />
64<br />
<br />
APPENDIX A. THE INTERVIEWS<br />
<br />
DOEL 3: Welke elementen moeten er in een tool zitten die het literatuuronderzoek ondersteund? Vraag 10a: Welke zijn de meest voorkomende irritaties en problemen tijdens de literatuurstudie? Bijvoorbeeld, het teveel aan informatie, een artikel niet meer kunnen terug vinden. Vraag 10b: Heb je behoefte aan een systeem dat literatuuronderzoek ondersteund? Vraag 10c: (Indien 10b ja) Bij welke onderdelen zou je ondersteuning willen van een systeem? (ook het zoeken naar nieuwe artikelen?) Vraag 10d: (Indien 10b ja) Welke is je top 5 van elementen die dat ondersteund systeem zeker moet bevatten? Vraag 11: [Uitleg over kennisdeling] Kennisdeling tussen projectleden, hoe gebeurt dat? Vraag 11a: Op welk gebied zou een systeem dat moeten ondersteunen? Bijvoorbeeld: feedback op een document, aanraden van een artikel aan een projectlid, aanrader krijgen over een artikel van een projectlid, een nieuw projectlid gebruik laten van de informatie voorhanden in het project, annotaties gebruiken. Vraag 11b: Moet het systeem ook kennisdeling met projectleden ondersteunt? Vraag 12: Kennisdeling met derden, hoe gebeurt dat? Vraag 12a: Op welk gebied zou het dat moeten ondersteunen (bijv. Feedback op een document, de mogelijkheid dat een gebruiker in contact kan komen met experts op gebieden die relevant zijn voor het project) Vraag 12b: Moet het systeem kennisdeling met derden moet ondersteunen? (Indien ”nee”:waarom niet?) Vraag 13: Zou jij gecontacteerd willen worden door derden als expert (naar aanleiding van vraag 12a)? Vraag 14: Welke is de volgorde van belangrijkheid dat de volgende onderdelen in het systeem voorkomen? • Ondersteuning voor de literatuurstudie • Kennisdeling met projectleden ondersteuning • Kennisdeling met derden ondersteuning DOEL 4: Vertrouwen in het systeem Vraag 15: Waaruit zou je wantrouwen in personalisation bestaan? (bijv. Onvoldoende of verkeerde informatie, te zeer in 1 richting geduwd worden, ...) Vraag 16: Wat zou je dan doen? Bijvoorbeeld: systeem niet meer gebruiken, zelf zoeken, het systeem testen,... Vraag 17: Wat zou het systeem kunnen doen om jou vertrouwen te winnen? • Zijn zoekweg zichtbaar maken • Alle gevonden informatie laten zien, maar in volgorde van interesse van de gebruiker • Het % zekerheid dat een artikel jou zou interesseren weergeven<br />
<br />
A.3. THE INTERVIEW ANSWERS<br />
<br />
65<br />
<br />
• Toegang verschaffen tot het user profile • Mogelijkheid tot aanpassingen maken aan het user profile • Anders, nl... DOEL 5: Verdraagbaarheid en oplossing voor het cold-start problem [Uitleggen wat het coldstart probleem is] Vraag 18: Ben je bereidt zelf aan te geven over welk onderwerp of over welke gebieden het project is georienteerd? • Door middel van keywords • Een beschrijving geven • Het projectvoorstel ingeven • Aanduiden dat een gevonden artikel relevant is • Anders, nl... Vraag 19: Hoe ver wil je maximum gaan om het systeem duidelijk te maken waarnaar jij op zoek bent voor een bepaald project in mate van tijd? (5 minuten, een half uurtje). En hoeveel moeite wil je hierin steken? • Een button click of copy/past gaat nog net • Ik wil er best enige moeite insteken • Anders, nl... Vraag 20: Zo, dat was het interview. Nog iets toe te voegen? Op- en of aanmerkingen?<br />
<br />
A.3<br />
<br />
The interview answers<br />
<br />
The the interview answers of one junior researcher and one senior researcher are transcribed below in Dutch.<br />
<br />
A.3.1<br />
<br />
Answers of a junior researcher<br />
<br />
Vraag: stel dat je een idee hebt voor een onderzoek, hoe is dan je eerste aanpak? Antwoord: meestal ga ik eerst algemene literatuur zoeken over het onderwerp om een beeld te krijgen wat er allemaal is gedaan, zo krijg je een idee over wat er allemaal reeds is gebeurd op dat gebied. Over wat je dan vind krijg je een idee in welke richting je je eerste onderzoek kan doen, daarvoor wil je zeker weten dat er nog niks gedaan is op dat gebied. Dan ga je op dat gebied steeds naar specifieker literatuur zoeken. Ga je dan ook zoeken naar mensen die zullen deelnemen aan het project of wil je die nog interviewen. Ja zeker, bij mijn projecten ben ik bezig met artsen, en dan ga ik op Internet zoeken om informatie te vinden over deze artsen of ik ga ze interviewen.<br />
<br />
66<br />
<br />
APPENDIX A. THE INTERVIEWS<br />
<br />
Hoe accuraat blijft die informatie, ga je die verwerken in je proefschrift? Ja, die algemene informatie is vaak wel bruikbaar bij een inleiding. Maar als je kijkt naar die accuraatheid, er is op dat gebied wel heel vaak literatuur te vinden, niet altijd meteen, maar na een tijdje merk je wel dat er heel wat literatuur is. Het is inderdaad moeilijk om met zekerheid te zeggen, ik weet zeker dat er op dit gebied nog niet veel onderzoek is gedaan, er is nog niet veel informatie, want je kunt inderdaad onmogelijk alles doorlezen. Je weet dat je literatuur gaat zoeken, maar weet je ook precies wat je gaat zoeken? Je hebt in het begin wel een globaal idee, maar maarmate je meer gaat zoeken raak je steeds meer vertrouwd met het onderwerp en kun je steeds beter bedenken wat je nog wil gaan vinden. Dus je hebt een paar steekwoorden over wat je wil vinden? Ja precies, en als je merkt dat werkt goed en hier vind je veel dan ga je steeds in die richting verder zoeken. Waar ga je eerste zoeken, online, bibliotheek, tijdschriften? Online is het makkelijkst, maar hier vind je vaak niet alles, alleen de titel of de schrijver maar niet de literatuur, en dan moet je toch naar de bibliotheek. Welke sites bezoek je dan? De sites ‘Scirus’, een soort Google maar dan voor wetenschappelijke doeleinden, ‘Science direct’, en verder, omdat mijn onderwerp medisch getint is, zoek ik naar medische zoekpagina’s. ‘CiteSeer’ gebruik ik ook vaak, hier vind je wel degelijke informatie. Als je niet vind wat je zoekt, welke bronnen gebruik je dan? Bijvoobeeld, gebruik je synoniemen of tips van anderen? Meestal gebruik ik synoniemen, tips van anderen vragen doe ik ook wel vaak, en soms doe ik een interview. Op de site van de bibliotheek vind je ook wetenschappelijke literatuur. In online wetenschappelijke tijdschriften zoek ik ook naar het specifieke onderwerp. En, even wachten en later nogmaals te zoeken, want misschien dat er nieuwe artikelen zijn bijgekomen. Welke zijn de criteria die je handhaaft bij je zoektocht (zoek je op inhoud, jaartal, of soort)? Na 1975: alleen als het een basis artikel is, later dan 1990 in het algemeen, maar liefst na 2000 natuurlijk. Vervolgens, liever artikelen uit een goede tijdschriften. Het jaartal van publicatie, hoe belangrijk is dat, heb je een grens? Bij mij is de grens rond 1995, liefst heb ik na 2000. Maar het is ook belangrijk is welk soort artikel is het; is het een standaard basis artikel dan is het jaartal niet zo belangrijk. Stel, je hebt een goed artikel gevonden hoe verwerk je dat verder? Bijvoorbeeld, abstract lezen, volledig lezen, conclusies trekken, samenvatting maken? Ik sla het altijd eerst op (soms ook bookmarken voor de zekerheid). Dan ga ik het abstract lezen, gevolgd door de conclusies. Is het dan nog steeds interessant dan ga ik het scannen of uitprinten. Daarna lees ik het hele artikel. Maar bij veel artikelen merk je al bij de abstract of het relevant is. Als je nu een interessant artikel hebt gevonden hoe verwerk je dat? Hoe sla je dat op?<br />
<br />
A.3. THE INTERVIEW ANSWERS<br />
<br />
67<br />
<br />
Ik heb een mapje waar alle artikelen bijeen zitten, onderverdeeld in submappen volgens onderwerp, omdat ik verschillende onderwerpen onderzoek. Die onderwerpen kunnen overlappen. En verder maak ik een referentie in een bibtex file, als het een belangrijk artikel is voor mijn proefschrift. Als je een interessant artikel hebt gevonden, is dat nadien gemakkelijk terug te vinden? Meestal wel, maar als je het niet hebt opgeslagen kan het zijn dat je het niet meer terug vind. Welke onderdelen zou een tool dat je literatuur bijhoudt moeten ondersteunen? Bijvoorbeeld, zoeken op attributen of op tekst, of een overzicht waarbij je bepaalde keywords zou moeten ingeven. Een systeem dat alles bijhoudt van wat en welke je al gelezen hebt, dat zou wel handig zijn. Een tool waarin je heel goed de kenmerken kan specificeren (auteur, keywords, titel), dan zou je het artikel moeten kunnen terug vinden als je voldoende kenmerken erbij kunt zetten hebt. En idealiter is dit geautomatiseerd, het zou veel tijd en energie besparen. Welke elementen zouden er in een tool moeten zitten die een literatuur studie moeten ondersteunen? Welke elementen in een literatuur studie vind je dat verbeterd kunnen worden? Het is vaak moeilijk om een beperkte set van resultaten te krijgen, meestal krijg je een duizendtal artikelen en dit is irritant, je zou onderscheid moeten kunnen maken in wat nu echt een goede bron: je krijgt veel websites, maar eigenlijk wil je referenties naar goede artikelen in je proefschrift, en geen websites. Dus je krijgt heel veel terug wat niet bruikbaar is. En verder dat je makkelijk op pagina’s kan komen waar ook de hele inhoud van een artikel staat, niet alleen de titel. Zou je behoefte hebben aan een systeem wat literatuurstudie ondersteund? Zou wel heel makkelijk zijn, en minder tijdrovend. Nu moet je om de zoveel tijd even kijken of er nog iets nieuws is. Het is veel fijner als jij een systeem hebt dat een beetje de boel in de gaten houdt en elke keer als er iets interessants bijkomt (dat je nog niet gezien had) dat ook aan jou presenteert. Wat zou je top vijf van elementen zijn die dit systeem zeker zou moeten bevatten? Een goed onderscheid maken in de bron (dit zijn artikelen, dit zijn websites, dit boeken). Dat het relevante informatie is die je krijgt en niet teveel natuurlijk want het moet ook haalbaar blijven. Verder, het systeem filtert alvast het abstract eruit zodat ik meteen kan zien waar dit over gaat. Dat het continue blijft scannen, op onderwerpen waar ik mee bezig ben. En dat je kan zien wat je al gelezen hebt, want je krijgt zoveel informatie. Ik zou het even graag willen hebben over kennisdeling, bijvoorbeeld het aanraden van een artikel aan een project, of aangeraden kregen door een project? Dat is heel nuttig en kan een heleboel zoeken voorkomen. Zeker als je dit met een grote groep mensen kan doen, kan dit heel goed weken. Nieuwe deelnemers informeren over de globale informatie in het project? Nuttig, ze kunnen dan meteen over die informatie beschikken en mee in het project stappen,<br />
<br />
68<br />
<br />
APPENDIX A. THE INTERVIEWS<br />
<br />
zodanig gaat er geen tijd verloren met een persoon in te werken, want hij kan het zichzelf eigen maken. Nog suggesties ? Als je met meerdere aan een project werkt en 1 iemand heeft al gezegd ’nou dit artikel is niks’ dan is het handig dat de andere ziet dat hij heeft gezegd dat dit niet interessant is, nou dan is het voor mij waarschijnlijk ook niet interessant. Kennisdelen met derden, feedback op een publiek beschikbaar gesteld artikel? Nee, alleen als ik kan zeggen wie het mag zien. Maar dan ook nog niet perse, ik kan het even goed mailen. Dit wegens het risico dat andere mensen met jouw ideen ervan door gaan. Maar als je het systeem zo ver kan krijgen dat een bepaalde persoon het document ziet en de rest van de wereld niet, dan zou ik dat wel doen ja. De mogelijkheid dat een gebruiker in contact kan komen met experts op gebieden die relevant zijn voor het project? Ja, dat zou ik heel fijn vinden. Als het systeem mij relevante mensen op een presenteerblaadje zou kunnen geven, zodat ik voor mijn project bijvoorbeeld onmiddellijk een afspraak zou kunnen maken met een arts, dat zou ik fijn vinden. Maar via een beschermd systeem zou de arts wel zelf moeten kunnen bepalen of hij aan onderzoekers interviews wil geven. Verder, het zou handig zijn als er een lijst of iets dergelijks zou zijn waarin vermeld staat wie juist met wat bezig is. Vind je dat een systeem die literatuurstudie ondersteund ook kennisdelen met anderen moet ondersteunen? Ja ik denk het wel. Niet met iedereen, wel met vooraf geselecteerde personen. Welke is de volgorde van jouw voorkeur om ondersteuning te bieden voor de drie onderdelen die hier aan bod zijn gekomen? 1 Ondersteuning voor literatuur studie. 2 Ondersteuning voor kennisdeling met projectleden. 3 Ondersteuning voor kennisdeling met derden. Waaruit zou jou wantrouwen in het principe van personalisation bestaan? Als zo’n systeem een profiel van jou gaat aanmaken heb je natuurlijk altijd wel in je achterhoofd ‘heeft hij wel het goede profiel?’, en krijg ik wel alles wat ik zou willen hebben. Of wel gerriteerd dat hij het nog steeds niet door heeft dat ik dat (niet) wil. Het is niet zo erg als je teveel krijgt dan dat je belangrijke dingen misloopt. Maar van de andere kant is het irritant dat je zodanig veel krijgt dat je het toch niet verwerkt krijgt. Wat zou je dan doen? Het systeem testen. En ik krijg met het systeem te weinig zou ik het misschien toch blijven gebruiken, maar daarnaast ook nog zelf zoeken. Als het systeem vanzelf werkt en het mij weinig tijd kost om de resultaten dan wil ik er best zelf naast zoeken. Wat zou het systeem dan kunnen doen om je vertrouwen te winnen? Zijn zoekweg zichtbaar maken? Ja, dat zou ik wel goed vinden. Als je naast de artikelen die je nodig hebt ook artikelen krijgt<br />
<br />
A.3. THE INTERVIEW ANSWERS<br />
<br />
69<br />
<br />
die er op lijken, zou ik op zich heel prettig vinden omdat je hier ook vaak relevante informatie vindt. Stel dat het systeem jou alle gevonden informatie in volgorde een bepaalde volgorde teruggeeft? Ja, maar dan wil ik ook kunnen bepalen welke de volgorde, bijvoorbeeld sorteren op relevantie of wanneer het artikel is geschreven wat voor soort artikel dat het is, een uitgave een artikel, of een website. De waarschijnlijkheid dat een artikel jou zou interesseren weergeven? Ja, als k merk dat het klopt dan zou ik daar heel veel vertrouwen van krijgen. Als k te vaak merk dat het niet klopt, dan is dat toch heel vervelend en dan haak je op een gegeven moment af. Soms ben je ook met een heel specifiek stukje van je userprofile bezig en dat moet je ook kunnen aangeven. Mogelijkheid tot het aanpassen van je eigen profiel? Als de gegevens er toe leiden om het systeem beter te maken en het op een gemakkelijke manier realiseerbaar is, zou ik het wel doen. Je hebt er nu meteen of later profijt van dus ben ik er zeker voorstander van. In welke mate ben jij bereid aan te geven in welke gebieden je project geori¨ enteerd is? Bijvoorbeeld, door middel van keywords, een samenvatting, of het projectvoorstel. Ja, als het een lang project is (bijvoorbeeld 4 jaar), dan zou ik best wel tijd willen steken in het aanmaken van een userprofile, als ik weet dat het gedurende de rest van het project een groot voordeel kan zijn. Maar als ik een keer bezig met het zoeken van artikelen dan wil ik ook door met die artikelen en wil ik niet meer bezig zijn met allerlei dingen aangeven. Een button click kan dan nog net.<br />
<br />
A.3.2<br />
<br />
Answers of a senior researcher<br />
<br />
Vraag: Als je een idee hebt voor een project of een onderzoek, wat doe je daar dan mee? Antwoord: nou ik ga niet kijken of het haalbaar is. Over het algemeen krijg ik ideen op gebied waar ik genoeg kennis van heb. Genoeg kennis dus om te weten van dit is een nieuw idee. Wel nog heel eventjes rondkijken of nog niet iemand anders daar eerder mee is gekomen. Maar meestal krijg ik ideen tijdens conferenties. Hier praten ze over ‘the state of the art’, en dan krijg ik het idee, ’he, ik kan dat in mijn onderzoek betrekken, of ik kan die stap zetten’ Dus dan weet je al vrij zeker dat het vrij nieuw is. En anders is er bij een paper nog een lijst van literatuur referenties. Wat ik doe als ik een idee heb, is daar een korte aantekening van maken, op een half A-4’tje opschrijven. En dan ga ik dat voorleggen aan mensen aan mensen hier op de vakgroep die daar meer van weten. Vervolgens ligt het aan wat voor onderzoek het is, welke de volgende stap is. Als het in mijn promotie traject valt, hoef ik natuurlijk geen project voorstel te schrijven, dan betrek ik het bij mijn onderzoek.<br />
<br />
70<br />
<br />
APPENDIX A. THE INTERVIEWS<br />
<br />
Ik heb recentelijk wel voor het eerst een project voorstel geschreven voor het NWO/STW. Hierin is de eerste stap: waar zou ik geld kunnen krijgen om dit onderzoek te doen. Hebben ze een programma waarbinnen je een voorstel kan doen. Vanuit de eisen (bij het programma) schrijf je dan het project voorstel. En doe je dan ook een literatuur studie? Ja dat moet wel. Meestal zijn dat referenties naar dingen die je al weet. Het moet representatief eruit zien, en dat wordt mede bepaald door de bijbehorende literatuur referenties. Voorts is het belangrijk dat ze een idee krijgen waar het in past, welke domeinen rondom dit onderzoek zijn. Bijvoorbeeld, commercile computer games zet zich zus en zo af tegen het onderzoek naar analytische spelen. Dan krijg je dus artikelen die erom heen zitten, en die eigenlijk niks met het onderzoek op zich te maken heeft. En ga je dan ook op zoek naar mensen die zouden deelnemen aan zo een onderzoek? Nou, ik ken mensen, die kom je tegen op conferenties, of die geven hier een lezing. Voor het projectvoorstel dat ik net gedaan heb, daar krijg je dus een team van mensen rond. Vijf mensen van hier, iemand uit de gaming industry, en iemand vanuit buitenlandse research instituten die hierin genteresseerd is die ik kende. Het is belangrijk dat ze in het team zitten. Ik zie ze als een klankbord. Als ik een paper schrijf, stuur ik hen dat toe, met de verwachting dat ze het lezen en commentaar geven. Bij het begin van het project weet je dus welke richting uit wil? Ja. Maar dat wil niet zeggen dat die richting in stand blijft gedurende het project. Maar als je een voorstel doet voor een project, moet je je einddoel hebben vastliggen, en weten dat het grotendeels haalbaar is. Dat weet je natuurlijk nooit zeker bij een onderzoek, anders hoefde je het onderzoek niet te doen. Ik wilde, toen ik begon bij mijn afstuderen, een realistisch einddoel hebben. En, eventueel, als de weg ernaar toe gemakkelijk is, dan wil je het einddoel verder kunnen leggen. Als je begint met naar literatuur te zoeken, je hebt natuurlijk al een basis, hoe ga je dan te werk? Ik heb bij het begin twee bronnen. De eerste zijn papers, conferenties die ik hier in mappen heb staan. En het tweede is het internet. Heeft iemand hier iets meer over weet? Vanuit papers en hun referenties, ga je naar de websites van de auteurs, kijken wat ze schreven hebben en dan download je hun paper. Er zijn heel veel mensen die zeggen dat ze hun onderzoek beginnen met twee maanden literatuur studie en dan hun onderzoek doen. Dat werkt niet voor mij, mijn inzichten veranderen, dan zoek je nieuwe dingen waar je informatie over wil, dan vind je nieuwe links naar literatuur. Er komen tijdschriften uit Voor mijn gevoel gaat de literatuur studie door. Tijdens promotie onderzoek, ik had wel achtergrond kennis omdat dat aansloot op mijn afstudeer opdracht, vond ik dat ik geen grip had op het vakgebied. Dus dan was ik afhankelijk van mijn begeleiders. En dat kan ook, want dat zijn mensen die er meer van moeten weten en ik ben hier de aankomende student. Ik verwachtte toen van hun te horen dit is goed, dit moet je niet doen want dit is al gebeurd, hier moet je eens naar kijken, . . . . Maar gedurende dat je bezig bent, bouw je steeds meer kennis op, en op een gegeven moment krijg je daardoor een<br />
<br />
A.3. THE INTERVIEW ANSWERS<br />
<br />
71<br />
<br />
specialisme. Dus nu heb ik wel een redelijk goed inzicht in wat ik tot nu toe gedaan heb, hoe dat ligt binnen het domein. En ook dan moet je literatuur natuurlijk continue bijhouden. En als je literatuur bijhoudt, dan weet je welke tijdschriften interessant, hier moet ik zijn voor die informatie? Ik doe het met name met een of twee conferenties. En eigenlijk zou je vakgebied gerelateerde tijdschriften moeten bijhouden. Maar, op het moment dat ik weer met iets begin, met een stukje onderzoek, dan zijn er een paar online tijdschriften waarvan je even de laatste nummers doorkijkt. Enkel online, er is al zoveel online, daar zit altijd wel iets bij. Ga je kijken bij zoekmachines die naar artikelen zoeken? Ja, er is er eigenlijk maar een, citeseer. Maar dat is niet bepaald up to date. Als ik kijk naar hoeveel van mijn eigen papers zijn daar inmiddels gendexeerd, dat is tot 2001, en daarna niks meer. Dus er komt niks meer bij. Google spiders daarentegen hebben die dingen binnen de dag opgepikt. De beste, degene die ik het meest gebruik, is google, want daar kom ik op site van mensen die bepaald research onderwerpen hebben, en iedere researcher heeft een lijst van publicaties, meestal met links erbij naar een source. Als je op zoek bent naar een artikel, naar welke criteria kijk je dan? De titel, de inhoud, het jaar, type artikel, het tijdschrift, de auteurs, Het belangrijkste is eigenlijk de auteur omdat ik eerst kijk of ik mensen kan vinden die op een bepaald gebied onderzoek hebben gedaan, en dan pas kijk ik naar wat ze geschreven hebben. Dus dat is de volgorde. Vervolgens kijk je naar de title. Dan hebben journal papers iets de voorkeur. Je gaat er namelijk ervanuit dat bij een journal paper, iemand echt heeft nagedacht over het onderwerp en heel goed heeft proberen op te schrijven. Terwijl met conference publicaties, dan wil iemand nog wel eens met een ideetje komen dat niet zo goed is uitgewerkt. Van de andere kant zijn in de informatica conference papers belangrijker. In wiskunde en biologie gaat bijna alles via journal papers, in de informatica via conferences. Misschien wel omdat het zo een snel vakgebied is. De tijd om in een journal te schrijven, sneller dan een jaar is het ieder geval nooit. Twee jaar is gebruikelijk, en dan is het onderzoek al bijna achterhaald. Naar jaartallen kijk ik niet specifiek. In de informatica is niet vreselijk veel uit, bijna alles van na 1990. Een paar oude papers die een cultuuromslag zijn geweest, ’toen is dit begonnen’. Boeken willen nog wel eens iets ouder zijn (rond 80-90). Sommige dingen zijn gewoon uitgewerkt in een bepaald boek in die periode. Dat zijn basis boeken. Als je dan kijkt naar wat je toepast, dan moet je wel heel recent zijn. Want zijn dingen die hebben hun tijd, iemand doet dat een keer. Bijvoorbeeld, je hebt iets ‘dat is tot op heden niet gebeurd’, en je hebt een referentie naar 2001, dan mis je 3 jaar, en dat kan niet. En dan is er nog eentje. Mensen hebben soms op hun website papers die niet gepubliceerd zijn. Daar kijk is dus niet naar, ja, ik kijk er wel naar om te weten wat ze doen, maar daar refereer ik absoluut niet naar. Soms zijn er papers die op een enkel online web journal gepubliceerd zijn, en dat wil ik nog wel doen, maar liever niet (ernaar refereren). Maar dat is eigenlijk een kwestie van instabiliteit, wie weet is het er over een jaar niet meer. Als je mogelijk relevante artikelen hebt gevonden, wat doe je daar dan mee? Uitprinten, opslaan, volledige lezen, samenvatten, scannen, . . .<br />
<br />
72<br />
<br />
APPENDIX A. THE INTERVIEWS<br />
<br />
Bijna nooit volledig lezen, wel altijd de abstract en de conclusies, vooral de abstract. Als het dan echt relevant is, maar op een gegeven moment heb ik het door. Het ligt eraan hoe belangrijk het is voor mijn research. Als het echt belangrijk is, ga ik het ook echt tot in detail te begrijpen, inclusief de formules die erin staan. Maar anders is het belangrijk te weten wat heeft deze persoon gedaan geconcludeerd, niet hoe hij dat precies heeft gedaan. En wat doe je daar dan mee? Opslaan, bookmarken? Als het belangrijk is print ik het uit, en stop ik het in een map. Ik heb op mijn computer ook een directory staan van 500MB aan documenten die ik ooit een keer heb gelezen, zelfs alleen soms alleen de titel en gedacht dat het relevant of interessant zou kunnen zijn. 1 grote map met alles door elkaar. Wat ik wel vaak merk is, als ik een paper lees met een referentie, dan heb ik die vaak al in die map staan, en dan haal ik hem even erbij. Ze zijn opgeslagen met ’auteurs - titel’. Dus, met zoekwoorden kan ik die directory doorgaan. Dus je hebt dan ook geen moeite met terug vinden van artikels? Uit die map wel, maar de dingen die ik wel iets belangrijker vind, die print ik uit en stop die stop ik mappen. Die mappen blader ik dan door, van zit hier iets relevant in. Die zijn wel gestructureerd, maar dat begint door elkaar te lopen. Maar dat zijn drie of vier mappen, een paar honderd papers zitten daar misschien in, en dat is niet veel werk om die te doorlopen. Ik weet wel waar die papers over gaan, dat is wel goed. Gebruik je geen document management systeem? Nee. Zou je dat willen zo een ondersteuning? Ja. Welke functies zou je daar dan graag in willen zien? Liefst zou je willen dat zo een tool die map automatisch vult. Stel dat ik mijn de documenten uit mijn directorie door die tool gooi, dan zou hij de keywords eruit moeten pikken. Vaak heeft zo een document al bepaalde keywords en bepaalde ordening. Indien dat niet zo is, dan zou je hij die keywords zelf moeten bedenken (die niet in het document zelf hoeven te staan). Titel, auteur en abstract samen zetten, dat gebeurd al bij onder andere citeseer. Maar die neemt maar enkele regels, en dat is weer niet goed, want je wil zeker de eerste regels zien omdat daar het probleem beschreven wordt. Dat zijn de voor de hand liggende dingen natuurlijk. Waar je naar heen zou kunnen gaan, maar dat is pas achter tien jaar ofzo. Dat is dat je een conversatie met zo een programma zou kunnen aangaan. Dat je zegt, ik ben bezig met dit onderwerp. Dan krijg je al een schift, maar dat zijn nog heel veel artikelen. En dat je dan tegen zo een systeem kan zeggen, ’het gaat om dit specifieke probleem’, en dat het systeem dan weet onder welke verschillende namen dat probleem bekent staat. En dat dat systeem dan weet, er zijn papers die dat probleem enkel noemen, en andere papers die echt over dat probleem handelen. Dus dat zo een systeem meer begrip heeft ervan, wat zit er echt in dit artikel. Natuurlijk is dat geen echte intelligent, hij moet dat niet echt begrijpen, maar er zijn volgens mij wel analyse tools die dat zouden kunnen doen. Of bijvoorbeeld de impact van het ene probleem op dat onderwerp, dat gaat om de vorm<br />
<br />
A.3. THE INTERVIEW ANSWERS<br />
<br />
73<br />
<br />
van de relatie. Ik denk ook dat dat mogelijk is omdat artikels gestructureerd zijn, je weet dat de conclusie belangrijk is, evenals de volgorde in het artikel. Ook dat zo een systeem bijhoudt of jij het al gelezen of verwerkt hebt? Wel, eigenlijk weet ik dat al, maar het zou een kleine moeite zijn om dat erin te zetten. Elk artikel heeft een lijst van referenties, en daar bouw je een semantisch netwerk mee op, daar zou je al iets mee kunnen gaan doen. Dan zou het mogelijk zijn om van een referentie naar een artikel te springen. Maar ik denk niet dat ik op die manier zou zoeken, en bovendien moet je dan rekening houden met copyrights. Maar ik heb al zoveel artikels verzameld, dat het zinvol zou kunnen zijn. Welke zijn de meest voorkomende ongemakkelijkheden tijdens een literatuurstudie? Belangrijkste is, denk ik, je moet ’related work’ op een gegeven moment vinden. Als je een paper schrijft moet je vertellen, dan moet je vertellen hoe het past in wat er nog meer gebeurd. Maar je kent niet het hele onderzoeksgebied, en niet iedereen gebruikt dezelfde termen. Dus het is al heel vaak moeilijk te vinden wat voor term zou worden gebruikt voor wat ik nu heb bedacht. Ik had het geluk dat een reviewer voor een van mijn papers me kon doorverwijzen naar andere artikels die net weer iets andere termen gebruikten. Hoe vind je dat nu, hoe vind je wat voor jou related work is? Dat is het belangrijkste. Zou je behoefte hebben aan een systeem dat literatuurstudie ondersteund? Dat hangt van de kwaliteit van het systeem af. Literatuurstudie kost mij nu niet vreselijk veel tijd. Ik denk dat het zelfs een nadeel zou kunnen vormen, als ik allerlei papers download, zonder te lezen, en op een gegeven moment een paar zoektermen ingeef en daar komen relevante papers omhoog. Want je moet een beeld krijgen van je vakgebied, dus je moet ze op een gegeven moment zelf lezen. Je moet die abstract en die conclusies lezen. Je komt er niet mee om alles aan het systeem over te laten. De manier waarop je nu veel kennis op doet is op conferenties. Je hoort ideen maar dat is toch maar een heel klein gedeelte, en je zal er op een gegeven moment toch moeten gaan kijken naar een andere manier om met de rest om te gaan. Je begint op een gegeven moment mensen in de gaten te houden die interessant zijn. Daar kijk ik ’wat doen ze nu eigenlijk’. Maar dan krijg je toch een beetje een kliekje, dezelfde mensen die je in de gaten houdt. Die jou waarschijnlijk ook in de gaten houden, en daar moet je toch buiten kunnen kijken. Dus ik denk dat het enerzijds contacten zijn met anderen waardoor je contacten krijgt. Na een conferentie presentatie komen er ook mensen naar je toe die je verwijzen naar relevante bronnen (wederzijds). Tweede is dat je naar referenties kijkt van een paper dat voor jouw relevant is. Maar alleen die referenties, dat is niet genoeg, je moet naar de papers kijken hoe belangrijk ze zijn. Dus, ondersteuning, dat zou in sterke mate zijn het terugvinden van dingen, of en het vinden van dingen die voor jou niet direct relevant zijn, maar belangrijk voor andere mensen zodat zij kunnen terug vinden hoe hun gebied past bij jouw gebied. Ik zou het even graag willen hebben over kennisdeling. Bijvoorbeeld, het aanraden van een artikel aan iemand anders. Dat doe ik al, ik stuur mensen meteen een e-mail als ik iets relevants heb gevonden, vooral<br />
<br />
74<br />
<br />
APPENDIX A. THE INTERVIEWS<br />
<br />
afstudeerders, ik denk mee met anderen. Maar ik denk, als zo een tool goed is, hoef je dat niet door te sturen. Ik zou zo een tool niet voor mezelf willen, maar bijvoorbeeld voor de hele vakgroep, waar iedereen zijn papers ingooit, misschien kun je dingen aanduiden die je echt interessant vind, een soort van filter. En verder kunnen dan ook studenten daar op kunnen werken. En dan zou ik tegen een student kunnen zeggen, typ die en die termen in en kijk wat er naar boven komt. Voor kennisdeling is het meest relevante dat je de tool niet voor jezelf moet gebruiken. Net zoals een echt goed document management system, die het schrijven door meerdere mensen ondersteund. Als er een nieuw projectlid of afstudeerder komt, dat deze ondersteund word door middel informatie terug te gegeven dat door ieder projectlid word gedeeld. Ik kan me voorstellen dat dat gebeurd. Je kan dat soort informatie op verschillende manier toepassen. Als je die documenten er eenmaal in hebt ga je proberen zoveel mogelijk attributen vastleggen, zo van, wie is waar genteresseerd, welk is het vakgebied dan heb je automatisch mensen gelinkt aan attributen. Vervolgens krijg je dan een soort van case-based reasoning system, en iemand gaat dat gebruiken, en legt dan het systeem aan zijn eigen profiel vast of aan zijn eigen interesses. Het systeem kan dat ook leren,als je op een gegeven moment gaat schiften dan blijft er op een gegeven moment een aantal over en dat is een training van het systeem, dat kan voor jou het meeste sturend zijn. Zou je ook iets zien in kennisdelen met derden,over een gerelateerd gebied waar je nog niet zo bekend mee bent ? Ik zou in ieder geval moeten weten wie het zijn,om hun artikelen te kunnen bekijken. Bij interviews of e-mail geef ik een kort antwoord, want meestal heb je daar geen tijd voor. Daarom ga je enkele keren per jaar naar een conferentie, hier zijn mensen met relevante dingen bezig, daar kun je langere tijd nemen voor gesprekken. Als je een artikel schrijft, dan is het in ieder geval om ideen uit te dragen. Als iemand mijn artikel leest,en daar gerichte vragen over stelt, van dit snap ik niet, dan wil ik daar best antwoord op geven, want dan heeft hij zijn best gedaan om zich te verdiepen in een artikel van mij en dat is dan weer relevant voor mij. Als iemand gewoon vraagt, jij kent daar veel van, vertel daar eens iets over, dat werkt natuurlijk niet, tenzij die persoon in de kamer langs jou zit. Welke is de volgorde van jouw voorkeur om ondersteuning te bieden voor de drie onderdelen die hier aan bod zijn gekomen? 1 Ondersteuning voor literatuur studie. 3 Ondersteuning voor kennisdeling met projectleden. 2 Ondersteuning voor kennisdeling met derden. (In volgorde van belangrijkheid) Kennisdeling met projectleden, dat gaat op andere manieren. Over het algemeen staat men wantrouwig tegenover informatie zoek systemen,waaruit bestaat jou wantrouwen. Te beperkt gezicht, je blikveld is niet ruim genoeg, je ziet net niet de dingen die je omringen. Een bron van ideen valt weg, want, andere niet relevante artikelen geven soms nieuwe invalshoeken waarmee je nieuwe ideen kan zien, dit is wel niet echt literatuurstudie, maar is wel nuttig.<br />
<br />
A.3. THE INTERVIEW ANSWERS<br />
<br />
75<br />
<br />
Literatuurstudie verdiept je gezichtsveld, en het hoeft niet echt je interesse te zijn, vb., je ziet een wiskundig artikel dat je niet direct interesseert maar wel nodig is voor je studie, dan ga je het toch lezen, want je hebt het nodig. Het systeem mag dan niet zeggen, dit maar niet, want het is wel nodig voor je onderzoek. Je moet weten dat wat relevant is voor het onderzoek iets anders kan zijn dan wat relevant is voor je interesse. Je kan dit systeem op twee manieren bekijken, geef mij wat artikelen die voor mij leuk zijn en in mijn vakgebied zitten, hier haal ik dan de relevante informatie uit. Of, ik wil specifieke literatuur in mijn vakgebied, dus wat voor mij relevant is en heel strikt gebonden, dat zou dan een andere manier van gebruiken zijn. Dit is een andere manier van privacy. Verder heb ik geen wantrouwen in dat soort dingen, bijvoorbeeld om mijn profiel aan wetenschappelijke zoekmachines te geven. Zou het voor jou nuttig zijn, als het jou profiel al weet, zich direct zichtbaar zou maken? Het is natuurlijk interessant als het systeem je zoekwerk herkend, zodat jij dan kan zien binnen welke context jou artikel ligt, en dan zou jij weer reageren met zie dat iets breder of, wijdt op dit artikel iets uit. Het is vaak wel interessant te zien op welke gronden iets geselecteerd is, vooral als je denkt waarom zou het voor mij relevant zijn het slaat nergens op zo kan het voor mij wel relevant zijn en kan het systeem wel testen en trainen. Hij laat alle vormen van informatie zien, maar wel in een volgorde die voor jou relevant is, is dit van belang voor jou? De volgorde van relevantie is vaak moeilijk te bepalen, want hoeveel gewicht moet je er aanhangen voor mijn interesse, of metwelk deel van je paper ben je bezig. Wat daar wel goed aan is, is dat het systeem filtert. Het haalt de kernartikelen boven die dan ook weer gelinkt zijn aan andere achtergrond artikelen, deze heb de beide nodig, en niet alleen de kernartikelen. Als je zo een systeem zou hebben, en je schrijft een artikel, zet het in het systeem, en het classificeert het meteen door te bepalen hoe het in relatie met andere artikelen gaat refereren, hoe belangrijk ze zijn. Je kent de cold-start problems van de user profile, het systeem moet eerst gevuld zijn vooraleer je het kan gebruiken, ben je bereid hieraan mee te werken? Als ik het wil gebruiken, moet ik er natuurlijk ook aan meewerken. Is voor mij niet zo een probleem, ik heb dertig artikels geschreven die ik in het systeem kan zetten, en mijn profiel is gekend. Studenten hebben het moeilijker, maar hier zou je het profiel kunnen aflezen aan de hand van de gelezen artikels, en de interesse die ze hebben getoond in het systeem. Studenten moeten zich nog vormen, hun profiel vormt zich ook nog. Hoe belangrijk is zo een profiel, stel je begint een nieuw onderzoek met andere mensen dan is dit systeem wel relevant. Je zou in zo een prototype kunnen proberen de reden mee te nemen waarom de onderzoeker het systeem gebruikt. Hoever zou je gaan met dit systeem om dat duidelijk te maken? Als er wordt gevraagd om een halve dag dat systeem te trainen, dan zou ik dat doen, als ik<br />
<br />
76<br />
<br />
APPENDIX A. THE INTERVIEWS<br />
<br />
weet dat het een goed systeem is. Maar uiteraard, op een nuttige manier trainen, niet lees 10 artikels en beoordeel ze. Met relevancy feedback, zoals spam-base, maar dan moet er wel vooruitgang in zitten. Als er vijf op tien relevant zijn, ben ik al tevreden. En als het iets heel specifieks is waar ik heel lang naar moet zoeken dan ben ik al heel tevreden met een op tien relevante resultaten, want dit vergt mij meer tijd.<br />
<br />
Appendix B<br />
<br />
The evaluations<br />
<br />
B.1<br />
<br />
Participants<br />
<br />
The two junior researchers who evaluated the prototype are listed below in alphabetical order. • Drs. L. Braun • Drs. F. de Jonge<br />
<br />
B.2<br />
<br />
The sets of items for testing<br />
<br />
Below, for set A the number of documents retrieved from CiteSeer given a certain search string are shown, a * indicates that the results are from a non-boolean query. • Medical information retrieval (9) • Medical information (26) • Retrieval results ranking (0) • Results ranking (44) • Query enhancement methods (18) • Information need extraction (1) • Medical passage retrieval* (6) Below, for set B the number of documents retrieved from CiteSeer given a certain search string are shown, a * indicates that the results are from a non-boolean query. • Conflicts in plan execution (2) • Conflict diagnosis (4) • Distributed discrete event systems (2) 77<br />
<br />
78<br />
<br />
APPENDIX B. THE EVALUATIONS<br />
<br />
• Multi-agent plan execution (0) • Conflicts in plan execution (3) • Distributed Model-Based Diagnosis and Repair* (89) (this is the title of the project) We showed the researcher who evaluated the prototype with this item set a detailed overview of which documents were retrieved from CiteSeer for the search strings she provided1 . For the documents of the second search string, for which four documents were retrieved, she examined that two of them were definitely not relevant.<br />
<br />
B.3<br />
<br />
Experiments<br />
<br />
Below, the experiments are transcribed. For item set A, one experiment is provided (another experiment had similar results). For item set B, two different experiments are described. An experiment on set A Iteration 1: Search string: ‘retrieval results ranking’. 8 sections are evaluated: sections on rank 6 and 7 received a positive feedback, sections on rank 1 to 5 and 9 received negative feedback. Accepted enrichment: answer (position 3). Iteration 2: Search string: remains unchanged. 5 sections are evaluated: sections on rank 1 to 5 received positive feedback. Enrichment: the first two words are accepted. Iteration 3: Search string: remains unchanged 5 sections are evaluated: sections on rank 2 to 5 received a positive feedback, section on rank 1 received negative feedback. Enrichment: words on first five positions are accepted. Experiments on set B Below, two experiments are described. The first is denoted by ‘Quest 1’, the second is denoted by ‘Quest 2’. Quest 1: Iteration nr.1: Search string: ‘Conflicts in plan execution’. 7 sections are evaluated: sections on rank 1 to 7 received a positive feedback. Accepted enrichment: Agent (1), coord (2), avoid (5), action (7), and solution(8). Rejected enrichment: level (3), permit(4), fig (6) (Note, each of the positively evaluated sections came from only two documents. These two documents were also the two documents provided by CiteSeer (see B.2).) Iteration 2: 1<br />
<br />
That are the first five search strings<br />
<br />
B.3. EXPERIMENTS<br />
<br />
79<br />
<br />
Search string: remains unchanged. 20 sections are evaluated: sections on rank 1, 2, 6 received positive feedback, sections on rank 4, 7, 14, 18 received negative feedback. Accepted enrichment: similar as previous iteration. (Note, according to the researcher, the second iteration did not provide more valuable information. It is likely that in the first iteration most relevant sections were already retrieved since there are only two relevant documents for this subject.)<br />
<br />
Quest 2 Iteration 1: Search string: ‘Durfee’. 3 sections are evaluated: sections on rank 1 to 3 received positive feedback. Accepted enrichment: plan, proceed, hierarch, and coordin. (Note, according to the researcher, the search string is an author who is familiar with ‘conflicts in plan execution’ (the search string in quest 1). This was also detectable in the resultant set, the document titles of the positive rated items from the resultant set, are similar to the document titles of positive rated items in quest 1. Furthermore, one of the positively rated sections contains part of a reference list, so, typical words, such as ‘proceeding’ appeared in the proposed enrichment.) Iteration 2: Search string: ‘Durfee’ 0 sections evaluated Accepted enrichment: none (Note, at this point, the researcher had sufficient information. Therefore, no sections are evaluated, and no enrichment is accepted.) Iteration 3: Search string: ‘Conflicts in diagnosis’ (Note that this search string is rather not related to the previous search string.) 0 sections evaluated. Accepted enrichment: none (According to the researcher, the influence of the previous search string ‘durfee’ is definitely present in the resultant set and the proposed enrichment. The word in the memory (which represents the short-standing interests) are merely related to the search string of the previous iteration (‘durfee’). Therefore, no sections received positive feedback, and no enrichment was accepted.) Iteration 4: Search string: ‘Conflicts in diagnosis’ 6 sections evaluated: sections on rank 2, 3, 5 received positive feedback. Memory: diagnosis received the highest weight. (Note, according to the researcher, this is an acceptable resultant set since the influence of the previous search string is nearly gone. Hence, it took the prototype three iterations to reduce that influence. This behaviour of the prototype is explained by the functioning of the memory: after the first iteration, the words in the memory have a small weight, and three iterations later, the weights are decreased through dilapidation, consequently their influence is nearly imperceptible).<br />
<br />
80<br />
<br />
APPENDIX B. THE EVALUATIONS<br />
<br />
B.4<br />
<br />
Opinions<br />
<br />
We gave a short presentation to each researcher about how our model proposes to personalise the items for a certain researcher (i.e. long-standing interests, short-standing interests (with respect to the search goals), and their personalising link with the possibly relevant items). Since they are researchers and the model is about researcher, we asked and expected that they commented the model on anything that is incorrect. Furthermore, during the evaluation of the prototype, we asked the researcher for their opinion on (1) items as sections, (2) the enrichment, (3) the memory, (4) the personalisation functioning, and (5) wether they would use such a system during their quest for information. Furthermore, they spontaneously commented on other issues regarding the prototype. These reactions are provided below in Dutch, first the opinions of the researcher who worked with item set A, then the opinions of the researcher who worked with item set B. The opinions of the researcher who worked with item set A are summarised below. Our model: heel aannemelijk (na de uitlege over de context van een document, die overeenkomt met de long-standing interests). Items as sections: Je hoeft maar een klein stukje door te lezen, dus feedback geven kan zonder veel moeite. The enrichment: goed idee, hij zet er termen bij waar je zelf niet op kwam. The memory: 1. Wat gebeurt er met het memory? Ik zou het graag beschikbaar willen houden om het een volgende keer te gebruiken. 2. Ja, je ziet toch dat ie belangrijke woorden een hoog gewicht geeft. Personalisation functioning: Ja, het prototype personaliseert heel zeker. Using such a system: Werkt gemakkelijk, en er is geen verplichting (als je niet wilt dat de personalisatie werkt, dan hoeft dat ook niet). Ik zou het alleszins uitproberen voor mijn literatuur studie. Other comments 1. Titel bij sectie: stuurt de gebruiker een beetje in de war, voor eenzelfde titel wordt naar eenzelfde feedback geneigd. 2. Ik zou graag een overzichtelijke history willen zien bij het ‘search object’, om te zien hoe mijn query is veranderd gedurende verschillende iteraties. Gui Issues: 1. Het zou duidelijker zijn te zien op wat het prototype gezocht heeft door in de sectie de woorden aan te duiden die ook in het ‘search object’ voorkomen. 2. dat het memory prominent aanwezig is, is niet storend, maar ik zou het liever zien als beschikbaar, bijvoorbeeld onder een knop. 3. Het zou gemakkelijker zijn als je enkel hoefde te klikken op een woord uit het ‘enrichment’ of uit het ‘memory’. 4. Het werkt relatief gemakkelijk, intuitief, en er is geen werk verplichting, als je niet wil, dan hoeft het niet. Dat vind ik ook goed.<br />
<br />
B.4. OPINIONS<br />
<br />
81<br />
<br />
The opinions of the researcher who worked with item set B are summarised below. Our model: ok (na de uitleg over de context van een document, die overeenkomt met de longstanding interests). Je merkt ook dat sommige secties van een document wel interessant zijn, maar de rest van het document is irrelevant. Sections as items: 1. Dat is goed, maar ik zou graag willen zien waarom. 2. Hij geeft ook een lijst van referenties terug als een sectie, dat had ik niet verwacht, het is wel slim. 3. Het zoeken in het gehele document op elke woord is goed, het is dikwijls dat je zonder de juiste termen niet vind wat je moet hebben omdat er bijvoorbeeld enkel in de titel en het abstract wordt gezocht. The enrichment: 1. Je ziet toch dat je op deze manier achter zoektermen komt waar je zelf niet was opgekomen, en je komt achter woorden die andere gebruiken. 2. Ik merk wel dat het prototype relvant termen erbij vind, dat is fijn. 3. Hij geeft ook termen erbij zoals ‘fig’ (de stam van figuur). The memory: 1. Het ene moment zoek ik naar ‘conflicts’, en andere momenten zoek ik naar ‘diagnosis’, als je ook hun memories bewaart. Omdat je heel gericht aan het zoeken bent, kan je beter zelf het memory kiezen en ik vind het zonde om het oude weg te doen. 2. ik denk wel dat de forgetting frequency de juiste maat heeft. The personalisation functioning: Het prototype komt heel zeker over alsof het personaliseert. Using such a system: 1. Het heeft relatief veel hits, normaal als niet de goede termen hebt, is het moeilijk om relevante informatie te vinden, dus hier is het goed dat het prototype die meerdere termen aangeeft, en zoekt in de inhoud van een document. 2. Ik zou het gebruiken, voornamelijk om twee redenen, het zoekt in de text van een document zelf, dat vind ik heel waardevol, en het voegt termen toe aan je zoek query, en dat kan je zelf ook doen. Het functioneert erg goed. Other comments: 1. Ik zou graag een soort van ‘back button’ funtie willen, om na een resultant set naar de vorige staat terug te gaan, zodat je dan andere woorden kan selecteren uit het enrichment. 2. Binnen je eigen literatuur zoeken dat zou ook fijn zijn. Gui Issues: 1. Je hebt het liefst snel door waar de secties over gaan, dat zie je niet duidelijk. Soms staat de titel van de sectie erboven, dat is natuurlijk fantastisch, maar dat heb je niet altijd. 2. Gebruiksvriendelijker, op een woord van de voorgestelde enrichment of het memory klikken om het bij de beschrijving te krijgen.<br />
<br /> </div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<div class="modal fade" id="report" tabindex="-1" role="dialog" aria-hidden="true">
<div class="modal-dialog">
<div class="modal-content">
<form role="form" method="post" action="https://adoc.pub/report/personalisation-for-researchers" style="border: none;">
<div class="modal-header">
<button type="button" class="close" data-dismiss="modal" aria-hidden="true">×</button>
<h4 class="modal-title">Report "Personalisation for researchers"</h4>
</div>
<div class="modal-body">
<div class="form-group">
<label>Your name</label>
<input type="text" name="name" required="required" class="form-control" />
</div>
<div class="form-group">
<label>Email</label>
<input type="email" name="email" required="required" class="form-control" />
</div>
<div class="form-group">
<label>Reason</label>
<select name="reason" required="required" class="form-control">
<option value="">-Select Reason-</option>
<option value="pornographic" selected="selected">Pornographic</option>
<option value="defamatory">Defamatory</option>
<option value="illegal">Illegal/Unlawful</option>
<option value="spam">Spam</option>
<option value="others">Other Terms Of Service Violation</option>
<option value="copyright">File a copyright complaint</option>
</select>
</div>
<div class="form-group">
<label>Description</label>
<textarea name="description" required="required" rows="3" class="form-control" style="border: 1px solid #cccccc;"></textarea>
</div>
<div class="form-group">
<div style="display: inline-block;">
<div class="g-recaptcha" data-sitekey="6LcEnMcZAAAAAFHzKKgaE-Qh57XzUTzpfRElD4dP"></div>
</div>
</div>
<script src='https://www.google.com/recaptcha/api.js'></script>
</div>
<div class="modal-footer">
<button type="button" class="btn btn-default" data-dismiss="modal">Close</button>
<button type="submit" class="btn btn-primary">Send</button>
</div>
</form>
</div>
</div>
</div>
<script>
$(document).ready(function () {
var inner_height = $(window).innerHeight() - 250;
$('#pdfviewer').css({"height": inner_height + "px"});
});
</script>
<footer class="footer" style="margin-top: 60px;">
<div class="container-fluid">
Copyright © 2024 ADOC.PUB. All rights reserved.
<div class="pull-right">
<span><a href="https://adoc.pub/about">About Us</a></span> |
<span><a href="https://adoc.pub/privacy">Privacy Policy</a></span> |
<span><a href="https://adoc.pub/term">Terms of Service</a></span> |
<span><a href="https://adoc.pub/copyright">Copyright</a></span> |
<span><a href="https://adoc.pub/contact">Contact Us</a></span> |
<span><a href="https://adoc.pub/cookie_policy">Cookie Policy</a></span>
</div>
</div>
</footer>
<!-- Modal -->
<div class="modal fade" id="login" tabindex="-1" role="dialog" aria-labelledby="myModalLabel">
<div class="modal-dialog" role="document">
<div class="modal-content">
<div class="modal-header">
<button type="button" class="close" data-dismiss="modal" aria-label="Close" on="tap:login.close"><span aria-hidden="true">×</span></button>
<h4 class="modal-title" id="add-note-label">Sign In</h4>
</div>
<div class="modal-body">
<form action="https://adoc.pub/login" method="post">
<div class="form-group">
<label class="sr-only" for="email">Email</label>
<input class="form-input form-control" type="text" name="email" id="email" value="" placeholder="Email" />
</div>
<div class="form-group">
<label class="sr-only" for="password">Password</label>
<input class="form-input form-control" type="password" name="password" id="password" value="" placeholder="Password" />
</div>
<div class="form-group">
<div class="checkbox">
<label class="form-checkbox">
<input type="checkbox" name="remember" value="1" />
<i class="form-icon"></i> Remember me
</label>
<label class="pull-right"><a href="https://adoc.pub/forgot">Forgot password?</a></label>
</div>
</div>
<button class="btn btn-primary btn-block" type="submit">Sign In</button>
</form>
</div>
</div>
</div>
</div>
<!-- Global site tag (gtag.js) - Google Analytics -->
<script async src="https://www.googletagmanager.com/gtag/js?id=UA-177279549-1"></script>
<script>
window.dataLayer = window.dataLayer || [];
function gtag(){dataLayer.push(arguments);}
gtag('js', new Date());
gtag('config', 'UA-177279549-1');
</script>
<script src="https://adoc.pub/assets/js/jquery-ui.min.js"></script>
<link rel="stylesheet" href="https://adoc.pub/assets/css/jquery-ui.css">
<script>
$(function () {
$("#document_search").autocomplete({
source: function (request, response) {
$.ajax({
url: "https://adoc.pub/suggest",
dataType: "json",
data: {
term: request.term
},
success: function (data) {
response(data);
}
});
},
autoFill: true,
select: function( event, ui ) {
$(this).val(ui.item.value);
$(this).parents("form").submit();
}
});
});
</script>
<!-- cookie policy -->
<div id="ADOCPUB_cookie_box" style="z-index:99999; border-top: 1px solid #fefefe; background: #FFC107; width: 100%; position: fixed; padding: 5px 15px; text-align: center; left:0; bottom: 0;">
Our partners will collect data and use cookies for ad personalization and measurement. <a href="https://adoc.pub/cookie_policy" target="_blank">Learn how we and our ad partner Google, collect and use data</a>. <a href="#" class="btn btn-success" onclick="accept_ADOCPUB_cookie_box();return false;">Agree & close</a>
</div>
<script>
function accept_ADOCPUB_cookie_box() {
document.cookie = "ADOCPUB_cookie_box_viewed=1;max-age=15768000;path=/";
hide_ADOCPUB_cookie_box();
}
function hide_ADOCPUB_cookie_box() {
var cb = document.getElementById('ADOCPUB_cookie_box');
if (cb) {
cb.parentElement.removeChild(cb);
}
}
(function () {
var ADOCPUB_cookie_box_viewed = (function (name) {
var matches = document.cookie.match(new RegExp("(?:^|; )" + name.replace(/([\.$?*|{}\(\)\[\]\\\/\+^])/g, '\\$1') + "=([^;]*)"));
return matches ? decodeURIComponent(matches[1]) : undefined;
})('ADOCPUB_cookie_box_viewed');
if (ADOCPUB_cookie_box_viewed) {
hide_ADOCPUB_cookie_box();
}
})();
</script>
<!-- end cookie policy -->
</body>
</html>
