The virtual museum tour guide An eye-fixation based recommender system

The virtual museum tour guide An eye-fixation based recommender system

Bachelor Project by Chris Janssen University of Groningen Department of Artificial Intelligence [email protected]

Supervisors: Leendert van Maanen Hedderik van Rijn

October 2005 - May 2006

The virtual museum tour guide An eye-fixation based recommender system In this Bachelor project an implementation of a personalized and adaptive tour guide within the setting of a virtual museum has been made. In accordance with our hypothesis that people look at items that they are interested in, we use eye-gaze information as a mechanism for deriving a user’s interests and directing the tour. The guide distinguishes several items within each painting and offers anecdotic information on the most attended items. In an experiment, the hypothesis that users are interested in this kind of information was tested. We used two conditions: one in which users received information about the most attended items and one in which they received information on the least attended items. Our results showed a trend in line with our hypothesis. They also show that a lot of factors are of influence on the interest of a user and his rating of the tour guide. This pleads for usage of virtual museum tour guiding systems that can adapt themselves to the interests of users.

On the cover: on top of the painting “The Sacrifice of Iphigenia” by Jan Steen (Rijksmuseum code sk-a-3984) the eye-fixations of a person, who gazed for 55 seconds at the painting, are positioned. It can be seen that the person mostly fixated on specific items. We hypothesise that the person is interested in those items and wants to receive audio information about them.

2

Introduction When you visit a museum it is often interesting to take a tour. Tour guides can give you detailed information about each art object. You could make a distinction between good and bad tour guides. A bad tour guide takes you on a fixed tour. He would for instance show only the top art pieces. A good tour guide however, would observe his audience: he notices how they react on his explanations and tries to find out what they find interesting. Based on this and other information about the visitors, he adapts his tour during the tour. This all to make sure that the visitor only sees art that, supposedly, is interesting for him. Besides real tour guides, there are also virtual museum tour guides: computer programs that give you a tour past digitalized art, for instance art that has been placed on a website. And just like in real life, you could also distinguish good and bad virtual museum tour guides. A bad virtual tour guide would have an algorithm which takes you on a fixed tour and would always give the user the same information for each art work. A good virtual tour guide however, would also notice the visitor’s interests and adapt his tour accordingly. This way, each user of a virtual tour guide would get his own, unique tour. In this Bachelor Project a first attempt was made for creating such an adaptive tour guide that gives information about items on paintings. We set as an objective that the tour guide should be personalized and adaptive (Van Maanen, Van Rijn, & Schomaker, 2005). So each visitor of the virtual museum should get his own unique tour, in coherence with his specific interests. In order to be personalized, the system has to infer a user’s interests. Based on these interests, the rest of the tour can be adapted. As it is often difficult for a user to give all his relevant and specific interests before he starts taking a tour, we wanted the system to derive these interests by itself. In order to trouble the user as less as possible, this should be done in an implicit and non-intrusive way. In the current virtual museum tour, we used eye-gaze information as an implicit input mechanism for this recommender system. The system tracked which items on a painting were attended to by a user. Based on these measurements it derived the most attended item and gave anecdotic audio information on those items. In an experiment the hypothesis that users like to receive information about the most attended items on a painting was tested. The results were compared to another condition in which users received information about the items that they attended to the least. Eye-gaze data were compared to the answers to a questionnaire to derive more conclusions.

Theory For our research we are interested in a recommender system that gives anecdotic information about paintings. The system implicitly obtains information about user interest, using eye-gaze information. The system gives audio information as output, as it tells interesting details about items on a painting. In this theoretical section, some background knowledge on all of these topics will be given. Recommender systems Recommender systems are defined as “systems that will make a personal (or otherwise relevant) selection of information from a stable, or semi-stable, domain” (Waern, 2004). In the case of a virtual museum tour guide, the domain is stable and two conceptual levels could be distinguished: art pieces and items that are depicted on the art pieces. Recommendations could be made about items of both levels. The recommender system that is discussed in this paper only gives information about items depicted on a painting.

3

Within the domain of recommender systems, two kinds might be distinguished: collaborative recommender systems and content-driven recommender systems (Balabanovic & Shoham, 1997). In collaborative recommender systems, future recommendations for current users are based on actions made by and recommendations made to previous users. The system analyses the similarity between users. If there is data from a previous user whose behaviour matches the behaviour of the current user, the system gives a recommendation or takes an action that was also done for the previous user. The other class of recommender systems are content-driven recommender systems. These systems make recommendations purely on the basis of the content of items that a user has liked in the past. For instance, if a user liked jazz music by Frank Sinatra in the past, it is likely that he likes jazz music in the present. A recommender system might therefore suggest jazz numbers of other artists in the future, because those are also jazz numbers. Our museum tour guide also uses a content-driven recommender system1. Explicit and implicit input The input for a recommender system might take several forms. In this paper I distinguish between explicit and implicit input. With explicit input, a user is conscious of the actions he wants the system to take and he explicitly states a command for those actions. Examples are command line prompts or voice commands (see Kaur et al., 2003 , for an advanced example using eye-gaze and voice commands). In a certain way, explicit input is not very desirable in a natural interaction setting. The user has to state the information that he wants to receive explicitly, even if he has got no clue which information is available or which information he wants to receive. In a virtual museum setting in particular, it is very unlikely that the user knows all the art-works and the items on the art-works. As a result, it might be very difficult for him to make a well-considered choice, stating which artworks he wants to see and what information he wants to hear about items on those artworks. The system therefore should anticipate on the desires of the user, even if he does not explicitly state them. A solution for this information retrieval problem is using implicit input. Implicit input is derived by the system, without the user consciously knowing which input he gives. The usage of implicit input might be compared to the usage of non-verbal communication between humans. In a normal conversation, people make a lot of gestures, eye-gazes and other nonverbal movements that influence conversation (see Cassell, 2000, for another application in which principles of non-verbal communication are applied). Thus, although implicit input is not explicitly stated, it can certainly influence processes. Recommender systems that are based on implicit input make decisions ‘themselves’, while using data of the user’s actions and not his commands. Some examples of implicit data types that could be used are clicking behaviour of the user (“what items does he click on?”), spontaneous speech (spontaneous saying “that is beautiful” for instance might indicate an interest) and eye-gaze information. Eye-gaze information Eye-gaze information is a general term for information that can be obtained from the movement of the eye over a time interval. While perceiving a scene, users constantly move their eyes, focusing on subsections of the scene. The movements of the eye from one point to another are called saccades. From time to time, humans tend to fixate on a more or less stable position with a mean fixation time of about 250-300 ms (Viviani, 1990). If a user can freely view a scene, fixations can even range from 100 ms to several seconds (Velichkovsky, 1

see the section “The virtual museum guide” for more information about this content-driven recommender system

4

Dornhoefer, Pannasch, & Unema, 2000). Fixations are used to perceive a scene, although it is not fully understood how they are used exactly and up to what level. At the moment there is no clear consensus among scientist how human eye-gaze is involved in high-level scene perception (see Henderson & Hollingworth, 1999, for a good discussion). However, past research has shown that people tend to fixate on regions of scenes that they find interesting or informative (Henderson, 2003; Henderson & Hollingworth, 1999). In the beginning of a scene perception, they are mostly guided by visual stimuli and not by semantic knowledge. People however have a longer first pass gaze duration (the sum of all fixations from first entry to first exit in a region) for semantically informative objects (Loftus & Mackworth, 1978) and also a longer second pass duration (Henderson, Weeks, & Hollingworth, 1999). They are guided more and more by semantic information in contrast to visual stimuli as they gaze longer at a scene. So in conclusion, eye-gaze might be informative for determining a user’s interest in items on paintings. There are numerous ways in which eye-gaze and eye-fixations can be used for research in human-machine interaction (see Goldberg & Kotval, 1999, for an extensive review on methods and constructs for eye-gaze measures). Our method is defined in the experiment section below and uses cumulative fixations. Audio and eye-gaze When using audio information in a virtual tour guide, special care has to be taken not to influence eye-gaze behaviour with the text from an audio file. It was found that people, when presented with a visual scene and audio texts, tend to gaze at items on the scene that are semantically related to the audio text (Cooper, 1974). As our system takes measurements of where the eye gazes on paintings as a basis for a recommendation, special care has to be taken that this process is not influenced by gazes that are made as a result of the presented audio information. A user might for instance look for a long time at Cupid on painting sk-a-38882 because his name was announced in a text, and not because the user is interested in Cupid. Keeping in mind these processes is especially important for the introductory texts of a museum tour, as the measurements taken during that time course are used for the whole tour. From a theoretical point this influence of the explanations of a tour guide on eye-gaze behaviour is a problem. But from a practical point of view this is not a problem: a normal museum tour guide also influences gazing behaviour of museum visitors, as he tells stories of items on paintings. A recommender system that used eye-gaze Creating adaptive recommender systems that use implicit input from eye-gaze and give explicit audio output is possible. A good example of such a system is iTourist, which helps tourists in planning a city trip (Qvarfordt, 2004; Qvarfordt & Zhai, 2005). There are differences however between this system and our system. The most important one is scene complexity. The complexity of the visual scene is far greater on paintings than on tourist maps (which are abstractions of the real world, with for instance plain blocks representing houses). As our visual scene is more complex, it is interesting to find out whether people still focus on specific objects and how they like the interaction with the recommending tour guide. Another difference is that the iTourist system gives both explanation on items (e.g. what kind of art works can I see at the royal palace?) as well as recommendations for the users actions (e.g. walk past the museum when you walk from your hotel to the restaurant). In the virtual museum setting, the tour guide gives only explanations on items. Motivation for this is that it is a pilot study which can form the basis for more complex interaction settings. 2

See Appendix A for the paintings that are used in this experiment and throughout this paper

5

As a consequence of all of the previous research described in this section, we hypothesise that on paintings, users look at the items that they find interesting, just like they do in natural scene perception (Henderson & Hollingworth, 1999). And just as in the iTourist setting we hypothesise that they would like to receive more background information on these items.

The virtual museum guide The virtual museum guide gives a tour past paintings that are part of the online collection of the Rijksmuseum in the Netherlands. On each painting items can be identified of which the system can give anecdotic information. The items on which information can be given might be selected by art experts. The selected items from paintings might be persons, animals or objects that play a role in the story that the painting represents. The items on paintings do not have to be fully connected to be regarded as one item. For instance in the painting of Venus and Adonis (museum code sk-a-3888), all Satyrs were categorised as Satyrs, although they were placed in different regions of the paintings. Which part of an object is selected as item might differ from painting to painting, depending on the explanations that the website of the Rijksmuseum gives. If an explanation is about a human, the whole body could be selected as item, if it is about a group of people, all people of that group might be selected as one item (for example the Batavians in painting sk-a-4853). If the explanation is only about an object that belongs to a person, then only this item was selected (for example the hats in painting sk-a-3984). In order to be able to process this data about items on a painting and to make a coupling with user data, as derived from an eye tracker, each painting has to be represented in a digital way. The current museum guide represents paintings as a matrix in which each cell maps a cluster of pixels. The cell coordinates can be mapped on the original painting dimensions. If a cell has value one, this indicates that at the corresponding pixel cluster (a part of) item one is painted. For all locations on which there is no specific item visible, the corresponding cells have value zero. In paintings, the size of items often differs. If you look for instance at painting sk-a3888, the size of Venus (item one) is larger then the size of Cupid (item three). As a result, some items are represented a lot in the matrix, while others are only present in a small number of cells. As a consequence, purely statistical there is a higher chance that participants look at these items in comparison to others. It seems very plausible that other visual stimuli (for instance, colour, orientation and luminance of items on a painting) might also influence eye-gaze patterns to some extent. But up to what extent these graphical stimuli influence eye-gaze is unknown. As all of these graphical measures were mostly intended by the painters (why else would they make those items so obvious), and because it is also hypothesised that eye-gaze in natural-scene perception is mostly influenced by the interest of users (Henderson, 2003; Henderson & Hollingworth, 1999), no effort has been made to make the statistical selection choice for each item equal. During the museum tour, each user sees several paintings. With each painting, they first hear a general introduction text. During this initial phase and subsequent texts (and during periods without audio), the museum guide keeps track of eye-fixations. For each eyefixation, the program determines at which item the participant looks, correlating fixation position to matrix position. A counter for that specific item can then be updated using the fixation time as update factor.

6

After some time, the initial text has finished and a new text has to be offered to the user. In the standard setting, the user gets information about the item that he attended to most but that he has not received information about earlier in the tour. The tour guide can give texts on several items using this algorithm, being at first hand only constrained by the number of items that are present on a painting. As our virtual museum tour guide uses the coupling of eye-gaze with data about items on a painting to determine a recommendation, it is a contentdriven recommender system. The content it uses is the spatial relationship between eyefixation position and the items that are depicted there. In early testing of the virtual museum guide we noticed that people found it comfortable to receive audio information almost constantly during the tour. Periods of silence that took longer then an estimated three seconds made the participants nervous. As a consequence, the algorithm of the museum guide has to be made either in a way that it notices when an audio file has quit playing, or in a way that the audio files are all of about the same length. As the subset of items that is presented to a user differs from one user to another, special care has to be taken that texts do not refer to prior knowledge (for instance about other paintings or items). Audio files should also only mention information about the items itself. This is because of the influence audio has in directing eye-gaze (Cooper, 1974): audio information on other items might influence the eye-gaze, and thereby the measurements of the eye tracker and the recommendations of the tour guide.

The experiment Hypothesis As stated above, the museum guide uses eye-gaze as a measure of a user’s interest in items on a painting. To test whether it is correct that users are interested in this information, the following hypothesis was tested: “Do people rate a virtual museum tour better if the tour guide gives information about items they attended to most in comparison to a tour guide that gives information on items they attended to least?” The following sub hypotheses were also tested: - Do people attend to items that they receive audio information on (confirming the results of Cooper, 1974)? - Is there a correlation between a user’s background knowledge and education and his ratings for the explanations of the tour guide? Implementation of the virtual museum guide In order to test the hypotheses, an implementation of the virtual museum guide in a controlled setting was made. The guide gave a tour past nine virtual paintings of the Rijksmuseum. Effort was made to let the paintings be of similar categories, avoiding indirect influences of painting theme on experimental results. The categories were Roman and Greek mythology, Roman history and Christian mythology. On each painting ten items were identified of which the system could give more information. The items and appropriate corresponding audio texts were selected by the researchers, using information from the website of the Rijksmuseum3 and from a book on classical mythology (d'Hane-Scheltema, 2001). Making these selections of items ourselves, and not basing them on user studies, was motivated mostly by practical matters. But besides those, this selection by one expert is

3

See http://www.rijksmuseum.nl/aria/aria_catalogs/

7

comparable to the setting of a normal museum in which explanations and recommendations during a museum tour are also given by experts: the tour guides. On each painting, ten items were selected. In order to make the transformation of a painting to an array that represents the location of items, all items were indicated with colours on a painting (see the appendix for examples). These marked paintings were transformed into numerical arrays with dimensions 102*76, using the open source program ASCIIPicture4. In these arrays each array item represents a field of approximately 10*10 pixels on the painting. We used the pylink 2.4 libraries (SR Research Ltd., 2003) to define the tracking algorithm. Measurements during the whole time interval since the first presentation of the picture were used for calculating the most attended to items. For example, for calculating the 3rd selected text, information of the interval from zero to forty-three seconds is used. We only offered information on three items for each painting. To make the period of silence between the presentation of two audio files as short as possible, effort was put in making the audio files all of about the same length. The general texts that were played during the initial phase had a length between 18.8 and 19.9 seconds. The texts that the user received after that phase ranged from 8.3 to 10.4 seconds. The whole process of the calculation and offering of audio information is visualised in figure 1.

Initial phase

1st selective text

2nd selective text

3rd selective text

time

0 21 32 43 55 Figure 1: The outline of the tour explanations for each painting. In the yellow phases users hear texts. In the green phases the next text is determined (in practice this takes less then one second).

All texts were spoken in Dutch, because the participants were native Dutch speakers. The voice talent for the tour guide was a female native Dutch speaker. The texts of the initial phase did not mention special items on the painting, but gave information about the painter, year of construction, painting style, some specific fact about the painter, and the title of the painting. All texts were spoken in more or less the same format, to avoid influence on experimental results by the order of sentences within the text. Items that are also placed on the painting were only mentioned in the text of the initial phase if they were in the name of the painting, and if so, they were named in the last sentence. This way, the influence of audio semantics on scene perception (Cooper, 1974) was mostly avoided. The contents of the selected texts all differed, as the items also differed. Special care was taken that the audio text for each item told about the same level of interesting detail, as could be obtained from the website of the Rijksmuseum. Of course, as each item is different, each text is also different; and as each user has different interests, it is possible, and even desirable, that certain texts are more interesting for one user compared to another. This makes an adaptive museum tour also more unique for each user. The first painting the participants saw was always the same, painting number sk-a-70, and served as a warm-up trial. All participants received the same audio information with this painting: a general introduction text (comparable to the other introduction texts) of about 20 seconds, followed by three texts in a fixed order that told information about specific items of the painting. The items and corresponding audio texts of the warm-up trial painting were chosen in such a way that they told about broad lines and small details, so participants would 4

Downloaded from: http://studenten.freepage.de/cgi-bin/feets/freepage_ext/41030x030A/rewrite/meph/ascii/eng/asciipic.htm in November 2005

8

also focus on small items and not only the obvious items of a painting. This indicated to the user that all items on the screen might be of interest. The results and review of the warm-up trial were not used in further analysis. The painting was purely used so participants could get used to the voice of the tour guide and the setting of the experiment. After the first painting, eight other paintings were displayed for the participant, in a random order. Experimental setup In the experiment 22 participants took the tour. Only data of 20 participants was used for statistical analyses, as we were unable to calibrate the eye tracker for two participants. All participants were students of the University of Groningen. There were 14 males and 6 females. Ages ranged between 18 and 26 with a mean of 21.85 and standard deviation of 2.69. The artificial intelligence students all volunteered for the experiment, the other students received one course credit. All participants had normal or corrected to normal vision. However, wearing glasses was an exclusion criterion. During the experiment measurements were done using a head-mounted eye tracker (eyelink 1 of SR Research). The experimental room was arranged according to guidelines of SR Research Ltd. (1996) with a small 60 watt light bulb for optimal lighting conditions and using a 17 inch screen for displaying the paintings. Loudspeakers were placed next to the monitor, and set at a fixed volume. All paintings were displayed as large as possible to the user, within a frame of 1024*768 pixels, with a black background (see the appendix for examples). The audio texts that were presented to the participants were selected using two conditions. In the “maximum condition” participants received information about the objects that they attended to most. In the “minimum condition” participants received information about the objects that they attended to least (or sometimes not at all). If two or more items were attended to equally often (for instance, both were not attended to), the algorithm randomly chose one of those items for giving information on. Each subject randomly watched a painting in the maximum or minimum condition. As a constraint, each participant watched four paintings in the minimum condition and four paintings in the maximum condition. Procedure All participants first received a brief explanation of the experiment, explaining the global outlines of three phases: calibrating the eye tracker, taking the museum tour and filling out a questionnaire. After this, the eye tracker was placed on the head, and calibrated using the training algorithm of SR Research Ltd. (1996). We used the calibration algorithm that was optimal for viewing a whole screen. After this phase, the participants got a last explanation of the tour. They were asked to sit as motionless as possible (so the eye tracker did not make much incorrect measurements), only using their eyes to watch the painting. The eye tracker could compensate for slight movements by the participants. They were told that they could look everywhere they wanted, as long as they looked at the painting. Participants were asked to rate the information of the tour guide after she had finished her explanation for a painting (55 seconds after the start of the display of the painting). A mark between 1 and 10 had to be given, with 10 indicating that it was very good, and 1 that it was very poor. When the participants had seen four or five paintings, the experiment was interrupted for a short period in which the validation program of the eye tracker was run. If validation failed, the eye tracker was adjusted and both the calibration and validation program were run to re-adjust the measurements of the program. All was done as fast as possible in order that the process did not interfere with the user’s experience of the museum tour. After this validation, the tour was continued.

9

After the tour had finished, participants received a questionnaire5. After participants had filled out the questionnaire, they got a debriefing, explaining the details of the experiment. Results and discussion of results The mean grades that the participants gave to the explanations of the tour guide for each painting are given in table 1. As can be seen, the explanations of items from paintings that were perceived in the maximum condition received a slightly higher grade. A paired t-test analyses, comparing the mean grades in the maximum and minimum condition, results in: t = 1.24, df = 19, p = 0.23. So although a positive trend can be perceived, there is not yet enough statistical evidence for confirming our main hypothesis that users like to receive information on items that they attend to most. Painting

Maximum condition Minimum condition Mean Std. dev Mean Std. Dev 6.708 1.033 rp-t-1965-290 7.313 1.280 6.714 0.951 sk-a-3746 6.846 1.125 6.714 0.611 sk-a-3888 6.917 0.492 7.357 0.945 sk-a-3984 7.077 0.732 7.182 1.437 sk-a-4690 7.278 0.507 7.429 0.732 sk-a-4853 7.192 0.902 7.115 1.121 sk-a-613 7.143 0.690 6.833 0.968 sk-c-1458 7.136 0.839 6.975 0.997 Means 7.106 0.850 Table 1: Mean grades given to the explanations of the tour guide.

There are a few explanations possible why the results can not confirm our hypothesis. The current experimental setting did not use a fixed effect for the manipulation of the testing conditions. Although each user watched four paintings in the maximum and four in the minimum condition, there was no equal distribution of the testing conditions over the paintings. Painting rp-t-1965-290 for instance was presented in the maximum condition to eight participants, and in the minimum condition to 12 participants. Given this unbalanced nature of the design, it is difficult to analyse if any effect found in the grades was caused by the painting itself or by the given information. A second possible explanation lies in the way in which the audio files are selected. In cases were people only look around and do not focus much on specific items, they perhaps are not interested in a specific item at all. There is therefore not much difference in the amount of fixation time between items that were attended most and other items, they were all almost equally attended to and perhaps all almost equally interesting. A significant difference between these two testing conditions might therefore be hard to find.6 In figure 2 the box-and-whisker plot of the means of the cumulative fixation times for items that the user received anecdotic information on are plotted. The cumulative fixation time was calculated in the time interval between the start of each audio file and the start of the next audio file (this interval lasted 11 seconds). The means were calculated for each subject, using the fixation times in the maximum and minimum condition, for each audio-presentation moment and each painting (so each mean is calculated using three values). It can be seen that the distribution of the data is slightly skewed, which is caused by the fixed lower boundary of zero seconds for the fixation time. Table 2 therefore gives the p5 6

The questionnaire can be found in Appendix B See the section “future work” for a more detailed analyses of this problem

10

values (calculated with a paired t-test) of both the mean fixation times and the logarithm of those means. There is a significant difference between the fixation times in the maximum and minimum condition during the total experiment and for each audio presentation moment, both for the real means and the logs of those means. This indicates that subjects fixated more on items that they already fixated on most of the time (the maximum condition) in comparison to items that they fixated relatively less on (the minimum condition). Our sub hypothesis that people look at items that they get information on is therefore rejected. If users already fixated much on an item, and they receive audio information about that item, they keep on attending to that item. If they receive audio information on an item that they did not fixate on much, they do not attend to that item much. This effect is in contrast with the effect that was found by Cooper (1974).

Figure 2: Box-and-whisker plot showing the mean cumulative fixation times of participants on items that they received information on during the presentation of the information (periods of 11 seconds). Moment of selected Mean max condition Mean min condition p-value text presentation (ms) (ms) Means Log(Means) 1st <0.001 <0.001 3091 827 nd 2 <0.001 <0.001 2808 775 3rd <0.001 <0.001 2370 1177 All texts <0.001 <0.001 2756 926 Table 2: Mean cumulative fixation times for each moment that users received a “selective text”, and the p-value for the difference between the means and the difference between the log means, calculated using a paired t-test.

An explanation for this effect might be the difference between stimulus driven and interest driven eye-gaze. In the experiments that were conducted by Cooper, participants watched a relatively simple visual scene (with simple drawn animals for instance) on which there was not much semantic coherence between the items (the items were not interacting).

11

Therefore, it is possible that in Coopers setting there were not much high-level processes that influenced the eye-gaze, the eye-gaze was mostly driven by (visual and) audio stimuli. In contrast, in the virtual museum guide there is a complex visual scene with a lot of semantic coherence on which high-level scene processes could be of more influence, and even blocking the stimuli driven processes (as described in Henderson & Hollingworth, 1999). In table 3 the mean grades given by students who took at least one year of classes in classical languages (ancient Greek or Latin) are given. The results are visualised in figure 3 and the p-values of several conditions are given in table 4. It can be seen that students who took classes in classics give on average a lower grade for the explanations of the museum guide than students that never took classes in classics. Students who took classes in classics give higher grades in the minimum condition than in the maximum condition. The others give higher grades in the maximum condition. All categories differ significantly from each other, except for the results in the maximum and minimum condition of participants that did not take classes in classics. No classes in classics Taken classes in classics (11 participants) (9 participants) Max condition 7.0778 (1.3100) 6.4423 (1.4660) Min condition 6.9987 (1.3237) 6.8357 (1.5117) Table 3: The average tour rates: mean values and standard deviations for several conditions.

Figure 3: The average tour ratings given by students who took or did not take classes in classics. Bars indicate standard deviations.

The results of tables 3 and 4 indicate that having taken classes in classical languages significantly influences the rating of the virtual museum tour. The general point to be derived from these results however is that previous knowledge and educational history can be of significant influence on the evaluation of a digital tour guide, confirming our second sub

12

hypothesis. This general fact can form a constraint on the interaction with a recommender system. Taken classes in classics Max condition Min condition

No classes in classics Max condition Min condition

Taken classes in Max condition 0.03802 0.04180 classics Min condition 0.03802 0.00023 No classes in Max condition 0.04180 0.00023 classics Min condition 0.00950 0.00003 0.20100 Table 4: The p-values between several conditions.

0.00950 0.00003 0.20100

In table 4 can be seen that only one correlation shows no significant effect. If you have not taken classes in classical languages you do not give significant different grades to paintings in the maximum and minimum condition. Although we can not give an affirmed explanation for it, we have a hypothesis that might explain these results. People who took classes in classical languages have discussed a lot of mythological stories during those classes. They know which characters and attributes are interesting and contribute to a mythological story. Other subjects perhaps do not have this background and a lot of information, for instance about mythological Gods, might be new for them. They therefore do not bother if they receive information about items they looked at or other items: they are all equally interesting. So this result also grounds the theory that a user’s interests and educational background are of influence on a museum tour. Besides these statistical results a verbal report of one of the participants has to be noted as a result. This participant thought it was strange that he did not receive information on “John the Baptist” with painting sk-a-3746. The user in question watched this painting in the maximum condition, and did not attend to this item, although he was interested, and therefore did not receive audio information on that item.

General discussion Objective of this study was to design a virtual museum guide that could give users a personalized and adaptive tour. In order to achieve this, we had to infer a user’s interest in a non-obtrusive manner, using eye-gaze. Creating such a system was successful and as the results show, we could derive a user’s interest. The data of the experiment suggest that our virtual tour guide system works quite well. It is very likely that users like to receive audio information about items that they attend to much on a painting. Most astonishing perhaps is the result that in the current setting of the virtual museum tour the effect that Cooper (1974) described does not seem to apply. Users keep on looking at items that they already looked at, ignoring audio information about other items. As a consequence, this effect as described by Cooper, does not have to be taken into account within the setting of a virtual museum tour guide as strict as we did for this experiment. In the current setting we referred in audio texts only to one item at a time, and only mentioned the title of a painting (in which often names of items were stated) at the end of the text in the initial phase. In the future it is less critical if names of other items are mentioned within the text of an item. However, since the results of Cooper have been reproduced several times, it would not be wise to totally ignore this effect. The correlation between whether a participant took classes in classical languages, the maximum or minimum condition and user rating of the tour guide shows that background 13

knowledge can be of strong influence on the appraisal of a virtual museum tour guide. This evidence pleads for personal and adaptive tours, as each individual has different interests. Students that followed classical languages (and have some background knowledge about some of the items that are depicted on the paintings) know what to look at on a painting and appreciate receiving background knowledge on those items. To other users it does not matter what information they receive. Another important conclusion that can be drawn from these results is that virtual museum tours have to be very flexible and can not be based on a predefined symbolic set of user characteristics. While creating our questionnaire, it was doubted whether having taken classes in classical languages would have an effect on the grades that participants gave the explanations of the tour guide, as it seemed to be only a small factor. But in the end, it turned out to have one of the most significant effects. As it was not foreseen that this small factor could influence the experience on this scale, it indicates that it will be difficult to define a large set of questions or interest parameters for users when they are shown a larger and more diverse art collection. This information can not be gathered in a predefined user-profile. Therefore, information has to be gathered implicit and the tour guide must learn during the tour for each user. Finally we should address the problem that one of the participants indicated about not receiving information about John the Baptist (painting sk-a-3746). The problem is that the painter painted John the Baptist in a gray area of the painting on purpose. He did not want John to be the centre of attention of the painting. As a consequence, the eyes are not drawn to this section very often. In future work a solution has to be found such that possibly important features of the painting, even those that are not very recognizable, are still mentioned by the tour guide and therefore noticed by the visitor.

Future work It is clear from the results and conclusions of this experiment that there is a lot of future work that can be conducted on the virtual museum tour guide system. As our hypothesis, that people want to receive information about items that they attend to most, is not confirmed, but indicated as leading in the right direction, more effort has to be made to test the hypothesis in other experiments. One option for a better experiment is one in which the testing conditions are in a more fixed order. In the current experiment the order of painting presentation as well as the division of maximum and minimum conditions was random, although all participants saw four paintings in the maximum and four in the minimum condition. As a consequence some paintings were presented mostly in a certain condition, making it difficult to make an analysis over all items. Another way of getting more certainty whether the hypothesis is correct is by testing the museum guide on a larger set of paintings. This limits the possibility that people rate certain paintings higher then others, because they like the painting style (or some other unavoidable feature) more. In those cases, they will be influenced by this unavoidable factor, but the paintings that show those characteristics can then be excluded from further analysis. For this Bachelor Project, extending the set of paintings was to large an effort, as selecting the paintings and the items on the painting (including the audio texts) took quite some manual labor. A hypothesis that is currently made by our research group is that people are interested in information about items that they gaze at most, but that they also gaze at for a longer time period. The current algorithm only selects items that were attended to most by the user, for

14

giving tour guide information. However, there is a difference between users that gaze 2.3 seconds at the most attended items and 2 seconds at other items, and users that gaze 5 seconds at the most attended item and 2 seconds at other items. Of the latter user could be said that he is more interested in the item then the former, who had about the same interest in several items. We think that this will reflect in the grades they give the tour guide. In the later case there is a clearer difference between offered information in the maximum and minimum condition. This indicates that the information selection algorithm must be more specified: only items that have been gazed at for a time course longer then a certain threshold must be presented by the tour guide to the user. Using this modification, we think that the effect that we addressed in the main hypothesis of this experiment might be present with statistical significance, while the effects that were shown in figures 2 and 3 are still present. In order to conduct a good experiment, the right threshold has to be determined. As told in the results section, one of the participants of the experiment indicated that he was interested in a semantically important item that is not visually very recognizable (John the Baptist on painting sk-a-3746). One way or the other, the tour guide system has to anticipate on this user interest that can not be derived directly from eye-gaze. There are several ways in which this problem could be solved, of which I mention two. The first option is expanding the interaction mechanisms. A system could be created that does not only focus on eye-gaze, but also on other input possibilities. Sound information (as used in Kaur et al., 2003 ) could be used to indicate a user’s interest. For instance if he exclaims “but where is John the Baptist?” It has been shown by others that systems that use several input mechanisms, so called multimodal interaction systems, have a lot of advantages. Multimodal interaction is universally strongly preferred by users, and makes interaction with a computer system more flexible, as users can switch between different modes of input (Oviatt, 1999). Disadvantages of this method would be however that it uses explicit input from the user, while our goal is to create a system that makes recommendations using implicit input. As this form for deriving user interest is not invasive however, it still is usable and the advantages possibly weigh up to the disadvantages.7 A better option however could be to use a spreading-activation based mechanism (Collins & Loftus, 1975; Van Maanen & Van Rijn, 2006). An example of a recommender system that uses such a system is the InterestMap system (Liu & Maes, 2005). In the InterestMap system, a network of persons and their interests is created, using knowledge of social network profiles. If you identify a person node, you can also identify his interests. If you identify an interest node, you can also derive all related interests and all persons that are interested in this item. A spreading-activation based system that is used in the virtual museum tour guide setting could be a semantic network that contains all items, paintings and painting related information (e.g. painter, style, and technique) as nodes. The connection strength between the nodes could be indicated by the researcher or could be trained by studies of users. Once the connections have been trained for some period, a coupling with the eyetracker could be made. Looking at items can ‘activate’ a node of the network that represents the current item. The amount of time that you have fixated on an item might determine the degree of activation with which the node of the item has to be updated. The activated node could spread its activation across the other nodes in the network. Once the virtual tour guide has finished telling a story about an item, it could tell a story about another item that is most activated by the network at that specific moment. This item could be an item that the user has extensively looked at, and therefore has been directly activated in the network. But it could 7

Although speech recognition has enough challenges by itself, making it difficult to directly implement this

15

also be an item that the user has not looked much at, but that has been activated by the spreading-activation of the network. This way, the semantic network could be used to select the stories the tour guide explains within one painting. For instance, if one looks at Venus on painting sk-a-3888, then perhaps the user is also interested in the turtledoves (as a symbol for Venus) and in Cupid (the son of Venus). Also, as they are implicitly interested in Cupid on painting sk-a-3888, they might be interested in painting sk-a-3984, where another Cupid like boy is painted. As Cupid is also related with Diana, they might also be interested in her. This way, the whole tour gets a semantic coherence. This idea is visualized in figure 4. The amount of red in the nodes indicates the amount of activation of an item.

Figure 4: Looking at the red item activates other items and paintings using the semantic network, both within (Venus and Turtledoves) and outside (Diana) of the painting. The degree of redness indicates the activation level of each node.

In a system that uses a semantic network and a spreading-activation based mechanism, another interesting cognitive process might be relatively easy implemented: the change of a user’s interests. In the currently used museum guide, the model selects items based on all eyefixations during the whole tour. If you look at Venus a lot in the beginning of the presentation of the painting, but look at Cupid more in the end, but slightly shorter then at Venus, you get information about Venus, although you are more interested in Cupid at that moment. The implementation can be made using a decay factor. The decay factor lowers the activation of a node, if it has not been activated recently. So in the case of Venus and Cupid, the activation of the Venus node decays when you do not look at her any more. In the meanwhile, the activation of the Cupid node is raised, the more you look at it. In the end, the Cupid node will be most activated, and information on Cupid can be given by the system. This way it will be likely that you receive information on items that you are interested in most at that specific moment, in comparison to receiving information on items that you were interested in, but not any more. The ultimate virtual museum guide might be a system that uses a semantic network with a spreading activation network and a decay factor, within a multimodal interaction setting. But most of all, this ultimate system gives users just a good time and a unique experience.

16

Acknowledgments This Bachelor Project took place within the larger OPTIMA project by Leendert van Maanen. OPTIMA is part of the I2RP-project, which in turn is part of ToKeN (TOwards Knowledge Engineering Netherlands). ToKeN is a NWO-funded research programme directed at crossover between knowledge engineers and industry, specifically in the fields of law, health care, and cultural heritage. For this project, art works of the collection of the Rijksmuseum were used. You can take a look at the digitalised art-works yourself on the website of the Rijksmuseum: www.rijksmuseum.nl The author would like to thank Merel Oppelaar for providing her voice to the virtual museum guide. He would also like to thank all participants who volunteered for the experiment.

References Balabanovic, M., & Shoham, Y. (1997). Fab: Content-based, collaborative recommendation. Communications of the Acm, 40(3), 66-72. Cassell, J. (2000). Embodied conversational interface agents. Communications of the Acm, 43(4), 70-78. Collins, A. M., & Loftus, E. F. (1975). Spreading activation theory of semantic processing. Psychological Review, 82(6), 407-428. Cooper, R. M. (1974). Control of eye fixation by meaning of spoken language - new methodology for real-time investigation of speech perception, memory, and language processing. Cognitive Psychology, 6(1), 84-107. d'Hane-Scheltema, M. (2001). Ovidius – metamorphosen (8th ed.). Amsterdam: AthenaeumPolak & Van Gennep. Goldberg, J. H., & Kotval, X. P. (1999). Computer interface evaluation using eye movements: Methods and constructs. International Journal of Industrial Ergonomics, 24(6), 631645. Henderson, J. M. (2003). Human gaze control during real-world scene perception. Trends in Cognitive Sciences, 7(11), 498-504. Henderson, J. M., & Hollingworth, A. (1999). High-level scene perception. Annual Review of Psychology, 50, 243-271. Henderson, J. M., Weeks, P. A., & Hollingworth, A. (1999). The effects of semantic consistency on eye movements during complex scene viewing. Journal of Experimental Psychology-Human Perception and Performance, 25(1), 210-228. Kaur, M., Tremaine, M., Huang, N., Wilder, J., Gacovski, Z., Flippo, F., & Mantravadi, C. S. (2003 ). Where is "It"? Event synchronization in gaze-speech input systems Paper presented at the 5th international conference on Multimodal interfaces Vancouver, British Columbia, Canada Liu, H., & Maes, P. (2005). Interestmap: Harvesting social network profiles for recommendations. Paper presented at the Workshop: Beyond Personalization 2005, part of the International Conference on Intelligent User Interfaces, San Diego, California, USA Loftus, G. R., & Mackworth, N. H. (1978). Cognitive determinants of fixation location during picture viewing. Journal of Experimental Psychology-Human Perception and Performance, 4(4), 565-572.

17

Oviatt, S. (1999). Ten myths of multimodal interaction. Communications of the Acm, 42(11), 74-81. Qvarfordt, P. (2004). Eyes on multimodal interaction. Linköping University, Linköping. Qvarfordt, P., & Zhai, S. (2005). Conversing with the user based on eye-gaze patterns Paper presented at the SIGCHI conference on Human factors in computing systems Portland, Oregon, USA SR Research Ltd. (1996). Eyelink system documentation. SR Research Ltd. (2003). Programming eyelink experiments in windows (python). Beta version 0.9. Van Maanen, L., & Van Rijn, H. D. (2006). An accumulator model account of semantic interference in memory retrieval. Paper presented at the Seventh International Conference on Cognitive Modeling, Trieste, Italy. Van Maanen, L., Van Rijn, H. D., & Schomaker, L. R. B. (2005). The virtual tour guide: A guide for intelligent multimedia information retrieval (Poster presented at ToKeN Workshop). Eindhoven. Velichkovsky, B. M., Dornhoefer, S. M., Pannasch, S., & Unema, P. J. A. (2000). Visual fixations and level of attentional processing Paper presented at the 2000 symposium on Eye tracking research & applications Palm Beach Gardens, Florida, United States Viviani, P. (1990). Eye movements and their role in visual and cognitive processes. In E. Kowler (Ed.), Eye movements and their role in visual and cognitive processes. Amsterdam: Elsevier Science. Waern, A. (2004). User involvement in automatic filtering: An experimental study. User Modeling and User-Adapted Interaction, 14(2-3), 201-237.

18

Appendix The appendix is divided in two sections: Appendix A: an overview of the paintings of the Rijksmuseum collection that were used in the current virtual museum tour guide. With each painting the following information is given: - Title - Painter - Painting code Rijksmuseum - Year of production - Technique - Dimensions of the original Painting - Picture of the painting - Picture with the items, that the tour guide could use for its explanations, marked - Transcription of the audio texts that the tour guide gave with each painting (in Dutch)

Appendix B: the questionnaire that was given to the participants of the experiment

19

Appendix A: painting information

20

Title: Latona and the Lycian Peasants Painter: Jan Brueghel I Painting code Rijksmuseum: SK-A-70 Year: c. 1605 Technique: Oil on panel Dimensions original: 37 x 56 cm

N.B. This painting was the introductory painting. No item ranges were selected, the texts were all presented in a predefined order. Texts: 0. Dit schilderij is gemaakt door Jan Brueghel en dateert uit de periode van 1595 tot 1610. Jan Brueghel was een aantal jaren hofschilder van het Habsburgse rijk. Hij heeft in totaal meer dan 450 schilderijen gemaakt. Dit werk is gemaakt met olieverf op paneel. De titel van het schilderij is “Latona en de Lycische boeren”. 1. De twee kinderen vooraan zijn de Romeinse goden tweeling Apollo en Diana. Apollo is vooral bekend als God van de muziek en poëzie en Diana als Godin van de jacht. 2. Vooraan ziet u enkele boeren. Deze maken het water opzettelijk ondrinkbaar, opgehitst door de godin Juno. Zij was namelijk jaloers dat haar man kinderen bij een andere vrouw verwekt had. 3. Op het schilderij worden boeren in kikkers veranderd. Enkele kikkers en boeren met kikkerhoofden zitten in de vijver. Verhalen over dergelijke metamorphosen waren populair in de tijd van de schilder. 4. De vrouw vooraan is Latona. Zij is een dochter van de Titanen. De titanen zijn kinderen van Gaia, moeder aarde, en Ouranos, de god van de hemel.

21

Title: Seascape with Aeneas on Delos Painter: Claude Lorrain Painting code Rijksmuseum: RP-T-1965-290 Year: c. 1670-72 Technique: Ink and chalk on paper/Pen in brown ink, brush in brown and grey, highlighted with white and grey Dimensions original: 16 x 22 cm

Texts: 0

De volgende tekening is gemaakt door de Franse schilder Claude Lorrain en dateert uit de periode rond 1670. Voor dit werk zijn verf, inkt en krijt gebruikt. Het is één van de vele voorstudies voor een schilderij dat nu in een museum in Londen hangt. De titel van de tekening is “Kustlandschap met Aeneas op Delos” 1 De man in harnas is Aeneas. Aeneas ontvluchtte Troje na de Trojaanse oorlog. Na jaren van rondzwerven belandde hij in Italië, waar hij de stamvader werd van het Romeinse volk. 2 De twee bomen markeren de geboorteplaats van de Griekse Godentweeling Apollo en Diana. Apollo was de God van de muziek en poëzie en Diana was de Godin van de jacht. 3 De tempel achterin is een orakel van de God Apollo. De tempel lijkt op het Pantheon, een van de bekendste tempels in Rome. Hiermee staat de tempel symbool voor Aeneas als stamvader van de Romeinen. 4 Het hoge gebouw links heeft de schilder geplaatst om diepte te creeëren.. Het werkt als een coulisse op het toneel, een ‘repoussoir’. Het lijkt de rest van de tekening naar achteren te duwen. 5 De rechtse man is Anius, koning van het eiland Delos. Zijn dochters konden alles wat ze aanraakten in eten veranderen. Bij een ontvoering door de Grieken werden zij door de God Bachhus in duiven veranderd 6 De derde man vanaf links is Anchises. Hij verwekte een kind bij Venus, de godin van de liefde en de vruchtbaarheid. Dit kind werd stamvader van de Romeinen, die daardoor van de Goden afstammen. 7 Links staat Ascanius. Hij was de zoon van de Trojaanse held Aeneas en volgde zijn vader op als heerser van het Italiaanse Latium en Alba, wat nabij Rome lag. 8 De lucht neemt een grote oppervlakte van de tekening in. Deze tekening is een voorstudie voor een schilderij. Hiervoor was de tekenaar vooral in de compositie geïnteresseerd en niet in allerlei details. 9 Deze tekening is een voorstudie. Hierin neemt de zee op de achtergrond, maar een klein deel in en zijn er nauwelijks details te herkennen. In het uiteindelijke schilderij is meer zee opgenomen. 10 Het landschap is opgebouwd uit transparante grijze en bruine tinten, met accenten in dekkende witte of grijze verf. Door het afwisselende gebruik van licht en donker heeft de schilder een ruimtelijk effect gecreëerd.

22

Title: Consul Titus Manlius Torquatus Beheading His Son Painter: Ferdinand Bol Painting code Rijksmuseum: SK-A-613 Year: c. 1661-63 Technique: Oil on canvas Dimensions original: 218 x 242 cm

Texts: 0

Het volgende schilderij is gemaakt door Ferdinand Bol en dateert uit de periode van 1650 tot 1669. Het is een werk van olieverf op doek en was bedoeld om boven een schoorsteenmantel van het Prinsenhof in Amsterdam te hangen. De titel van het schilderij is “consul Titus Manlius Torquátus laat zijn zoon onthoofden 1 Op de verhoging zit consul Titus Manlius Torquatus. In de Romeinse Republiek werden jaarlijks twee consuls aangesteld. Zij voerden het leger aan en waren voorzitter van de Senaat. 2 Vooraan ligt het lichaam van Titus Manlius. Hij was slaags geraakt met de vijand. Dit was in strijd met de wet en daarom heeft zijn vader bevolen om hem te onthoofden. 3 Op de achtergrond ziet u vele gebouwen die uit de stad Rome lijken te komen. Deze zijn opgenomen, omdat het Romeinse bestuur een voorbeeld was voor bestuurders uit de 17de eeuw. 4 Op het vaandel links ziet u de letters “SPQR”. Dit is een afkorting van het Latijnse “Senatus Populusque Romanus”: de senaat en het volk van Rome. 5 De ronde toren links lijkt op de Engelenburcht uit Rome. Deze burcht is pas vele eeuwen na de Romeinse tijd gebouwd, maar is door de schilder opgenomen om te benadrukken dat de scène in Rome speelt. 6 Het Romeinse volk rechts onderin kijkt geschrokken naar het vonnis dat voltrokken is. Net als de toeschouwers van dit schilderij, moeten zij leren dat ze moeten luisteren naar het gezag. 7 De beul staat in het midden en laat het hoofd zien van een man die de wet overtreden heeft. Hiermee laat hij, en de schilder, zien wat de gevolgen zijn als er niet geluisterd wordt naar het gezag. 8 In schilderijen wordt vaak gebruik gemaakt van perspectief. Het paard met soldaat is in verkort perspectief geschilderd. Dit was bijzonder voor werk uit de 17de eeuw . 9 Een officier is geroyeerd uit het leger, omdat hij niet had geluisterd naar een bevel van het hoofd van het Romeinse Rijk. Hij is daarom ontdaan van zijn harnas, dat u vooraan ziet liggen. 10 Vooraan staat een officier uit het Romeinse leger. Hij is te herkennen aan verschillende attributen, zoals de rode cape, het geschubde harnas en de grote helm met de veer.

23

Title: Landscape with the Ministry of John the Baptist Painter: Abraham Bloemaert Painting code Rijksmuseum: SK-A-3746 Year: c. 1600 Technique: Oil on canvas Dimensions original: 139 x 188 cm

Texts: 0

Het volgende schilderij is gemaakt door Abraham Bloemaert en dateert uit de periode van 1590 tot 1610. Bloemaert verwierf bekendheid en aanzien dankzij zijn schilderijen met mythologische en religieuze onderwerpen. Dit werk is gemaakt met olieverf op doek. De titel van het schilderij is “Landschap met de prediking van Johannes de Doper.” 1 In de schaduw staat Johannes de Doper. Hij reed langs de rivier de Jordaan door Palestina. Ondertussen doopte hij mensen en preekte over de komst van Christus. De naam Johannes staat voor “God is genadig”. 2 Een grote groep mensen is naar deze plek gekomen om te luisteren naar een preek van Johannes de Doper. Velen luisteren echter niet en zijn meer met zichzelf bezig. 3 Rond 1600 was het zeer ongebruikelijk om gewone mensen als hoofdthema van een schilderij te nemen. Desondanks heeft de schilder er enkelen afgebeeld, zoals de vrouw die borstvoeding geeft. 4 De schilder gaf in zijn schilderij gewone mensen zeer fantasierijke en kleurrijke kleding. Hij vond dit passen bij inwoners van het heilige land. De rode drummer is het fraaist versierd. 5 De halfnaakte mannen vooraan luisteren naar de preek van Johannes de Doper. Zij waren in de minderheid, de meeste van de aanwezige mensen luisterden totaal niet. 6 Bloemaert schilderde in de stijl van het maniërisme. Een kenmerk van deze stijl is dat vaak halfnaakte mensen in de meest ingewikkelde poses staan afgebeeld. Een voorbeeld is de man in de boom. 7 Bloemaert schilderde volgens de stijl van het maniërisme. Hij schildert daarom veel mannen in naakte poses, zodat hij zijn kennis van menselijke anatomie kan demonstreren. 8 Het tafereel vindt plaat in een fantasierijk landschap. De weelderige kleuren en de rusteloosheid hiervan zijn kenmerken van het maniërisme, de stijl van de schilder. 9 Het landschapheeft 2 vergezichten die de kijker om aandacht vragen. De berg rechts is er een van. De aanwezigheid hiervan zorgt voor rusteloosheid, een kenmerk van de maniëristische schilderstijl. 10 De groepjes mensen nemen een groot deel van het schilderij in, maar zijn niet het hoofdthema. Dit werd pas in de 17de eeuw gebruikelijk. Hier hoort dat nog een bijbels verhaal te zijn.

24

Title: Venus and Adonis Painter: Bartholomeus Spranger Painting code Rijksmuseum: SK-A-3888 Year: c. 1586-07 Technique: Oil on panel Dimensions original: 135 x 109 cm

Texts: 0

Het volgende schilderij is gemaakt door Bartholomeus Spranger en dateert uit de periode van 1585 tot 1590. Spranger was jarenlang hofschilder van de Paus en is één van de belangrijkste vertegenwoordigers van de Haarlemse academie. Dit werk is gemaakt met olieverf op paneel. De titel van het schilderij is “Venus en Adonis” 1 Op de voorgrond ziet u Venus, de godin van de liefde en vruchtbaarheid. Zij klampt zich vast aan haar minnaar omdat zij voorvoelt dat hij niet terug zal keren van de jacht. 2 Voorin ziet u Adonis die afscheid neemt van zijn geliefde. De moeder van Adonis, myrrha, was in een strijd met de Goden veranderd in een mirreboom. Uit de bast van deze boom is Adonis geboren. 3 Links ziet u Cupido die de twee geliefden uitzwaait. Cupido is de god van de vleeselijke liefde en is het hulpje, en volgens sommigen zelfs de zoon van de liefdesgodin Venus. 4 In het landschap op de achtergrond verpozen zich nymphen, vrouwelijke godheden die van lager rang zijn dan de bekende olympische goden. Nymphen vertegenwoordigen de krachten van de natuur. 5 Op het schilderij staan diverse saters, halfgoden die de wilde natuur vertegenwoordigen. Zij hebben menselijke lichamen maar zijn verder te herkennen aan hun bokkenpoten, geitenstaarten en hoorns 6 De herder bij de schapen is Pluto. Pluto is een broer van de Romeinse oppergod Jupiter en is de God van de onderwereld. Pluto staat in dit schilderij symbool voor een naderende dood. 7 De schilder gebruikte veel symboliek in dit schilderij om thema’s extra te benadrukken. Zo staan de twee tortelduiven rechts onderin symbool voor de liefde. 8 Deze scène speelt zich af in een fantasielandschap. Een van de elementen die bijdragen aan die fantasierijke sfeer is het onrealistische kasteel op de achtergrond. 9 Om thema’s extra te benadrukken gebruikte de schilder veel symboliek in dit schilderij. Zo zijn de twee honden symbolisch, voor de naderende jacht 10 Om het landschap fantasierijker te maken heeft de schilder er diverse dieren op geplaatst. Zoals de schapen in de weide op de achtergrond, en kikkers en watervogels rond de vijver.

25

Title: The Sacrifice of Iphigenia Painter: Jan Havicksz. Steen Painting code Rijksmuseum: SK-A-3984 Year: 1671 Technique: Oil on canvas Dimensions original: 135 x 173 cm

Texts: 0. Het volgende schilderij is gemaakt door Jan Steen en dateert uit de periode rond 1671. Jan Steen werd vooral bekend om zijn genrestukken met vrolijke gezelschappen. Maar ook met portretten en schilderijen over mythische verhalen. Dit werk is gemaakt met olieverf op doek. De titel van het schilderij is “Het offer van Iphigeneia”. 1. In het midden ziet u Iphigeneia. Omdat het oorlog is zal zij worden geofferd aan de goden om hen gunstig te stemmen. Het offeren van mensen, dieren en voorwerpen was zeer gebruikelijk in de Griekse oudheid. 2. De afgebeelde koning is Agamemnon uit Griekenland. Hij wil Troje belegeren, maar telkens als hij wil uitvaren is het windstil. Om de boze godin Artemis gunstig te stemmen moet hij zijn dochter offeren. 3. Bij dit geschilderde offer vallen geen doden. Als door een wonder verandert het menselijke slachtoffer, zonder dat iemand het merkt, in een hert. Hierdoor komt alleen het hert om het leven. 4. Boven in de rook, zit Artemis, de Godin van de jacht. Ze is herkenbaar aan de maansikkel op haar hoofd en de pijl en boog. Artemis werd ook vereenzelvigd met de maangodin Selene. 5. Een jongetje loopt bedroefd weg op het schilderij. Het is amor, de god van de vleselijke liefde. Hij is herkenbaar aan zijn pijl en boog waarmee hij mensen verliefd kon maken. 6. De figuren rechts hebben een exotisch tintje vanwege hun verschillende hoofddeksels. De tulband, de lauwerkrans, een soort bisschopsmijter en een variatie op een Romeinse helm zijn te zien. 7. Jan Steen hield zich niet aan de regels van de historische schilderkunst. Om het verhaal voor zijn tijdgenoten herkenbaar te maken, gaf hij de meeste personen geen oud Griekse, maar 17de eeuwse kleding. 8. De geknielde vrouw is met veel zorg afgebeeld, met details als de glanzende kleding en de voeten die vies zijn van het lopen. Het licht van het vuur rond haar afgewende hoofd geeft haar zelfs iets mysterieus. 9. De schilder beeldt het moment af vlak voordat een offer wordt gebracht. De beul staat al klaar om zijn slachtoffer te doden, het mes glinstert in zijn hand en hij kijkt bloeddorstig naar het offer. 10. Rechts bovenin is een vrouw aan het bidden. De gevouwen handen zijn een christelijk gebaar wat niet hoort bij de Griekse oudheid. Door dit wel te gebruiken is het verhaal herkenbaarder voor de tijdgenoten van de schilder.

26

Title: The Continence of Scipio Painter: Karel van Mander Painting code Rijksmuseum: SK-A-4690 Year: 1600 Technique: Oil on copper Dimensions original: 44 x 79 cm

Texts: 0

Het volgende schilderij is gemaakt door Karel van Mander en dateert uit de periode rond 1600. Van Mander is de oprichter van de Haarlemse Academie. Dit werk is gemaakt met olieverf op koper. Ook op de achterkant ervan staat een afbeelding, vermoedelijk omdat het een kastdeurtje was. De titel van het schilderij dat u ziet is “De grootmoedigheid van Scipio.” 1 De Romeinse generaal Scipio veroverde de Noord Afrikaanse stad Carthago in 202 voor Christus. De lauwerkrans op zijn hoofd en maretak in zijn hand staan symbool voor zijn overwinning. 2 Het meisje in het geel behoorde tot de oorlogsbuit van de Romeinen. Toen de Romeinen echter hoorden dat zij al verloofd was werd zij terug gegeven. Dit verhaal was een toonbeeld van standvastigheid en zelfbeheersing. 3 De man in het groen is een bruidegom uit Carthago. Hij heeft zojuist zijn verloofde terug gekregen, die door de Romeinen was meegenomen als oorlogsbuit. 4 Links onder zitten twee ouders geknield. Hun dochter was als oorlogsbuit meegenomen. Op dit moment krijgen zij te horen dat hun dochter teruggeven wordt omdat zij al verloofd was. 5 Bij een ontvoering van een bruid werden de geschenken linksonder als losgeld aangeboden. Toen de bruid uit grootmoedigheid werd teruggeven konden de geschenken als bruidsschat worden gebruikt. 6 De ruïne op de achtergrond is de Noord Afrikaanse stad Carthago, die zojuist door de Romeinen veroverd is. Carthago en Rome waren in de oudheid gezworen vijanden. 7 Van Mander schreef boeken met schilderregels. Eén daarvan beschreef hoe je diepte kon creëren door meerdere niveaus te gebruiken. De romeinse soldaten vormen zo’n niveau en zijn enkel in monochrome tinten geschilderd. 8 Karel van Mander hield van symmetrie in zijn schilderijen. Om die reden komen bepaalde felle kleuren dan ook twee keer voor. Een voorbeeld is het blauw van de jurk van de vrouw rechts wat links terug komt. 9 De soldaat links zit in een onnatuurlijke houding. Deze gekunstelde houdingen werden vaak gebruikt om beroemde meesters te overtreffen. Zoals Michelangelo, die lichamen in ingewikkelde poses schilderde. 10 De soldaat rechts maakt een overdreven beweging. Dit soort poses zijn kenmerkend voor het maniërisme waarbij de schilders hun kennis van de anatomie van het menselijk lichaam willen laten zien.

27

Title: The Peace Negotiations between Claudius Civilis and Cerealis Painter: Ferdinand Bol Painting code Rijksmuseum: SK-A-4853 Year: c. 1660-70 Technique: Oil on canvas Dimensions original: 122 x 112,5 cm

Texts: 0. Het volgende schilderij is gemaakt door Ferdinand Bol en dateert uit de periode van 1658 tot 1662. Ferdinand Bol was een leerling van Rembrandt. Dit werk is gemaakt met olieverf op doek. De titel van het schilderij is “de vredesonderhandelingen tussen Claudius Civilis en Quintus Petilius Cerealis”. 1. In de lucht zweeft Fama, de Godin van de roem. Haar symbolische aanwezigheid benadrukt dat hier geschiedenis wordt geschreven. Als teken van een overwinning heeft zij twee lauwerkransen meegenomen. 2. In het midden staat de Bataafse veldheer claudius civilus die een opstand tegen de romeinen leidde. Hij was daarin zeer succesvol maar bezweek uiteindelijk toch voor de romeinse overmacht. 3. Rechts ziet u de legeraanvoerder van de romeinen, cerealis. Hij boekt hier een historische overwinning voor de Romeinen in hun gevecht tegen de Batavieren. 4. De schilder heeft enkele figuren weggeschilderd, onder andere in het midden van het schilderij. Hierdoor lijkt het alsof er schimmen zweven. Deze overschilderingen zijn pas na 4 eeuwen zichtbaar geworden. 5. De kapotte brug loopt over de Rijn bij Xanten in Duitsland, het oude Germania. Hier werd vrede gesloten tussen het Romeinse Rijk en de opstandige Batavieren. 6. Links ziet u de Bataven. Deze werden beschouwd als roemruchte voorouders van het Nederlandse volk. Hun strijd tegen de Romeinen herinnerde de Nederlanders aan hun strijd tegen de Spanjaarden in de tachtigjarige oorlog. 7. Rechts ziet u de troepen van de Romeinen. Ze zijn herkenbaar aan de rode vlaggen en de adelaarsstaf. Het Romeinse leger was in de oudheid een van de meest gestructureerde legers. 8. Onderaan loopt de rivier de Rijn. Rivieren waren in oude veldslagen vaak cruciaal voor de verdediging van een land. De Bataven leefden in het gebied tussen de rivieren de Rijn en de Waal. 9. Het paard vooraan lijkt exact op dat van een schilderij van een andere schilder, Otto van Veen, die deze scène al in 1613 schilderde. Dit was een inspiratiebron voor schilder Ferdinand Bol. 10. De man in het midden met de rode cape draagt een harnas dat meer weg heeft van een harnas uit de middeleeuwen dan een harnas uit de Romeinse tijd. Dit is gedaan om het herkenbaarder te maken voor de tijdgenoten van de schilder.

28

Title: Middle panel of a triptych 'The Adoration of the Magi' Painter: Pieter Aertsen Painting code Rijksmuseum: SK-C-1458 Year: c. 1560 Technique: Oil (?) on panel Dimensions original: 167,5 x 179 cm

Texts: 0

Het volgende schilderij is gemaakt door Pieter Aertsen en dateert uit de periode van 1570 tot 1575. Het is een werk van olieverf op paneel. Oorspronkelijk was het een drieluik, waarvan nog maar twee delen zijn overgebleven. U ziet er hier één van. De titel van dit drieluik is “De aanbidding der koningen” 1 De baby vooraan is Jezus Christus, de zoon van God. Zijn geboorte en leven vormden de basis van het christendom. Jezus symboliseert in dit schilderij de nieuwe Christelijke wereld. 2 De vrouw is Maria. Zij was nog maagd bij haar bevalling omdat ze geboorte gaf aan een kind van God. Maria is een van de bekendste heiligen van de katholieken. 3 De man met de staf is Jozef. Hij trouwde met zijn vrouw ondanks dat ze zwanger was van een ander. Een engel kwam hem namelijk vertellen dat zijn vrouw zwanger was van God. 4 Drie wijzen reden vanuit het Oosten naar de geboorteplaats van Jezus om deze nieuwe koning te eren met geschenken. De knielende wijze in het geel is Melchior, de oudste van de drie wijzen. 5 Op de achtergrond is een paleisruïne zichtbaar. Deze verwijst naar de tijd van koning David, een voorouder van Jezus. De ruïne staat symbool voor de oude wereld van voor de geboorte van Jezus. 6 In de middeleeuwen werd het geboorteverhaal van Jezus zeer verfraaid. Zo werd er gesteld dat deze plaatsvond in een stal. Sindsdien zijn er vaak een os en een ezel op schilderijen van dit verhaal te vinden. 7 De geboorte van Jezus die u ziet vond plaats in een stal omdat een overnachting in de lokale herbergen geweigerd werd. Het rieten dak staat symbool voor deze stal. 8 De halfnaakte mannen zijn in verschillende posities afgebeeld. Door ze zo te schilderen kon de schilder zijn kennis van de anatomie tonen. Dit is kenmerkend voor de schilderstroming genaamd “maniërisme”. 9 De schilder heeft geprobeerd alle elementen uit het Bijbelverhaal te gebruiken. Daarnaast heeft hij ook enkele details verwerkt, zoals de huiselijke mand met doeken die vooraan staat. 10 Drie wijzen uit het oosten brachten vele geschenken voor de nieuwe koning Jezus. Volgens de traditie waren dit goud, mirre en wierrook. In de vaas zit ook een van die geschenken.

29

Appendix B: Questionnaire

30

(dit vak wordt ingevuld door de experimentator) Proefpersoonnummer: Dag meting: Tijdstip:

Instructie Voor u ligt de vragenlijst behorend bij de virtuele museumtourguide. U wordt gevraagd de vragen goed te lezen en zo correct mogelijk te beantwoorden. Indien de vraag geen antwoord bevat dat uw keuze volledig weergeeft, kies dan het antwoord dat hier het dichtste bij in de buurt komt. Bij elke vraag kan slechts één antwoord worden aangekruist of ingevuld, tenzij anders aangegeven. Mocht u onduidelijkheden tegenkomen, dan kunt u deze te allen tijde voorleggen aan de experimentator. De gegevens die u hier invult zullen anoniem blijven. Algemene gegevens 1. Leeftijd: ____ 2. Geslacht: M / V 3. Welk type onderwijs hebt u genoten? (meerdere antwoorden mogelijk, indien van toepassing)

o o o o o o o o

Openbaar

o o o

Algemeen bijzonder

Rooms-katholiek Christelijk Islamitisch Joods Humanistisch Een andere speciale levensbeschouwing Een bepaalde onderwijsvernieuwingsvorm, zoals Montessori, Jenaplan of Dalton Vrije school Anders, nl:

4. Hebt u minimaal 1 jaar een klassieke taal (d.w.z. Grieks of Latijn) gevolgd?

o o

Ja Nee

5. Wat is uw huidige studierichting of opleiding? ___________________________________ 6. Hebt u het Algemeen Vormend Vak “Doos van Pandora” of een ander vak waarin mythologische of bijbelse verhalen behandeld worden wel eens gevolgd?

o o

Ja Nee

31

Kunstaffiniteit 7. Ik ben geïnteresseerd in kunst: Zeer mee eens

O O O O O O

Zeer mee oneens

8. Ik ben geïnteresseerd in schilderkunst: Zeer mee eens

O O O O O O

Zeer mee oneens

9. Ik ben geïnteresseerd in schilderkunst waarop mythologische verhalen staan afgebeeld: Zeer mee eens

O O O O O O

Zeer mee oneens

10. Ik ben geïnteresseerd in schilderkunst waarop bijbelse verhalen staan afgebeeld: Zeer mee eens

O O O O O O

Zeer mee oneens

11. Hoe vaak bezoekt u gemiddeld een museum per jaar?

o o o o o

0 keer 1 keer 2 keer 3 of 4 keer meer dan 4 keer

12. Als ik in een museum ben, dan:

o o o

Volg ik een plattegrond Neem ik een rondleiding Kies ik het liefste mijn eigen weg

32

Evaluatie virtuele tourguide 13. Ik vond de aangeboden schilderijen overwegend: mooi

O

O

niet mooi

14. Ik vond de aangeboden geluidsfragmenten overwegend: interessant

O

O

oninteressant

15. Ik had langer naar de schilderijen willen kijken: ja

O

O

nee

16. Ik had meer achtergrondinformatie willen ontvangen bij de schilderijen: ja

O

O

nee

17. In zijn algemeenheid ben ik tevreden over de virtuele tour: ja

O

O

nee

33

18. Evaluatie schilderijen In het volgende deel van deze vragenlijst krijgt u elk van de schilderijen te zien die u tijdens de tour bent tegengekomen, in een willekeurige volgorde. Per schilderij wordt u gevraagd 2 vragen te beantwoorden: 1. Op het schilderij moet u de delen omcirkelen die u het interessantst vond. Indien u meerdere delen wilt omcirkelen, geef dan aan in welke volgorde u ze het interessantst vond. Het interessantste deel moet u het cijfer “1” geven, het op één na interessantste deel “2”, enzovoorts. 2. Van elk schilderij wordt gevraagd of u dit mooi vond in vergelijking met de andere schilderijen.

34

Latona en de lycische boeren Jan Brueghel I Rijksmuseum, object nummer SK-A-70

1. Omcirkel de delen van het schilderij die u het interessantst vond. Indien u meerdere delen wilt omcirkelen, geef dan aan in welke volgorde u ze het interessantst vond. Het interessantste deel moet u het cijfer “1” geven, het op één na interessantste deel “2”, enzovoorts. 2. Dit schilderij vond ik in vergelijking met de andere schilderijen: Zeer mooi

O O O O O O

Totaal niet mooi

35

De vredesonderhandelingen tussen Claudius Civilis en Cerealis Ferdinand Bol Rijksmuseum, object nummer SK-A-4853

1. Omcirkel de delen van het schilderij die u het interessantst vond. Indien u meerdere delen wilt omcirkelen, geef dan aan in welke volgorde u ze het interessantst vond. Het interessantste deel moet u het cijfer “1” geven, het op één na interessantste deel “2”, enzovoorts.

2. Dit schilderij vond ik in vergelijking met de andere schilderijen: Zeer mooi

O O O O O O

Totaal niet mooi

36

Consul Titus Manlius Torquatus laat zijn zoon onthoofden Ferdinand Bol Rijksmuseum, object nummer SK-A-613

1. Omcirkel de delen van het schilderij die u het interessantst vond. Indien u meerdere delen wilt omcirkelen, geef dan aan in welke volgorde u ze het interessantst vond. Het interessantste deel moet u het cijfer “1” geven, het op één na interessantste deel “2”, enzovoorts.

2. Dit schilderij vond ik in vergelijking met de andere schilderijen: Zeer mooi

O O O O O O

Totaal niet mooi

37

Venus en Adonis Bartholomeus Spranger Rijksmuseum, object nummer SK-A-3888

1. Omcirkel de delen van het schilderij die u het interessantst vond. Indien u meerdere delen wilt omcirkelen, geef dan aan in welke volgorde u ze het interessantst vond. Het interessantste deel moet u het cijfer “1” geven, het op één na interessantste deel “2”, enzovoorts. 2. Dit schilderij vond ik in vergelijking met de andere schilderijen: Zeer mooi

O O O O O O

Totaal niet mooi

38

Het offer van Iphigenia Jan Havicksz. Steen Rijksmuseum, object nummer SK-A-3984

1. Omcirkel de delen van het schilderij die u het interessantst vond. Indien u meerdere delen wilt omcirkelen, geef dan aan in welke volgorde u ze het interessantst vond. Het interessantste deel moet u het cijfer “1” geven, het op één na interessantste deel “2”, enzovoorts. 2. Dit schilderij vond ik in vergelijking met de andere schilderijen: Zeer mooi

O O O O O O

Totaal niet mooi

39

Kustlandschap met Aeneas op Delos Claude Lorrain Rijksmuseum, object nummer RP-T-1965-290

1. Omcirkel de delen van het schilderij die u het interessantst vond. Indien u meerdere delen wilt omcirkelen, geef dan aan in welke volgorde u ze het interessantst vond. Het interessantste deel moet u het cijfer “1” geven, het op één na interessantste deel “2”, enzovoorts. 2. Dit schilderij vond ik in vergelijking met de andere schilderijen: Zeer mooi

O O O O O O

Totaal niet mooi

40

Landschap met de prediking van Johannes de Doper Abraham Bloemaert Rijksmuseum, object nummer SK-A-3746

1. Omcirkel de delen van het schilderij die u het interessantst vond. Indien u meerdere delen wilt omcirkelen, geef dan aan in welke volgorde u ze het interessantst vond. Het interessantste deel moet u het cijfer “1” geven, het op één na interessantste deel “2”, enzovoorts. 2. Dit schilderij vond ik in vergelijking met de andere schilderijen: Zeer mooi

O O O O O O

Totaal niet mooi

41

De grootmoedigheid van Scipio Karel van Mander Rijksmuseum, object nummer SK-A-4690

1. Omcirkel de delen van het schilderij die u het interessantst vond. Indien u meerdere delen wilt omcirkelen, geef dan aan in welke volgorde u ze het interessantst vond. Het interessantste deel moet u het cijfer “1” geven, het op één na interessantste deel “2”, enzovoorts. 2. Dit schilderij vond ik in vergelijking met de andere schilderijen: Zeer mooi

O O O O O O

Totaal niet mooi

42

Middenpaneel van een drieluik met 'De aanbidding der koningen' Pieter Aertsen Rijksmuseum, object nummer SK-C-1458

1. Omcirkel de delen van het schilderij die u het interessantst vond. Indien u meerdere delen wilt omcirkelen, geef dan aan in welke volgorde u ze het interessantst vond. Het interessantste deel moet u het cijfer “1” geven, het op één na interessantste deel “2”, enzovoorts. 2. Dit schilderij vond ik in vergelijking met de andere schilderijen: Zeer mooi

O O O O O O

Totaal niet mooi

43

19. Rangschik de volgende schilderijen. Ga hierbij af op de mate waarin u de gegeven achtergrondinformatie interessant vond. Het schilderij met de meest interessante informatie geeft u een “1”, het schilderij met de minst interessante informatie een “9”. U mag elk getal maar 1 keer gebruiken. Rang (1 = hoogste, 9 = laagste)

44

Open vragen 20. Kunt u hieronder aangeven wat u van de virtuele tourguide vond?

24. Hebt u nog overige opmerkingen over de tour, het experiment of gerelateerde zaken?

Dit is het einde van de vragenlijst. U kunt deze nu inleveren bij de experimentator. Mocht u geïnteresseerd zijn in het eindverslag van dit onderzoek, dan kunt u dat bij de experimentator aangeven door uw e-mailadres achter te laten. Bedankt voor uw medewerking!

45