USABILITY ENGINEERING EN USER EXPERIENCE College 3, 08/05/2015
Joske Houtkamp Departement Informatica en Informatiekunde Universiteit Utrecht
Common ground Toetsbaar/meetbaar (kwantificeerbaar)
Nieuwe applicatie Meetinstrumenten
Vraag: Usability en user experience doelen
Onderzoek: Observatie Interviews … Testen Surveys …
Design guidelines
Usability problemen
Resultaat: Requirements en richtlijnen: invloed op design Bij redesign: verbeterpunten
Onderwerpen Interaction Design Hoofdstuk 4 Social Interaction • Uitgelicht: skype/webconference, • Social presence
Hoofdstuk 5 Emotional Interaction • Uitgelicht: engagement; game Communicate!; training stressvolle
omstandigheden
Hoofdstuk 6 Interfaces • Uitgelicht: succes? https://iactivedesign.wordpress.com/tag/interaction-design/
Sociale interactie • Op welke manier veranderen ICT en sociale
media onze interactie, communicatie, en gedragsregels daarbij? • Vraagstukken voor sociale wetenschap;
gevolgen voor usability&user experience van sociale media en applicaties voor communicatie. • Steeds meer samenwerking op afstand
mogelijk – vragen over effectiviteit en gevolgen voor het proces. • Bijvoorbeeld skype, videoconferencing: via
beeld ook nonverbale cues, gezichtsuitdrukkingen, gebaren. Maar: werkt meer als alleen audio dan als F2F. Hoe verbeteren?
Toepassingen o.a. Samenwerking op afstand (CSCW)
Beschikbaar, eenvoudig in gebruik, combinatie met presentaties (WebEx); Efficient, easy to learn, good utility, helpful
Delen expertise (bijv. medische sector)
Betrouwbaar, hoge kwaliteit geluid en beeld; Effective, efficient, safe to use, good utility
Privégebruik
Betaalbaar, eenvoudig; Easy to learn, effective, enjoyable
Medische begeleiding (thuis)
Betrouwbaar, vertrouwenwekkend (trust); Safe, good utility, easy to learn, helpful
Coaching
Vertrouwenwekkend (trust), social presence; Safe, motivating,
Conference call 26/02/2014 • The conference call is one of the most familiar rituals of office life—and
one of the most hated. • Abuses are rife. People on the line interrupt others, zone out or
multitask, forgetting to hit "mute" while talking to kids or slurping drinks. • One of the biggest problems with virtual meetings is that it is hard for participants to build rapport with each other, a hurdle cited by 75% of 3,301 businesspeople surveyed in 2012 • The absence of nonverbal cues such as facial expressions makes many people hesitant to speak up and makes it harder to pay attention. In the survey, 71% of participants cited a lack of participation by others as a problem with virtual meetings. • Time spent in audio conferences in the U.S. is expected to grow 9.6% a
year through 2017, according to Wainhouse Research, a Boston market-research firm; about 65% of all conferencing is still done by audio calls.
Conversatie-analyse en videoconferencing Sacks et al. (1978) - beurtwisseling Regel 1: de spreker kiest de volgende spreker door vraag te stellen, een mening te vragen; (geen reactie:) Regel 2: een ander begint te spreken (zo nee:) Regel 3: de spreker blijft aan het woord. Cues: verandering toonhoogte, lichaamshouding, blikrichting, gebaren.
Videoconferencing: (eerst zonder video) • A conference call in real life • https://www.youtube.com/watch?v=DYu_ bGbZiiQ&list=WLAD3A829C42391601
Wat ging er mis? • Beurtwisseling (cues) • Technische storingen, latency, echo
(ongemerkt, niet bewust, storend) • Usability problemen (inloggen, mute) • Soms geen “awareness” • Oplossingen met video: bijvoorbeeld
oogcontact mogelijk maken, non-verbale cues, bovenlichaam in beeld • Nieuwe problemen, bijv. appearance
consciousness…
Kuster, C., Popa, T., Bazin, J. C., Gotsman, C., & Gross, M. (Email in real life (2012). Gaze correction for home video conferencing. ACM https://www.youtube.com/watch?annotation_i Transactions on Graphics (TOG), 31(6), 174. d=annotation_3665710195&feature=iv&src_vEsbensen, M., Tell, P., & Bardram, J. E. (2014, October). id=DYu_bGbZiiQ&v=HTgYHHKs0Zw ) SideBar: Videoconferencing System Supporting Social Engagement. In Collaborative Computing: Networking, Applications and Worksharing (CollaborateCom), 2014 International Conference on (pp. 358-367). IEEE.
Voorbeeld van onderzoek: effect van nonverbale informatie Uitgangspunt: oorzaak verschil tussen audio-visual en face-to-face communicatie is afwezigheid nonverbale kanalen (en verbaal compenseren): • Lichaamshouding en handelingen deelnemers (blikrichting, gebaren) • Objecten die bij de gedeelde taken horen (aanwijzen) • Gedeelde werk context, voor ‘grounding’.
Experiment Hauber et. al 2006: vergelijking samenwerken face-to-face, met 3 verschillende interfaces. Onderzoek naar effect op • social presence. Short, et al. (1976) "degree of salience of the other person in a mediated communication and the consequent salience of their interpersonal interactions”; • “copresence, as a sense of spatial co-location, or “the feeling that the people with whom one is collaborating are in the same room” • usability, met name efficiëntie. Mixed measurement approach
Hauber, J., Regenbrecht, H., Billinghurst, M., & Cockburn, A. (2006, November). Spatiality in videoconferencing: trade-offs between efficiency and social presence. In Proceedings of the 2006 20th anniversary conference on Computer supported cooperative work (pp. 413-422). ACM.
Experimenteel ontwerp • Een factor (onafh, variabele), repeated
measures (Latin square) • Afhankelijke variabelen: aspecten communicatie en samenwerking • 30 ppn, teams van 2 • Taak: matchen foto’s van honden met hun eigenaar. Media Richness Theory: taken met grote onzekerheid vragen meer communicatiecues. Handelingen: bespreken, aanwijzen, verplaatsen, draaien. Vier collaboratieve interfaces • 1: ongemedieerd face-to-face; papier, interactie tastbaar • 2: gemedieerde samenwerking, videoconferencing; gedeelde interactieve table (touch sensitive); ondersteunt ruimtelijke cues
Experimenteel ontwerp • 3: gemedieerde samenwerking, 2D video-
conferencing interface, gedeeld foto-applicatie window. Beiden zien hetzelfde scherm. • 4: gemedieerde samenwerking, virtuele tafel in
immersieve desktop virtuele omgeving. “Spatial remote”, ondersteunt ruimtelijke cues. Interactie met de foto’s met muis. Head tracking.
Resultaten • Co-presence: scoren van 4
uitspraken (bijv: it felt as if my partner and I were in the same room) • Social presence
(semantische schaal, bijv. formal-spontaneous) • Preference (rangschikking)
Usability resultaten • Usability issues: vragen over ervaren communicatie,
gebaren en wijzen, verstaan, samenwerking • Analyse van video-observatie: • Task completion time • Turns per minute (beurtwisseling)
• Turn content: taakgericht of gebruik van de technologie • Deictic references vs descriptive references (bijv. “that dog”, “her”,
versus “ the labrador”, the girls with the glasses”)
Resultaten usability: 1. face to face; 2. 2D.
Conclusie experiment Ruimtelijke condities: grotere mentale belasting, meer verwarring, afleiding van de taak, dus minder goede taakuitvoering. Eindconclusie: • de ruimtelijke interfaces hadden positief effect op social presence and copresence; • negatief effect op taakuitvoering.
Verschillende andere factoren hebben invloed: bijvoorbeeld individuele verschillen in voorkeur en ervaring, effect van oogcontact, andere taken, enz.
Onderwerpen Interaction Design Hoofdstuk 4 Social Interaction • Uitgelicht: skype/webconference, • Social presence
Hoofdstuk 5 Emotional Interaction • Uitgelicht: engagement; game Communicate!; training stressvolle
omstandigheden
Hoofdstuk 6 Interfaces •
Uitgelicht: succes? https://iactivedesign.wordpress.com/tag/interaction-design/
Emoties en interactieontwerp Veel onderwerpen • Effect van applicatie op emoties en gebruikerservaring • Bijvoorbeeld in games: engagement, pleasure, fun, voor verhogen van aandacht (en verwacht leereffect) • Emoties in virtuele trainingen: • Affectieve respons in trainingen • Trainen onder stress: “Affectively intense training” • Emoties meten
Emoties • Emotion - a response of the whole organism, involving (1)
physiological arousal, (2) expressive behaviors, and (3) conscious experience. • *James Lange theory - the theory that our experience of
emotion is our awareness of our physiological responses to emotion arousing stimuli. • *Cannon Bard Theory - the theory that an emotion arousing
stimulus simultaneously triggers (1) physiological responses and (2) the subjective experience of emotion. • *Two Factor Theory - Schachter-Singer's theory that to
experience emotion one must (1) be physically aroused and (2) cognitively label the arousal. http://surviveappsychology.weebly.com
Emotie
Basic emotions Bijvoorbeeld: Paul Ekman Universeel
http://visual.ly/robert-plutchiks-psychoevolutionary-theory-basic-emotions
Emoties: circumplex model of affect (Russell) Derde dimensie: dominance
Measuring affect: example SAM Pleasure, arousal and dominance on the Self-Assessment Manikin (Lang, 1985)
Engagement • An exciting and enjoyable state of mind in which attention is willingly
given and held (Rozendaal et al., 2009) • Gerelateerd aan begrippen als flow, immersion, involvement • Complex construct (psychologisch concept): gedrag, cognitief en
affectief; lastig te meten (zie Phillips, R. S., Horstman, T., Vye, N., & Bransford, J. (2014). Engagement and Games for Learning Expanding Definitions and Methodologies. Simulation & Gaming)
• Vaak doel van designers van games, applicaties voor leren en
training, persuasieve technologie • Verwacht positief effect op prestatie en leren
Engagement en leren (Garris, Ahlers, Driskell) Input-process-output model van games voor instructie en leren; proces engagement ondersteunt game play. Game kenmerken kunnen een game cyclus in gang zetten, een herhalende cyclus van beoordelingen van gebruikers, gedrag, en feedback. De combinatie van game kenmerken met de juiste content kan leiden tot gewenste leeruitkomsten. 6 categorieën game kenmerken: Fantasy Rules/goals Sensory stimuli Challenge Mystery Control Garris, R., Ahlers, R., & Driskell, J. E. (2002). Games, motivation, and learning: A research and practice model. Simulation & gaming, 33(4), 441-467.
Engagement in Levee Inspection Simulator (Deltares) https://www.youtube.com/watch?v=CexMBkKwWUY
Serious game in which levee patrollers inspect levees in a 3D environment and locate failures. Sounds added: a continuous soundtrack of rain, strong wind and thunder; positioned sounds for the animals, and the events related to the failures. Dynamic (visual) elements added: lightning, moving trees, more and stronger waves, seagulls, sheep, passing cars, a boat, a sinking dinghy and a street lamp pole that falls over in the storm. Goal: increase engagement, convincingness
Measurement instruments and procedure • Personal characteristics
age, computer experience etc
• Fantasy proneness
derived from Creative Experience Questionnaire (Merkelbach et al 2001) • Pleasure and arousal score on the Self-Assessment Manikin (Lang, 1985)
Task: locate three failure mechanisms; draw map afterwards Maximum play 8.40 mins • Environment: • Weather:
Russell and Pratt list of environmental descriptors nine adjectives, and questions on representation
Measurement instruments and procedure
• Engagement, fun, attractiveness of
game: • SAM
(2nd)
• Facial muscle tension
• questionnaire based on several sources
(a.o. Mallon and Webb, ITC-SOPI) • Game aspects • User experience • pleasure and arousal dimensions
• EMG (electromyography) of the
corrugator (frown muscle) and the zygomatic muscle (smile muscle) as indicator for the experience of arousal and valence
Results Laboratory (controlled) experiment, 50 participants
1 Static No sound
2 Static With sound
3 Dynamic No sound
4 Dynamic With sound
Weather Sounds and visual dynamics, especially in combination are very effective in suggesting strong wind, threatening . weather, and thunderstorm Ambience Environment with sound is more arousing (exciting) Engagement No overall effect, but condition with sounds and dynamic elements was considered more convincing
Results SAM, all conditions: • decrease in valence • increase in arousal EMG: • No significant difference between the
conditions • Large individual differences • Usefulness EMG? Variations may be due to problems with navigating, wires, reaction to the experiment itself etc. General conclusions on determining emotional response, effect on engagement etc: • Often minor change in reactions, difficult to assess with current measuring instruments • Confounding variables (individual differences, task, context, interaction with application)
Corrugator region Cond. Average frequency of action potentials
Std. dev.
1 2 3 4
11.2 8.9 14.1 9.7
13.9 11.9 16.5 8.0
Voorbeeld van een gevalideerde schaal: user engagement scale
Wiebe, E. N., Lamb, A., Hardy, M., & Sharek, D. (2014). Measuring engagement in video game-based environments: Investigation of the User Engagement Scale. Computers in Human Behavior, 32, 123-132.
Reageren op emoties in game Communicate! Trainen communicatievaardigheden Studenten beroepen gezondheidszorg Reageren op uitspraken en emoties van cliënten Belangrijk: hoe interpreteren de studenten de emoties?
Emoties in Communicate! • http://science-vs76.science.uu.nl/backend/ • Discussie: voldoende waarheidsgetrouw?
http://www.txchange.nl/dilem ma-games/
Only fun?
Marsh, Tim, and Brigid Costello. “Experience in serious games: between positive and serious experience.” In Serious Games Development and Applications, pp. 255-267. Springer Berlin Heidelberg, 2012.
Emotionele respons in virtuele trainingen en serious games
Artesis Virtual/VSTEP BV
E-Semble: examens van weginspecteurs
Examination program for RWS road inspectors
Road inspectors survey the Dutch main highways • usually the first to arrive at the scene when an accident or other situation occurs • ensure safety of the people involved and other road users • quickly restore and maintain the traffic flow on the road • clear the way for other emergency services like the police • In 2008: 350 road inspectors in service on the road, every day • E-semble application is used to asses knowledge of procedures and
behaviour with different incidents
Examination program for RWS road inspectors Setup: • candidate/trainee uses joystick to navigate • operator • instructor • display on projection screen • ambient sounds RWS instructors are positive about the application Comments of trainees: • Problems with navigation • Application is not convincing, no subjective experience of ‘being on the road’ • Presented situations do not appear dangerous, resulting in wrong decisions
Manipulations Risk assessment related to • estimations of traffic speed • estimations of distances Improving depth cues (exocentric and egocentric) • scenery (more ‘layers’, e.g. trees) • 3d person PoV • atmospheric effect: haze Improving speed cues • more traffic, more trucks, variation in speed and size • improvement of sounds (ambient and passing cars) (volume must allow communication) Improving context • distance to projection screen • dim lights in room
Not possible: dynamic shadows
Manipulations
Experimental design • 28 road inspectors, between subjects design • ‘mix’ of controlled and field experiment • 5 short clips of scenarios • not interactive (each situation would be different, navigation
distracted) • questionnaires on perceived danger of situation • questions from ITC-SOPI (presence questionnaire) • SAM • distances, speed estimates • 1 interactive scenario: cross the road
Results • estimation of traffic speed lower in manipulated version, but closer to
modeled speed • experience of risk higher in manipulated version • estimation of danger for self sign. higher in manipulated version
Virtual scenario training for leading firefighters Sounds provide information on the
environment and events give feedback on actions make the environment more ‘real’
Convincingness may increase stress level, reminder of real danger
Artesis Virtual/VSTEP BV attract attention to the training environment
Sounds may increase interest of trainee and thus engagement
Artesis Virtual/VSTEP BV
Virtual scenario training • Trainee :
leading firefighter, navigates through environment. • Instructeur: all other roles in the scenario (dispatch center, police, neighbours, etc.)
• Operator:
responds to actions and instructions of trainee, manages crew, fire, vehicles etc.
Training set-up
screen controls avatar
trainee
communication role 2
role 3 role 4 etc
role 1
instructor
controls events operator
Virtual scenario training
Operator view
Trainee’s view
Sounds added – example from storyboard Background
Events
Motor/pumps fire truck
Explosion
Arrival extra truck
Fire spreads
Arrival ambulance
Glass breaking Glass falling
Arrival police car
Ambient sounds Continuous soundtrack: traffic, talking, wind, etc.
Victim crying for help
The scenario Trainee is located in fully equipped fire engine
Call: fire in a suburban house Trainee must follow procedures: gather information
locate the fire determine sources of danger instruct crew members communicate with other services
Communication Communication sounds (speech) not included in this training application. In reality, communication is difficult and confusing:
sounds are modified by headset and helmet
↔ ↔ ↔
messages from crew members, dispatch center, police etc simultaneously
First series of experiments On location, Lieshout fire station 22 fire commanders sound and no-sound condition
5.1 speaker system display on projection screen Questionnaire: items from ITC-SOPI to assess spatial presence, engagement, immersion, convincingness questions to assess the perceived danger for the trainee and the crew members
GSR (Galvanic Skin Response) as indication of emotional arousal
Results Artesis Virtual
Questionnaire after the training: Condition without sounds
Condition with sounds
trainees said sounds would improve the training and increase realism
trainees approved of the sounds, because they increased realism
But (after first analysis): No significant differences on items in questionnaire, e.g. engagement, convincingness, SAM etc. No differences in performance Decrease in convincingness!
GSR: Low arousal in both conditions (and unrealiable)
Results Artesis Virtual Possible causes: • constant communication with the instructor • low volume • navigating distracts attention
Participants were professional leading fire fighters • used to stressful circumstances and frequent training and assessment • ‘drilled’ in applying protocol • focused on building mental model of location of fire, victims, crew etc
Verbal information predominant, visual and auditory information less important.
Second series of experiments, more volume, less verbal information by instructor
Second series of experiments No increase in arousal or engagement
Information sources for building the cube (in percentages). Conditions without sounds
Information sources for building the cube (in percentages). Condition with sound
Second series of experiments
Information sources assessing the danger of the situation (in percentages). Condition without sounds.
Information sources for assessing the danger of the situation (in percentages). Condition with sounds.
Sounds in virtual environments for training Cognitive response Affective response (arousal, pleasure ..)
Sounds affect: verisimilitude → convincingness information
→ validity Sounds are interesting and attract attention
Motivation
Engagement Engagement refers to the emotional state in which continuation of the activity doesn’t require any effort
Transfer of Training Morris, Hancock , Shirkey 2004
Conclusions
Media characteristics training content and tasks other information (communication with instructor)
more interesting and valid training environment
engagement
navigation
arousal, stress
lab or training environment
user characteristics and preferences
Conclusies Experimenteel onderzoek voor verbeteren virtuele trainingsomgevingen: • Gecontroleerde experimenten (in lab) zijn bruikbaar om kenmerken
van het medium te onderzoeken • Onderzoek in het veld is nodig om effecten van gebruiker te onderzoeken • Ervaring, domeinkennis, ervaring met games etc kan sterker zijn dan de effecten van een virtuele omgeving.
Fysiologische maten zijn vaak onprettig voor participanten, mogelijk effect op betrouwbaarheid • Resultaten vaak moeilijk te interpreteren • Gebruik in combinatie met vragenlijsten, interviews
Onderwerpen Interaction Design Hoofdstuk 4 Social Interaction • Uitgelicht: skype/webconference, • Social presence
Hoofdstuk 5 Emotional Interaction • Uitgelicht: engagement; game Communicate!; training stressvolle
omstandigheden
Hoofdstuk 6 Interfaces • Uitgelicht: succes? https://iactivedesign.wordpress.com/tag/interaction-design/
“Nieuwe” interfaces Wat gaat er “fout”? http://en.wikipedia.org/wiki/Nabaz tag http://waag.org/nl/project/scottie http://www.forbes.com/sites/ada mhartung/2015/02/12/thereason-why-google-glassamazon-firephone-and-segwayall-failed/ Google Glass was cool, but it had no killer application
http://www.nu.nl/reviews/404136 1/eerste-indruk-indrukwekkendehologrambril-hololens-mistverfijning.html Https://www.youtube.com/watch? v=_CQA3X-qNgA (Even Apeldoorn bellen)
Opdracht 2 Evalueer een webapplicatie of een (mobiele) app 1. EN 2.
Heuristische evaluatie OF cognitive walkthrough (zelf uitvoeren) Think-aloud protocol; concurrent (4 ppn) OF retrospective (2 ppn).
Applicatie: • Volante online survey http://www.volante-project.eu OF • De SHINE app (http://www.centerdata.nl/nl/projecten-van-centerdata/shine ) OF • een zelfgekozen applicatie. Rapportage: Protocol - analyse van de resultaten - vergelijking van de resultaten – aanbevelingen - vergelijking van de methodes
Usability evaluation – in het kort • Usability testing
• Field studies • Analytical evaluation
Methods and techniques: • Quantitative or qualitative • Formative or summative • Laboratory or field • Users or experts
Inspection method: heuristic evaluation • Experts inspect a system or design using
a set of heuristics or guidelines. • Developed Jakob Nielsen in the early
1990s. • Traditional set is based on heuristics
distilled from an empirical analysis of 249 usability problems. • Empirical evidence suggests that on
average 5 evaluators identify 75-80% of usability problems • Heuristics are developed for mobile
devices, wearables, virtual worlds, games, and for information visualizations (Zuk et al 2006).
Nielsens Heuristics Visibility of system status
Error prevention
Match between system and the real world
Recognition rather than recall
User control and freedom
Flexibility and efficiency of use
Consistency and standards Help users recognize, diagnose and recover from errors
Aesthetic and minimalist design
Help and documentation (Nielsen 2001)
Other inspection methods Cognitive walkthrough, stepping through a pre-planned scenario noting potential problems. Expert is told the assumptions about user population, context of use, task details, and a scenario. Guided by 4(3) questions: • Will the user try to achieve the right effect? • Will the user know that the correct action is available? • Will the user associate the correct action with the effect one is trying to achieve? • If the correct action is performed, will the user see that progress is being made toward solution of one‟s task? •
The answers of these questions lead to credible success or failure stories. To identify usability problems, critical information is recorded from the credible failure stories.
Think aloud protocol • Demo door Dion en Michiel
http://www.life2040.eu/
Usability test 1. Online vragenlijst (parallel aan de applicatie), 40 jongeren uit verschillende Europese landen. Resultaten: positief, nauwelijks problemen gemeld. 2. Think-aloud methode met 5 jongeren. Resultaten: twijfels over begrip van verschillende onderdelen (functionaliteit, formulering vragen, betekenis afbeeldingen). Enkele reparaties. 3. Nader onderzoek, naar aanleiding van de resultaten (onvolledig, vrijwel geen tekst gebruikt, nauwelijks spreiding over Europa). Resultaten: veel onderdelen slecht begrepen, gebruiker slaat die over, heeft weinig geduld. Foute aannames?
SHINE app SHINE Sampling Happiness In Natural Environments. • Alterra (Wageningen Universiteit&Research Center), Planbureau voor de Leefomgeving (PBL), CentERdata. • GPS nodig! • Installeren: hulp van SA’s Eerste test met gebruikers: • Veel aanmerkingen op lengte van de vragenlijst, detaillering • Te weinig volledig ingevuld • Vragen: • Wat zijn gevolgen van gebruik van app buiten? • Hoe gebruikers te motiveren?
Planning opdracht 2 15/05/2015: collegevrij 22/05/2015: tussentijdse toets en gelegenheid voor vragen 29/05/2015: collegevrij (herkansing) 05/06/2015: inleveren
