UNIVERSITY OF AMSTERDAM GRADUATION SCHOOL OF HUMANITIES
DATA VISUALIZATION IN ONLINE DASHBOARDS Optimal visualization of abstract data in online dashboard design Jaimy Quadekker June 2013
Data Visualization in Online Dashboards Optimal visualization of abstract data in online dashboard design
By Jaimy Quadekker, Graduation School of Humanities, Media Studies, University of Amsterdam
Abstract In the data-driven culture that we are living in today it is a challenge to deal with all this data. Visualization is a great tool to display big and complex datasets. Using information visualization for abstract data in the most optimal way represents a different way for people to view and better understand data and the underlying structure of the data. This thesis adresses a specific form of data visualization: Online Dashboards, which are a specific form of information visualization and can be seen as visual summaries that provide insights on online performance. Key purposes of these insights are discovery, decision making, and explanation. But what is the most optimal visualization of abstract data in online dashboard design? In this thesis I develop a new method for the most optimal visualization design in online dashboards. Focusing on the main question draws on a two part empirical approach, combining qualitative and a quantitative methods. The findings show that an effective dashboard should be focused on the most important information and should provide immediate and clear insights to the user. For different information types, different visualization forms should be used, containing mainly bars, lines, or stacked areas. With the results a bridge is created between practical experience, theory and scientific testing. I combine the seeking mantra of Shneiderman, the adjusted principles of the data visualization design process of Tufte, the adjusted principles for the general information visualization process of Mazza, and two key points to amplify cognition in online dashboards by Card, Mackinlay, and Shneiderman. And merging this combination with the general background information of Few and added by the perceived results of this thesis, I create a new method. Together they form a new basis for the design of an optimal online dashboard.
Table of Contents Abstract
ii
1
GENERAL INTRODUCTION
4
2
THEORETICAL FRAMEWORK
7
2.1
Big Data
7
2.2
Visualization back in the days
9
2.3
Information Visualization
13
2.4
The role of perception in information visualization
14
2.5
Online dashboard
17
3
METHODOLOGY
22
3.1
Research goal
22
3.2
Conversion Company
24
3.3 Qualitative research 3.3.1 Research strategy 3.3.2 Target audience
26 26 27
3.4 Quantitative research 3.4.1 Research delimitation 3.4.2 Research strategy 3.4.3 Credibility of research findings 3.4.4 Data Filtering
28 28 34 40 42
3.5
4
Data interpretation
RESULTS
43
45
4.1 Results qualitative research 4.1.1 Basic use of online dashboards 4.1.2 Objectives and types of dashboards 4.1.3 Level in organization and tasks 4.1.4 The dashboards 4.1.5 Word Cloud
45 45 46 47 48 54
4.2 Results quantitative research 4.2.1 Results: Information type 1, comparing magnitudes of independent values
55 56
4.2.2 4.2.3 4.2.4 4.2.5 4.2.6 4.2.7
5
Results: Information type 2, particular comparison with different entities Results: Information type 3, particular comparison with equal entities Results: Information type 4, change over time Results: Information type 5, the overall trend Results: General Dashboards Results: Amount of Numbers
INTERPRETATION AND DISCUSSION
58 61 63 66 68 70
71
5.1
Interpretation qualitative research
71
5.2
Interpretation quantitative research
72
5.3
Discussion
75
5.4
New Method for optimal online dashboard design
78
6
CONCLUSION
83
7
BIBLIOGRAPHY
89
8
APPENDIX
91
8.1 Interview qualitative research 8.1.1 Interview 8.1.2 Dummy dashboards
92 92 93
8.2
Online survey quantitative research
94
8.3
Results qualitative research
97
8.4
Results quantitative research
112
8.5
Recommendation Report for Conversion Company
112
1 General Introduction In 2013 there is significantly more data available than the previous year. On IBM’s website, it is stated that we currently create over two and a half quintillion bytes of data a day. This is referred to as “Big Data” and is generated on the internet from numerous sources, including social media sites, web shops, digital pictures and more. Now information can be obtained anywhere and by anyone, it is not surprising that ninety percent of the data we know today has been created in the past two years. Because of the amount of data that is available, the challenge is how to deal with all this data, including capture, storage, search, sharing, analysis, and last but not least visualization. The recognition of visualization is growing along with the data growth particularly now as it is coming into use in a broader range of branches, such as the business industry (Card, Mackinlay, and Shneiderman 7). With the combination of visualization and abstract data, information visualization is a great way to display these big and complex datasets. It represents a different way for people to view and better understand data and the underlying structure of the data (Mazza 1). So on the one side we have big data, data sets with an enormous assortment of data available and on the other side, data visualization, both providing insights in this enormous assortment of data. The big question becomes: how to use visualizations in the most optimal way to keep up with the data-driven culture that we are living in today? In data where there is a lot of obvious and apparent information, there is also a great deal of information and knowledge which is not apparent and obvious, yet valuable. Such as large automatically obtained performance based data of websites, where information is hidden in the data. Visualization represents a process which enables better insight into the data, potentially revealing the patterns within the data enhancing the value of the data and becoming a greater source of information and knowledge. One of the tools to reveal patterns within big datasets is the form of visualization ‘online dashboard’: a tool for monitoring what is going on and for having control over what is going on at the online part of an organization (Few, Mazza). Therefore, online dashboards can be viewed as visual summaries that provide insights on the online performance. While there are many, a few key purposes of these insights are discovery, decision making, and explanation (Card, Mackinlay, and Shneiderman 6). Therefore information visualization is a powerful tool, which increases the ability to perform these and other cognitive activities. 4
According to Stephen Few, author of the popular book ‘Information Dashboard Design’, it is a challenge to communicate the information that is hidden within the data. It only succeeds if the one that implements the dashboards understands visual perception, and applies this understanding in the design principles of the dashboard. If dashboards are aligned with the way people see and think (4), they can communicate an idea, because a dashboard is an effective communication medium (Mazza 20). But what is the best method to design an online dashboard according to these principles? The issue then becomes which trends and theories could best be applied on the visualizations in online dashboards? What is the optimal visualization of data on these online dashboards? How do you design the layout of a complete online dashboard? What is the best visualization per graph, if you look at different kinds of information? There are a number of answers on these questions, but what is remarkable is that many answers of the current approach of designing online dashboards are based primarily on observations and practice, rather than on empirical research. Because the shape of online dashboards evolved through the years rather than through intentional or studied design and in and of themselves, they did not get a lot of attention yet (Few). Through adaption and inclusion of the desires of clients and users, the dashboards evolved automatically out to what they look like currently with current displays and shapes that are not tested scientifically. While in this field of visualization I found a lack of theory and scientific research, my goals is to bring a more scientific tested approach to the design of online dashboards and that is the focus of my thesis: Optimal visualization of abstract data in online dashboard design.
Also Conversion Company is confronted with the challenge of the use of online dashboard and how data is actually visualized in the most optimal way. Their dashboards are based on practical experience and clients desires, which is closely connected to the wider debate. Conversion Company provided the opportunity to make use of their databases and online dashboard for the research of my thesis. Conversion Company is an organization that works with online dashboards and online performance on a daily basis. Conversion Company as an organization was founded to provide tools to support the adjustment process that companies go through, caused by the Internet boom and thereby the changing behavior of consumers and companies. One of the key areas is data analytics, where they use online dashboards to present performance based data to their clients. With large amounts of real time data and web analytics, they use visualizations in online dashboards to make sense out of the data and 5
provide insights. They also confront the challenge of the use of online dashboard and how data is actually visualized in the most optimal way. Their dashboards are primarily based on practical experience and clients desires, which is closely connected to the wider debate. Conversion Company provided the opportunity to make use of their databases and online dashboard for the research of my thesis. In this thesis my interest lies in learning more formally and systematically about the actual insights to be gained and the optimal structures and results to be achieved from online dashboards. The question, then, would be: how can the design of online dashboards be improved and what is the benefit of looking at their current practical in a more empirical, scientifically tested manner? It is my objective to provide an improved method which could be applied on online dashboards and that they can enhance their online dashboards with a more scientific approach and scientific results. Based on theory and practical experience in combination with datasets, I have designed a convenient research method, which I apply to find answers to the research questions and redirect them into a bridge between practical organizations and scientific theory. My empirically based research has been divided into a qualitative and a quantitative research part, which, I believe, provides a new method for designing the most optimal dashboard, including the most optimal visualizations of data in online dashboards, based on information type. Beginning with a qualitative research, in the form of five interviews, I collect insights and define desires regarding online dashboards. Followed by a quantitative research, that can be delimited using the results of the qualitative research. Resulting in a focused online survey where the data of 149 respondents, that are professionally connected to the field of online dashboards, has been collected. With a combination of the respective qualitative research and the quantitative research I empirically test the usability of dashboards. With this I have provided answers on the specified research questions and have developed a new methodology with an empirical tested optimal design for online dashboards.
In this master’s thesis I have first composed a theoretical framework. I begin with a general overview of the history of visualization up to now and I continue with the current state of visualization and of the specific forms of information visualization, using theories of Tufte, Ware, Card, Mackinlay, Shneiderman, Few and more. Explaining why the role of perception is very important in visualizations and information visualizations should offer depth in the understanding of the power of visualizations. Narrowing down to online 6
dashboards, I discuss the history, followed by the current state to outline where online dashboards stand right now. After discussing the relevant theorists in above theoretical framework, this thesis turns to the methodology to explain how I am going to answer the research questions. How I approach the research is illustrated and demonstrated in different research sections, containing the qualitative and quantitative research. My approach for data collection and data interpretation are explained elucidated and the results of the qualitative research are summarized. I continue with the presentation of the quantitative research results, subdivided into the different information types, the general dashboard, and numbers. The result of this serves as foundation for the interpretation section, where results of the two research approaches and theory is consolidated to merge practical experience, scientific research, and theory. The outcome is a new method which can be used to design the optimal online dashboard, which is presented after the discussion. The conclusion finishes the thesis. And the appendixes, containing the recommendation report for Conversion Company, close this thesis.
2 Theoretical framework In the theoretical framework I discuss the different theories to create a clear framework. I start with the debate on big data and how visualization fits in this area of big data. Continued by drawing some historical background, I approach visualization back in the days. Followed by the transition of visualization to information visualization. Continuing with the importance of the role of perception that could not be left out. Concluding with narrowing down to the specific field of information visualization namely online dashboards, to specify the key topic of this thesis.
2.1 Big Data ‘Data, data everywhere’ is the special report of the Economist in February 2010 in which the growing effect of big data is discussed. The term ‘Big Data’ has gained increasingly attention the past years and is used more and more in areas like popular media, business, computer science and the computer industry (Manovich 1). In ‘Trending: The Promises and the Challenges of Big Social Data’, Lev Manovich puts forward the following account of what big data is, that is applicable on the computer industry: “Big Data is a term applied to data sets whose size is beyond the ability of commonly used software tools to capture, manage, and access the data within a tolerable elapsed time” (2). Through big data massive quantities of information are now accessible. The interaction of people and objects online creates 7
information about the digital traces that they leave behind (boyd, Crawford 662), such as user generated content like photos on Flickr, web usage statistics, and created metadata of the performance of websites like tags (Manovich 2). All this information that is created is a potential rich source of knowledge. Boyd and Crawford claim that big data even emerge and change the system of knowledge that people currently know. They aim to reveal that big data has the power to inform how people understand things, such as human networks and communities (665). Manovich elaborates on two types underlying big data. The first type is surface data, which is data about many people but more on a more general level. The other type is deep data, which is data about small groups and contains more detailed information about them on an individual level. We deploy surface data in all kinds of disciplines like communication studies or marketing research. Prior, a researcher had to choose carefully a sample set of representative deep data to use for a research (3), but because of big data a complete deep dataset is easily available. It is no longer necessary to choose between the size of data and the depth of data. Detailed data are automatically obtained and therefore available to a lot more people. It contains knowledge and insights of for example transactional data in a web shop. Surface is the new depth, according to Manovich. Big data is really powerful and provides endless possibilities. A quote from the ‘Petabyte Age’ in Wired “Because in the era of big data, more isn't just more. More is different” (Manovich 4), explains the opportunity of change that big data could provide in the future. Boyd and Crawford highlight the importance of the way of dealing with the emergence of an era of big data. They think that this phenomenon is coming about in an environment of rapid change in which current decisions could have a great influence on the future of different branches such as the business and computer science industry. Currently more and more people can analyze large datasets on their own computers with rather standard software like Excel or more advanced like Tableau. Big data is not necessarily about the massive datasets, like Manovich claims, but more about the capacity to search, aggregate, and cross-reference large data sets (boyd, Crawford 622). It is important to focus on the questions ‘how to deal with all this data?’ and ‘how to engage with this information?’ Following boyd and Crawford in contrast to Manovich, it is not the size that matters, but the interaction within big data. Therefore the procedures of how the data can be processed are taken under the loop. Not just raw data, but the analytical relation, how can large datasets be understood? And how can large
8
datasets be combined with other datasets? The part of understanding and sense making is an important focus of this thesis. Manovich claims that with the rise of big data three different groups of people and organizations have been formed based on their access possibilities to data. He begins with the group that creates the data by leaving their digital footprints. This group contains almost everybody who is using the web or mobile phones. The second group is smaller and contains those who have means to collect the data. The third group refers to those who actually have the expertise to analyze the data. This is the smallest group and Manovich calls them the big data society (10). This division in different groups impedes the usage of big data because not everybody has the same access possibilities and this contains beneficial and negative aspects for the different groups. But concentrating on the several ways of analyzing big data, visualization could uncover significant meaning hidden in big data (Manovich 9). Bollier claims that visualizing data is one of the best tools to identify meaning out of all the data and making correlations and exploring them in a way that could develop new models and theories. With the use of data visualization, all groups mentioned by Manovich are meaningful to this process. The first group creates the data by means of their digital traces, which forms the basis of the information for the visualization. The second group collects the data and makes the data accessible so it can be used and visualized. The third group has the expertise to make the big data understandable, for example via visualization. Especially the second and third group focus on sense making and discovering the data. I strongly agree with Bollier and see lots of opportunities in the use of visualizations within this field. Visualization is becoming a popular tool for communication (Bollier 10) such as for example online dashboards, which I discuss below in section 2.5. The power of visualizing big data provides a new method to “find things that you had no theory about and no statistical models to identify, but with visualization it jumps right out at you” (Stensrud, cited in Bollier 10). Data visualization has the potential that it could reveal large-scale patterns (boyd, Crawford 664).
2.2 Visualization back in the days For centuries cognitive tools like a piece of paper, writing instruments and other geometry instrument are used to support the thoughts. Looking at science back in time one can see that writings, diagrams and other visualization tools are essential to scientists. Hutchins, as cited by Ware, claims that only little cognitive work, like thinking, is done with the eyes and ears 9
closed. Most of this cognition is done in combination and interaction with cognitive tools, like paper or calculations, or other ancient visualization tools (2). Visualization has been used for many years already. Tufte, a highly reputed visual designer, statistician and academic, describes in the famous book ‘Visual and Statistical Thinking’, the information visualization used during the cholera epidemic in London back in 1854. This demonstrates that clearly important decisions are made based on visual representations of evidence at that time already. Back in the days, originators like William Cleveland and Edward Tufte were few of the first to write about the basic principles of the visualization of data. Highly important is the quality of these representations, because they are responsible for the decisions and the consequences of these decisions. Like the launch of the space shuttle, described by Tufte in 1997, when a poor quality visualization was used to make an important decision for launching a space shuttle. Through the wrong interpretation of this visualization, the wrong decision was made, which resulted in the loss of many innocent lives. Therefore one of his key points is that “Visual representations of evidence should be governed by principles of reasoning about quantitative evidence. For information displays, design reasoning must correspond to scientific reasoning. Clear and precise seeing becomes as one with clear and precise thinking”(53). He aims that scientific principles are important to provide a controlled comparison and guidance during the design process of data visualization. There are different principles of visualizing data, but these are the six main principles of Tufte, and are important ground rules to take in account for both reasoning about statistical evidence and for designing statistical graphics (53): the first is the documentation of the sources and characteristics of the data. The second is the focus on an adequate comparison. This is followed by the demonstration of the cause and effect with comparable mechanisms. And back in return those mechanisms should be expressed quantitatively. The fifth attention point is the acknowledgement of the multivariate nature of analytic problems. And the last sixth principle is the inspection and evaluation of the alternative explanations. In my opinion there is an extra principle that could be added to make the ground rules of Tufte more complete. This seventh principle focuses on the ‘making sense’ of the visualization. This last step is crucial in the process of statistical evidence and designing graphics and one of my objectives of this thesis. Tufte says that there are right and wrong ways to visualize data; there are visualizations that reveal the truth and visualizations that do not (45). Therefore the seventh principle is essential, because sense making should reveal the
10
truth within a visualization. These seven principles provide a good visualization design guidance for representing of data. Other key points of Tufte are that visuals support the cognition and good design brings absolute attention to data. You should also keep in mind that the visualization should be very clear in visualizing the cause and effect. Moreover the ordering of the data is an important element. And the sequential order conceals a possible link or relation between variables (Visual and Statistical Thinking 48). But how does this good design of Tufte looks like? What is considered clear and what is the perfect lay out? Through this thesis I provide answers to these crucial questions. Also more and more scientific fields are searching for ways to give guidance to present the static data and information in a clearer and more understandable way, for example fields of perceptual psychologists, statisticians, and graphic designers (Bertin, 1983; Tufte, 1983, 1990; Shneiderman, 1996). First if one thought of the word ‘visualization’ there was thought of a visual image in the mind (Shorter Oxford English Dictionary 1972), but currently a new meaning of this word has emerged and we could also refer to a graphical representation of data or concepts. In 2004, Mazza demonstrated even more definitions of the word ‘visualization’ in the dictionary: -Visualize:
form a mental image of ...
-Visualization:
the display of data with the aim of maximizing comprehension rather than photographic realism.
-Visualization:
the act of process of interpreting in visual terms or of putting into visible forms.
(Mazza 5)
But how is presentation, representation and visualization seen today? The definition is shifting, but it becomes clear that in the definitions of visualization the activity of humans still remains. Through the years a shift is emerging from an internal construct of the mind, where people visualize a visual image in the minds, to an external artifact supporting decision making, where something else like a visualized table helps people to make decisions (Ware 2). Currently the modern definition of Card, Mackinlay, and Shneiderman of visualization “The use of computer-supported, interactive visual representations of data to amplify cognition” is prevalent, where the last three words communicate the main purpose of visualization. I consider the definition of the User Interface Research Group Web-site of the 11
Palo Alto Research Centre (PARC-XEROX), which includes the foregoing definition, the most complete and suitable to apply on my research: “Information Visualization is the use of computer-supported interactive visual representations of abstract data to amplify cognition. Whereas scientific visualization usually starts with a natural physical representation, Information Visualization applies visual processing to abstract information. This area arises because of trends in technology and information scale. Technically there has been great progress in highperformance, affordable computer graphics. At the same time, there has been a rapid expansion in online information, creating a need for computer-aid in finding and understanding them. Information Visualization is a form of external cognition, using resources in the world outside the mind to amplify what the mind can do” (Mazza 6). I use this definition for visualization, because it describes my focus on the process of visualizing online data and information, which are the objectives of my thesis. Also the recall of big data and the expansion of online information and the urge of finding and understanding visualizations are key focus points within my thesis. The last sentence how visualization is a form of external cognition which amplifies the mind fits perfectly with my objectives of my thesis and research. The term information visualization is often used in combination with the term ‘abstract data’. There is a distinction between data that is closely related to mathematical structures and models and between data that is more abstract in nature. Abstract data could for example exist out of stock market fluctuations or website performance measures. The first distinction is mostly known as scientific visualization and the latter one as information visualization (Mazza 8). I focus on the latter; information visualization. Also Ben Shneiderman believes in the strength of information visualization. In 1996, Shneiderman introduces his mantra, the Visual Information Seeking Mantra; overview first, zooms and filters, then details on demand. A visualization begins with overview, that is, to get the broadest picture of the entire collection. This is followed by zooming in on the items of your interest. The next step is to filter out uninteresting items. Then, the objective is to close with details that selects an item or group and get the details needed for the information that is sought.
12
2.3 Information Visualization Hearst proposes one of the main questions in the field of information visualization of the last years, and that is the question: how do we convert abstract data into a graphic representation, preserving the underlying meaning and, at the same time, providing new insights (cited in Mazza 9)? The main goal of information visualization is to deal with large amounts and unstructured abstract datasets. Therefore the relevant questions are, how is information visualization constructed? Or how could the visualization of abstract data be applied on the visualization form where I focus on in this thesis; online dashboards? Different elements are important in the building process. Listed below are the five basic principles by Mazza that need to be considered at the general information visualization process: 1.
The problem. This relates to what has to be presented, demonstrated, or found.
2.
The nature of the data. There are different data types, data could be numerical (a top-5 list), ordinal (non-numerical data having a conventional ordering, such as days of the week), and categorical (data with no specific order, like cities).
3.
Number of data dimensions. Depending on the number of dimensions, representations can be univariate- (one dimension), bivariate- (two dimensions), trivariate- (three dimensions), and multivariate data (four or more dimensions). We perceive our world in three spatial dimensions, therefore interpreting up to three dimensions is rather easy. But things with more than three dimensions however, are very frequent in real world situations and represent one of the most challenging tasks in Information Visualization.
4.
Structure of the data. This could be linear (data coded in plain data structures, like arrays or tables), temporal (data which changes during the time), spatial or geographic structure (like maps or something physical), hierarchical (like structures in organizations), network structure (representing relationships between two nodes).
5.
Type of interaction. Whether the resulting graphical representation is static (like a print or a static image on a display screen), transformable (features like zooming or filtering), or manipulable (users may control parameters during the process of image generation).
When looking at these five basic principles for the design of an information visualization, I noticed that they are really focused on the goal of the visualization and the type of data. Only the last principle, type of interaction really focuses on the representation of the data. Although I cannot deny that the goal of the visualization and data entities are important to keep in mind 13
during the visualization process, I find this list incomplete. The attention to the part of making sense and understanding, where the visualization is perceived by the cognitive ability of people, is completely left out. Like Mazza quotes in his book, the purpose of graphics is that they display the facts about data in such a way, that others could see and understand them as well as recognizing the underlying structure and the hypothesis about the data that is shown in those graphics (1). The underlying structure could be nicely designed in an information visualization, but if it is not perceived and understood, it could not be judged as a successful visualization. Information visualization is an inductive method, because if the graphic is understood better, it creates new insights and ideas. The graph should contain graphical excellence, like Tufte claims, and this consist a complex idea that is communicated and carried out with clarity, precision and efficiency. It is all about the data and information that should be visualized in the most clear, precise, and efficient way which results in the most fitting interpretation of the data and information revealed by this visualization. In this thesis I focus on the step between the data and information and the step of how the visualization is interpreted. But while focusing on this specific part of the process, I cannot leave out the part that takes place in our head while we interpret something. People use the perceptual and cognitive system during this process.
2.4 The role of perception in information visualization In context of my research questions I elaborate on the role of perception. Because thinking and seeing are closely connected to vision. It does not matter what the activity is, it is likely that the mental work of thinking and perceptual interaction of seeing are interwoven (Card, Mackinlay, and Shneiderman 1). The amount of sense receptors in a humans body that are dedicated to vision goes up to seventy percent. How people see and how people think is tightly connected with each other (Few “Information Dashboard Design” 79). People give meaning to what they see with their eyes through their brains. The eyes are the sensory mechanisms, but the brain processes the impulses into a sense making perception. It is a given that people perceive more information through vision than if they combine all of the other senses together (Ware 2). They possess twenty million neurons in the brain that analyze visual information, with a pattern-finding mechanism that is a fundamental component in most parts of a humans cognition (Ware 2). Also Ben Shneiderman believes in the strength of the human visual domain and proposes that the bandwidth of visual information presentation is potentially higher for media 14
in combination with the visual domain compared to media in combination with other senses. The remarkable perceptual abilities are under-utilized in design, as he claims in his work of 1996. Information Visualization is an inductive method. With our sense making perception people use information visualization to creates new insights and ideas. People use the human perception system as a fast filter and because of the vision that perceives patterns, people see the patterns within the data that reveal a structure. The same perception system confirms or refutes the patterns in a quick way. Therefore it takes his role as an insight generating method (Fekete et. al. 4). But how does the perception system actually works? The first thing one needs is the idea of information, and then one can begin to communicate. The second is the use of graphical means, to create or discover the idea itself. As Bertin in 1977 would say, “you have to use the special properties of visual perception to resolve a logical problem” (cited in Mazza 1). You have to use vision to think (Card, Mackinlay, and Shneiderman 1). And using vision to think is a very old method. Currently with the evolution of computers, it is easier to create and develop graphics, like real-time interactivity, improved rendering, pictures, or revealed hidden patterns that are just a few clicks away. These quite new features enable also new methods of amplifying cognition with knowledge and insights. This power is discovered by a broader range of branches, like business and education, with the use of abstract data. Therefore the definition of information visualization is used (Card, Mackinlay, and Shneiderman 1, 7).
It is clear that information visualization depends on the properties of the human perception. But how do you amplify information? To begin with visual perception, therefore you need your head, eyes and attention in an active process that could amplify information per unit from the visual world. There are two different ways how this information can be processed. The first one is controlled processing, for example reading. Here you need the ‘fovea’, the central area of the human eye. This type of processing is detailed, serial, slow, low capacity and able to be inhibited. The second way is automatic processing and is characterized by different targets that pop out during search. It is superficial, parallel, fast, high capacity, cannot be inhibited, and, is independent of load and consciousness. But there is not a strict contrast between the two, the contrast is more practical. Visualizations could be designed to make optimal use of these processes. For example the textual description aside of a 15
visualization is accessible by controlled processing, but coding techniques for search and pattern detection are accessible for automatically processing (Card, Mackinlay, and Shneiderman 25). But cognition cannot react on information independently. The power of the unaided mind is highly overrated. Without any external aids, the memory, thoughts, and reasoning are all constrained. But human intelligence is highly flexible and adaptive. The real powers actually derive from devising external aids that enhance the cognitive ability (Norman, cited in Card, Mackinlay, and Shneiderman 1), and therefore information visualizations are a suitable solution. The trick that makes visualization such an assistance in the cognitive process, is because it is difficult holding partial results in your memory unsupported and kept in your memory until they can be used. Visualization supports this part of your memory. An internal memory task is converted into an external visual search (Card, Mackinlay, and Shneiderman 2). Therefore the method for expanding the power of the unaided mind is to provide external aids, especially notational systems and ways of representing an idea in some external medium. Therefore it can be maintained externally, free from the limits of working memory (Card, Mackinlay, and Shneiderman 34). They claim that there are in total six ways in which visualizations could amplify cognition. Beginning with increasing the memory and processing the resources that are available to the users. For example expanding the working memory or storage of information, by which they mean that visualizations could expand the working memory that is available for solving problems or that it could be used to store large amounts of information in a quickly accessible form. Secondly visualizations can amplify the cognition by reducing the search for information, through grouping information together or high data density. In third place the use of visualization could enhance the detection of patterns; recognition instead of recall is a first example of this way of amplifying cognition. Also organizing the information helps here. The fourth way of amplifying cognition by visualizations is the perceptual inference operations. Here are inferences more easily done that are otherwise not that easy performed. Followed by the fifth way and that is using perceptual attention mechanisms for monitoring. Concluding with the sixth and that is by encoding information in a medium which can be manipulated, which can allow exploration of a space of parameter values and that could amplify user operations (Card, Mackinlay, and Shneiderman 16). Looking at the goal of information visualization, it is to provide insights. The main purposes of these insights are discovery, decision making, and explanation (Card, Mackinlay, 16
and Shneiderman 6). Therefore information visualization is useful because it increases the ability to perform these and other cognitive activities, because the visual system is closely connected to the adaptive decision-making mechanism that people own (Ware 2). Mazza claims that visual perception of data could facilitate the finding of relationships, trends, revealing hidden patterns (1) and this equals main purposes of online dashboards, which is the specific type of information visualization where I focus on in this research.
2.5 Online dashboard With the perception principles and tricks in mind, I continue to the specific visualization area of online dashboards, which is the focus point of my thesis. Online dashboards can contain different types of information. Some online dashboards are summarized reports that show for example the digital traces that users leave behind or performance based data of a website. Other dashboards communicate the information about how the website of an organization is doing. In the figures 1, 2, 3, and 4 below, you find four examples of online dashboards in their early stages, these are illustrations out of the book of Few ‘Information Dashboard Design.’ According to Few a dashboard only succeeds in communicating if the one that implemented the dashboard understands visual perception. And if he applies this understanding in the design principles of the dashboard, by making the dashboard design aligned with the way people see and think (4). When looking at the six ways by Card, Mackinlay, and Shneiderman, in which information visualization could amplify the cognition, this could also be applied on a visualization such as online dashboards. Especially the second way, making it easier to find information by grouping, could be applied to the research of online dashboards in this thesis. During the visualization process this would be helpful to amplify the cognition. Also the third way where the detection of patterns is proposed, is crucial for this type of visualization. Taking into account that recognition works in a more beneficial way than recall for amplifying the cognition, should definitely be considered during the design process. With these tricks, information visualization, in general, and online dashboards, in particular, could amplify the cognition and support the perception and leads one from abstract data to more specific and concrete insights and knowledge. But what are online dashboards exactly? And how did online dashboards developed in time? In this section I elaborate on the history of dashboards and how they developed over time. Although there was, and still is, an enormous enthusiasm around online dashboards (Few 17
“Information Dashboard Design” 3), there was only little definition. But Stephen Few made a serious effort of defining the ins and outs of dashboards as described in his book ‘Information Dashboard Design’. Online dashboards existed already in the 1980s, but the former name of online dashboard was actually Executive Information System (EIS). In the world of information technology, it had the functionality to display a few key financial measurements through a simple interface (6), comparable with the online version of a dashboard in figure 1. However, it lacked required technologies and without these, its utility and potentiality could not be effectively realized. EISs stepped back in the shadows. In the 1990s the information era offered new opportunities as new methods of collecting, correcting, integrating, storing and accessing information began to get more attention, mainly focused on new technologies to make information available and useful. A new management approach also emerged in the early 1990s, named key performance indicators. This feature used specific metrics, called indicators, to measure the business landscape. Targets and goals of organizations were focused around those indicators. A new interest in management through the use of these specific metrics, broader than only financial metrics, which are still very dominant elements today in the business landscape. With reference to these metrics, performance data could be monitored and analyzed more efficiently with EIS-type displays (7), in figure 2 a comparable dashboard is shown. Monitoring what was going on and having control over what is going on in organizations became more important.
Figure 1: Example online dashboard Source: Information Dashboard Design, Stephen Few
Figure 2: Example online dashboard Source: Information Dashboard Design, Stephen Few
With the growing push for accountability, new business intelligence (BI) systems were established. Visual summary software with key metrics of (online) performance were applied with far more regularity. Dashboards, and the concept name, were used more frequently.
18
Suddenly vendors in the BI space were offering dashboard software, without clearly explaining what a dashboard really was. An IBM account manager defined the term as “[...] at IBM we mean whatever the customer thinks it means” (8). The software that was offered to support the making of dashboards did not really think this and other features through. Rather, they implemented ‘slash and dazzle’ as the marketing effort instead of concentrating on the visual perception possibilities in the design (5) (see figures 3 and 4). The gap between the software and how the users perceiving the results lacked engagement of interaction with human perception resulting in several types of software and several types of definitions of online dashboards. Everybody was asking for online dashboards, but nobody actually knew what this meant. But the definition of Few, gives more clarification: “A dashboard is a visual display of the most important information needed to achieve one or more objectives; consolidated and arranged on a single screen so the information can be monitored at a glance.” Clarifying that, online dashboards are not a type of information or technology, but rather, are forms of visual representation. Usually they exist out of a combination of text and graphics, with the focus on the graphics (35). Dashboards and information visualization make an excellent combination, due to the close connection between our visual system and the adaptive decision-making mechanism.
Figure 3: Example online dashboard Source: Information Dashboard Design, Stephen Few
Figure 4: Example online dashboard Source: Information Dashboard Design, Stephen Few
But an apparent deficiency in this field of business work and intelligence, is the lack of attention to the understanding of visual perception, and how this understanding can be applied to visualization of data (Few “Information Dashboard Design” 37). While designing a dashboard, the use of understanding of perception can be implemented right away. In the iconic memory people unconsciously process. For example elements being grouped together, 19
like Card, Mackinlay, and Shneiderman are describing in their basic principles. The next type of memory is the short term memory, also called the working memory. A person is able to store three to nine ‘chunks’ or ‘clumps’ of visual data at the same time in this working memory (81). A chunk could be one individual number or a pattern with more lines within a graph. This could be seen as the third principle of Card, Mackinlay, and Shneiderman, recognition rather than recall. Taking this into account during the design process, makes the use of graphs in dashboards more beneficial for the working memory. By clumping information together as optimal as possible the support of the perception and understanding is most efficiently. But pay attention with what you include in one clump. When it becomes too complex a person would not be able to perceive these chunks of information automatically. Card, Mackinlay, and Shneiderman call this phenomena pre-attentively processing. Because our eyes see what lies within their span of perception. Also eyes have a funnel; only a part of what our eyes sense becomes objects of focus. But only a part of that is transformed to something more than a vague sense. This small part keeps our attention and becomes a conscious thought. Again a fraction of those thoughts are stored in our brain (80). With the research questions in mind, this is the context for my research. When is information written as one chunk, and when does it become too much for pre-attentively processing? Supported by Ware, who categories pre-attentive attributes of visual perception into four categories: color, form, spatial position, and motion. In color use, recommended is that you should not use too many hues (Few “Information Dashboard Design” 89). For form, ‘simple’ shapes like squares, lines and circles are preferable. Looking at spatial position the Gestalt theory explains important principles that are perceived by the visual system and help understand an image better. To respond on our perception, Gestalt properties such as proximity or closure are easily perceived as perceptual features and should be taken into account (Card, Mackinlay, and Shneiderman 28). The last category ‘motion’ is the only one not applicable on online dashboards. But applying these other three categories of pre-attentive attributes, benefits the perception of chunks in the working memory. The attributes color and shapes are taken into account during the research of these thesis, to provide a scientific tested and more precise recommendation their use in online dashboards. A person can perceive three to nine chunks at the same time, so how does this affect the complete layout of a dashboard? Already one of the benefits is that dashboards offer a supporting solution for the problem of information overload (Few “Information Dashboard Design” 37). But not only in dashboards but an interesting phenomena of the past years is the 20
experience of too many things placed in a limited space during everyday live. Books on shelves, addresses in agenda, windows on a computer screen. The information explosion leads to the existence of more data than what can easily be displayed at once. Like Spence referenced to in his book Information Visualization ‘Too much data, too little display area’ as a common problem of information visualization (cited in Mazza 20). Just like in the case of online dashboards. There are too many interesting pieces of information that can be displayed at once. Zooming, planning, scrolling, focus/context, magic lenses could serve as a solution for this problem. Although these are a few solutions, they are in my opinion not ideal to apply on online dashboards. A more suitable solution is the mantra of Shneiderman; which contains overview first, zoom and filter, then details on demand. Keeping in mind that online dashboards and visualizations are cognitive tools, they also improve your "span of control" over a lot of business data. But how to transform data into something that people can understand for optimal decision making (Ware 5)? Visualization and cognitive tools help people visually identify trends and patterns, reason about what they see, and guide them towards effective decisions. Therefore it is a challenge to find the balance between as much information needed to keep the span of control and no information overload. Brath and Peters elaborate on the upcoming trend of online dashboards and other visualization tools that are widely available for business users to review their abstract data. The issue of visual information design is more important than ever (cited in Few ‘Information Dashboard Design’ 37). At the same time, one of the key characteristics of a dashboard is that they have small, concise, clear, and intuitive display mechanisms (36) and finding the combination to monitor this data is a challenge. Present only the information that is needed and choose the visualization that works the best. Few argues that you should choose functionality over design, present only the information that is needed, and choose the visualization that works the best. Together those would provide the optimal solution for the design of online dashboards. But what are these functionalities and which information is considered needed? When does which visualization works the best in an online dashboards? Those questions are asked in this thesis to provide the optimal method for the design of online dashboards. If I look at the relevance of information visualization in the age of big data, information visualization tools like online dashboards are needed. They provide an answer on the question of how to deal with all this data that can be obtained automatically. I have discovered gaps and focus points that need more development. The six main principles of Tufte that could use 21
a seventh principle which focuses on sense making and understanding. Also the different elements of Mazza that need to be considered at the information visualization process could use some more influence of the concept of perception and the ability of cognition. Two of the six ways, which could amplify cognition by Card, Mackinlay, and Shneiderman are perfectly applicable for online dashboards and are beneficial for my research. There are also a lot of great recommendations presented by Few and other authors. But how many, if any of these recommendations have ever been scientifically tested? Most results are obtained through many years of experience which is also very characteristic for the field of online dashboards. Firstly I hope, with this approach for my research for the thesis, to start to seal these gaps between practical experience and scientific testing. Secondly I hope to develop certain insights and focus points further.
3 Methodology In this chapter I elaborate first on the research goal. I continue with some insights about Conversion Company, where I illustrate some current used dashboards. Then I discuss the qualitative research, with special attention for the research strategy and target audience. Followed by the quantitative research, with special sections on the research delimitation that are used to provided a very focused quantitative research, on the research strategy of this research part, and the credibility of the research finding. Then I explain the data is going to be interpret.
3.1 Research goal The concrete goal of the research of this thesis is to develop a new method which would be applied to the designing process of online dashboards, hence a new method with a general approach of the designing process with the implementation of a connection between the practical organization and the scientific theory. Concluding into developing a method to find the most optimal visualization of data for online dashboard design. This optimal dashboard should be personalized and tailored specifically to the requirements of a given company, otherwise it would not serve its purpose (Few “Information Dashboard Design” 36), however shaped in a more scientifically tested and general design provided by this new method. How do you translate raw data into meaningful data tables, and then into visual structures (Card, Mackinlay, and Shneiderman 33) and what is the best design of these visuals? I pursue a 22
general empirical interest in the process how different types of data are visualized in different ways and which are the most suitable visualization forms? How are these different forms of visualizations interpreted and how are they made sense? Those are interesting empirical questions for my research approach. By researching these elements I want to contribute to the wider debate around online dashboards and in the broader debate of data and information visualization. Which visualization amplify cognition and support the perception of the most optimal? The responses to these questions might well lead to the resolution of the key question: ‘How can you design an online dashboard in the most optimal way?’ This provides a new method to design this online dashboard in the most optimal way. Online dashboards have a history of practical experience, and developed through time, based only on these practical experience. Through the years the dashboards were adapted to the desires of the client without empirically testing the effectiveness, meaning that there is currently no structured process or standard and thoughtful format for developing them. They are in use without any analytical or scientific testing or systematic analysis or results and their impact on effectiveness. At this point there is the need for serving clients and customers a dashboard that is build on an empirical, more scientifically developed method. As a result of the research problem, the theoretical framework and the exploratory nature of the research, different questions are proposed to develop this new method. The main question is ‘What is the most optimal visualization of abstract data in online dashboard design?’ To answer this main question, three sub-questions need to be asked. Beginning to obtain a clear view on online dashboards, based on practical experiences, the question ‘What is the context and what are the requirements of an online dashboard?’ should be stated. The next sub-question is ‘How do you design the layout of a general online dashboard?’. Looking at the general layout, this question again could be subdivided into three questions ‘How does the design of a general online dashboard look like if you look at the amount of graphs?’, ‘How does the design of a general online dashboard look like if you look at the numbers?’, and ‘How does the design of a general online dashboard look like if you look at the remaining elements?’ The third and last sub-question focuses on the visualization per graph and therefore is ‘What is the best visualization per graph?’ Here I decided to focus on five different information types which provides the questions ‘What is the best visualization in combination with comparing magnitudes of independent values?’, ‘ What is the best visualization in combination with particular comparison with different entities?’, ‘ What is the best visualization in combination with particular comparison with equal entities?’, ‘ What is the best visualization in combination with change over time?’, and the last question ‘What is 23
the best visualization in combination with and the overall trend?’ With these five questions a clear focus per graph per information type is drawn. Looking at the nature of the questions, I decide to break the research into two different parts, a qualitative and quantitative part. First I use a qualitative approach to mainly focus on the first sub-question about the context and requirements. With an empirical approach which contains personal interviews I try to provoke answers on the research questions, based on the practical experience of the interviewees. More details on this approach are discussed in section 3.3. The second part is a quantitative part, where the research questions are scientifically tested with a group of 148 respondents. More details on the approach of this research part are discussed in section 3.4. Merging the two research types together results in a extensive answer on the main question. But before elaborating on the qualitative and quantitative research approaches, I provide more detail on the organization Conversion Company whose databases and online dashboards serve as a basis for both my research approaches.
3.2 Conversion Company Conversion Company works with online dashboards on a daily basis. They use online dashboard to summarize and present performance-based data of their clients websites. To optimize the website and the conversion ratio of the different clients, both Conversion Company employees and the clients work with the online dashboards. An ideal situation for Conversion Company would be an optimal general online dashboard. A great combination was found between the research for my thesis and the profit my results could have for the improvement of their online dashboards. Doing a research internship at Conversion Company gave me the opportunity to get support; in the way of making use of their datasets, their network, and their expertise. The figures 5 and 6 below show two currently used online dashboards (anonymized).
24
Figure 5: Online dashboard example at Conversion Company
Figure 6: Online dashboard example at Conversion Company
25
By providing some background information about the organization Conversion Company, it becomes clear in what kind of professionally orientated work field the used online dashboard originate. Conversion Company is an organization that focuses on online performance and it was founded to support the adjustment process that companies go through, that is caused by the Internet boom and thereby the changing behavior of consumers and companies. They turned the challenge of big data into their expertise. They operate in the fields of online marketing, channel management, customer contact and interaction, e-business consultancy, media management, performance based partnerships, outsources e-commerce, developing and programming, and analytics and intelligence, and is therefore really an allround company. The clients that they work with within different field of expertise, such as Automative, Financial Service, FMCG & Retail, Government, HR Solutions & Service, Logistics, Telecom & Media, and Travel & Leisure. Further details about this company and their clients are requested not the be disclosed.
3.3 Qualitative research In this section the qualitative research strategy is explained, what the goals are and how the interview is contructed. Continued by the target audience used in this part of the research.
3.3.1
Research strategy
The qualitative research design is firstly developed in such a way as to perceive answers to the developed research questions above. The aim is to ensure insights from within the field of online dashboards and define the expectations from respondents with different perspectives. The goal of this qualitative research is to outline the general expectations of both sides of parties that work with online dashboards, namely Conversion Company and the clients of Conversion Company. Secondly this research section could be considered as an exploratory research, prior to the quantitative research. The obtained insights accompanied by the reviewed literature are a comprehensive preparation for the quantitative research. The third goal is to provide answers to the second and third research question by combining this qualitative research with the quantitative research for a complete answer on the research questions.
26
The data gathered from this section is non-numerical data. The qualitative research contains an interview with five different participants. The layout of the interview is showed in the first appendix (section 9.1). To obtain a broad scope of data I selected five participants with dispersed professional work backgrounds, containing three employees of Conversion Company and two clients of Conversion Company. Although both parties are working with the same online dashboards, the expectations of online dashboards of employees of Conversion Company differs from those of their clients. This expectation gave a framework for the interview questions. Simple key questions about objectives, expectations and opinions can elicit answers that draw the different opinions about online dashboards. I choose to read the questions out loud to provide the opportunity to improvise during the interview for optimal results. The first step, prior to the interviews, is to explore all dashboards in use at Conversion Company. Thereafter all features incorporated in the diverse dashboards are outlined. Proceeding from those features, a set of questions is established, focusing on eliciting spontaneous responses. The construction of those questions is open and clear to evoke optimal responses. The first part of the qualitative research contains questions focused on the goals and expectations one has with online dashboards, based on their own experience and desires. In the second part I present four dummy dashboards, with disguised data. The intention here is to evoke reactions on the different designed dashboards to identifying what is important according to the different respondents. The dummy dashboards are very diverse on several fronts, for example amount of information and type of visualizations. The third part of the qualitative research contains statistical questions, to map the characteristics of the respondents. Recording the conversation and transcribing them completely afterwards provided detailed qualitative data. I conducted the interviews in the language that suits the participants best.
3.3.2
Target audience
I interviewed five participants in total, three internal and two external participants. Out of the employees of Conversion Company I selected an internal web analyst (someone who analyses the obtained data), an internal channel manager (someone who manages the online channel 27
for the client), and an internal manager/director (owner and founder of the organization) to interview. And on the opposite side I interviewed an external online media manager and external web analyst. I selected two web analysts because of their everyday work with dashboards and therefore their experience with designing, reading and interpreting them. Including a channel manager is interesting because of the experience between Conversion Company and another company and therefore he is familiar with both sides of the use of online dashboards. One member of the management team is included, not because of the amount of experience with online dashboards he has, but to present a dispersed point of views. His background in marketing and experience with different types of decision makers provides fresh insights as well as different intentions and expectations on a high organizational level. Conversion Company gave me the opportunity to interview two of their clients. The functions of these two respondents were left to chance.
3.4 Quantitative research In this section first the research delimitation is explained. Beginning with the information flow that is used, and continued with the level in the organization, audience and span of data. Then the role of visualization, the goals and the data domain is discussed. Also the type of measures, data and frequency are discussed in the delimitation, as well as the mechanism of display, interactivity, color and visualization form. This is followed by the research strategy of this quantitative part, continued by the credibility of the results. This sections ends with the data filtering that needs to be done to provide a clear focused dataset to work with.
3.4.1
Research delimitation
For Conversion Company their key motivation is a specific ‘information flow’, when it comes to online dashboards (see figure 7). Dealing with the age of big data, the performance based data that is obtained needs to be summarized and visualized in the online dashboards, which they present to their clients. This information flow contains the obtained data, with information hidden in it. This information is visualized to go to the next step of interpretation. The visualization is interpreted and insights and patterns should be discovered in the visualization. Based on the obtained insights or patterns, an action or decision should follow. The last component, which is action, is connected with the information through result or feedback. This result or feedback is created if the visualized information does or does not lead to an action or result, which can be a change of elements or concluding that an element 28
provides good results. If the visualized data does not lead to an action or result, it means that there is a misinterpretation of the visualized information. Apparently the information that is (hidden) in the visualization is not interpreted correctly or perceived at all. This results in positive or negative feedback, allowing information to reshape and possibilities to visualize the data and information with another design. This again results in a different visualization design, where a better interpretation should develop the correct related action.
Figure 7 - Information flow Conversion Company; focused on data/information and interpretation.
During the preparation and the design of online dashboards, the subject of big data is discussed again. Lots of data is available, finding the balance between data overload and clear data is a challenge. There are several variables that should be taken into account during the design of an online dashboard. These variables could be covered by seven theoretical categories, provided by Stephen Few (Information Dashboard Design 39) and six additional variables provided by the qualitative research results. The categories of Few are: the type of data, the data domain, the type of measures, the span of data, the update frequency, the interactivity, and the mechanisms of displaying. The five additional variables are level in the organization, the audience, the visualization role, the main goal, color use, and the visualization form. In total these thirteen variables are combined together and below I discuss them in three subsections to provide a detailed overview. The first covers the level in the organization, audience and span of data, continued by the second which contains visualization role, goal and data domain. The third discusses the type of measures, data and frequency and in the last and fourth I elaborate on the mechanism of display, interactivity, color and visualization form. 29
The challenge during the making of a more generalized method for the design of online dashboards is to keep most categories as consistent as possible. When you adjust only few variables during the design process, it provides quickly the desired result. I connect the theoretical categories with the practical categories and below I discuss these individually. Also I elaborate on the decisions made concerning each category if this should be a consistent variable or not in the approach and design of the quantitative research. Level in the organization, audience and span of data On which level in the organization are online dashboards used and what kind of audience is reached? And what is the span of spread of the data, within or even outside the organization? Observing the body of organizations like Conversion Company and organizations where they work with (and for), they are roughly divided into three levels as demonstrated in figure 8. Although the companies are situated in many different industries, the same division could be applied to all the companies. At the top you recognize the so called C-level, containing chefs, CEO’s, etcetera. The next level is the management level, consisting of the managers controlling the different departments. Subsequently, there is the operating level, where the tasks are executed. The focus of this research is situated between the management level and the operating level. Interaction between those two levels encompasses performance from the operating level and justification towards the management level. And back again the managing level directs the operation level. An online dashboard is a fitting tool to enrich this interaction between the two levels.
Figure 8 - Organization levels; focused on Management level and Operating level.
30
The expectation is that differences within the level in the organization goes hand in hand with different interests and expectations of the online dashboards. For Conversion Company the focus of interaction is located between the management level and the operating level. The workflow goes from justification from operating level to managing level, to the managing role of this level to the operating level again. Decisions are made at the managing level, and linked back to operating level, and their job is to accomplish these decisions. Looking at the audience, the question is asked if the one that is going to be working with the dashboard is experienced or just a beginner with reading data? Keeping the audience as a variable in the research approach could give difficulties, because adjusting the dashboard based on the audience asks for a lot of different dashboards. Moreover it is difficult to choose which criteria of audiene you should use. An experienced web analyst looks differently towards an online dashboard than a novice web analyst. After some pre testing, keeping this variable consistent results into a reduced amount of work and a generalized dashboard. Deciding that a dashboard gives the feeling of control should cover every audience. Obviously different designs fit the span of individually used dashboards, department used dashboards or enterprise wide used dashboards. At Conversion Company the span of data is departmental and therefore I choose to use this span as well for the research approach.
Visualization role, goal and data domain What do you want to achieve with this visualization and in which domain? Both strategic, analytical and operational roles are used in practice of the online dashboards at Conversion Company, which complicates this variable. Resulting from the qualitative research, it is evident that the goal of visualizing information is to give a feeling of control of the website and to give detailed insights on what is happening on the website and the dashboards that are presented to the clients of Conversion Company have these goals as well. Conversion Company provides visualized information in the form of an online dashboard and the client interprets the information. Then the step to action in on the side of the client and Conversion Company does not have any influence on this section. Therefore the task of Conversion Company is to provide complete and clear information, to obtain the desired action. I avoid focusing on decision making, and I concentrate on the data and information, and the interpretation. The ultimate goal is the correct interpretation and feeling of control over the website. Of course the action part is interesting as well, because this part provides feedback what could reshape the information in a more optimal way. 31
The data domain is different in every case claims Conversion Company. This variable is chosen to left out of the consideration of the research approach, based on the argument of Few: ‘choose functionality over design.’
Type of measures, data and frequency What kind of measurements and data are we dealing with? At Conversion Company key performance indicators are defined and measured. Therefore they work with univariate data, which are measurements of a single quantitative variable, (Cleveland 17) and bivariate data, which are paired measurements of two quantitative variables (Cleveland 87). A few times non-quantitative information is found in a dashboard. A benefit of online dashboards is the possibility to monitor many types of data (Few 38) and therefore I choose this variable to be one of the focus points of the quantitative research. I categorize the types of data based on the online dashboards of Conversion Company into five different information types: Comparing magnitudes of independent values; particular comparison with different entities; particular comparison with equal entities; change over time; and the overall trend. The research is divided into these five types of information, to measure which form of visualization suits best in combination with which type of information. Another variable is the frequency of updating the data. The expectation is to develop a different dashboard design depending on the frequency of usage. A weekly designed dashboard should look different compared to a monthly designed dashboard. But since Conversion Company uses weekly dashboards most often, this results in the choice of focusing only on weekly online dashboards in this part of the research.
Mechanism of display, interactivity, color and visualization form Looking at the mechanisms of display, primarily graphs and graphic forms are used. There is also the possibility to include non-quantitative information in a dashboard. Simple lists are used quite common (Few “Information Dashboard Design” 46), because sometimes it is not an addition to visualize data. Then lists are satisfactory. For the choice to provide a static or interactive display, Conversion Company chooses the static option. It is a ‘snapshot’ of a certain moment once a week. Although there is a side note that needs to be made because lots of dashboards have the opportunity to navigate, which means that the user can click on certain features and look at the data that is needed. This could be seen as interactivity as well. Through obtained insights in the qualitative research part, this is difinitally one of the key desires of an online dashboard. After considering if I include this 32
variable into the quantitative research, I decide not to. Including this variable could cause less detailed results on other important aspects therefore I use element of a statical dashboards in the quantitative research. An important variable is the color usage, how does color influences the interpretation and action? The hypothesis is that people with less data skills attach more value to the usage of color. This is definitely an interesting research, especially when the focus on interpretation in combination with color is tested. But I decide to exclude this variable in the research, because it could enlarge the quantitative research to much. By excluding color, I create a more specific scope and focus on the other variables. The final variable is the form of visualization. It is expected that every client wants different visualizations, from bar graphs to illustrations. But this variable suits the best for my research questions and could provide the answer to the gap between science and practice. Therefore the visualization form is selected as key variable together with the data in the form of different types of information.
I want to use different forms of visualizations and find the best combination between type of information and visualization. The type of visualization form is discussed in the book of Stephen Few ‘Now you see it’, which serves as a basis for the design of the quantitative research approach. He discusses and recommends different forms of visualization based on his experiences. Countless visualizations are recommended by Few and used in the different online dashboards of Conversion Company, but I have to narrow down to a few potential visualization forms. But focusing on the online dashboards of Conversion Company and the results from the interviews, I concluded in a focused choice of seven visualization forms. The collection contains bar graphs, line graphs, graphs with points, and stacked area graphs. Also combinations of bars, lines and points provide the graphs that I call barline, barpoint, and linepoint and those are included in the collection. The choice to use these four basic types of graphs and three combinations of these basic types is because of the fact that they are mostly used in online dashboards and also most preferred in the qualitative research results. Only stacked area chart does not provide a clear combination, therefore I only use this form on its own. The explicit choice to leave out the textual visualizations of data shifts the focus even more on visualization in graphs. But not all the graphs are a good combination with the different information types. Therefore every information type with corresponding question forms a subset with all the visualizations that match with this type of information. Section
33
4.2.2 ‘Results per information type’ demonstrates the overview of every subset per specific information type.
The choice of these delimitations result in a focused research approach. For most optimal results I keep most of the variables consistent and diversify only the type of data and the visualization forms. Testing different visualization forms with different types of data should provide the most optimal combinations between those two.
3.4.2
Research strategy
The second research section gathers quantitative data and could be considered as the key part of the research. The intention is to investigate the most optimal visualization of data. This part of the research contains an online survey. As result of the qualitative research, the scope of the quantitative research can be delimited into a clear focused research. For the design of the first part of the research all dashboards used by Conversion Company are observed again. Most dashboards are subdivided into units (see fig. 9, 10, and 11). One unit has a title, sometimes a subtitle, statistical numbers and a visualization. For this research I exclude the statistical numbers and form a new unit with a title, a visualization, and a legend (see fig. 11). I call this total unit in the research and from now on a ‘graph’.
34
Figure 9: Example of complete online dashboard
35
Figure 10: Dashboard unit of a online dashboard of Conversion Company .
Figure 11: Re-designed unit, used in the quantitative research as ‘graph’.
After that the data from the databases are examined. The databases contain all the information that is obtained from the websites that Conversion Company works with. Focusing on the variable ‘type of information’ the information in the databases could roughly be defined into five different types of information. Each is given a number to keep a clear overview. The information types are comparing magnitudes of independent values (type 1), particular comparison (type 2 and 3), change over time (type 4), and the overall trend (type 5). The particular comparison is again subdivided into a particular comparison with different entities (type 2) and equal entities (type 3). The five types of information have corresponding visualizations, which provide five subsets. Each information subset is represented by a specific question that is characteristic for that type of information and contains a question about something that is visualized in the graph. The potential visualizations for that type of information are attached to this subset. So each subset is a conglomeration of one type of information, a specific characteristic question, and different visualizations (see table 1). Which concludes in thirty-four different graphs with questions that are used in the quantitative online survey (see appendix 9.2 the thirty-four different graphs).
Information type number
Example Question
Visualization Forms
Information type 1
Has Mango increased twice as much as Strawberry?
bar, line, points
Information type 2
Is it likely that the positive trend of the total order value is caused by the average order value?
lines, points, barline1, barline2, barpoint1, barpoint2, linepoint1, linepoint2
Information type 3
Is it likely that the negative trend of total visitors, from week 50, is caused by returning visitors?
bar, lines, points, stacked, barline1, barline2, barpoint1, barpoint2, linepoint1, linepoint2
Information type 4
After which week did the downward trend of the bounce rate begin?
bar, line, points, stacked
36
Information type 5
Is this company on target if you look at the first nine months?
bars, lines, points, barline1, barline2, barpoint1, barpoints2, linepoint1, linepoint2
Table 1: quantitative research structure, subsets containing information type, specific question and visualization forms.
For the quantitative research I use the online tool Qualtrics (http://bit.ly/133MZtc). After the design of the quantitative-research is complete, the design is implemented online with support of this online tool (see figure 12). Using a multichannel approach which provides a wide range of respondents, I approach people through the professional network of Conversion Company combined with my personal LinkedIn, Twitter, Facebook and e-mail to distribute my research. The audience that I selected for this research contains people that are professionally involved with the use of online dashboards. People that create, interpret, use, and understand online dashboards or are professionally involved with online performance could be beneficial for this research. More than three hundred people where addressed and this resulted in a response rate of fifty percent of completed surveys.
Figure 12: Screenshot of one of the questions of the quantitative online research
37
In this research I am focusing on the interpretation. How is the visualization form interpreted and understood. The first part, which focuses on the different graphs, contains a question above the visualization that is proposed to the respondents and in order to answer this question the respondents need to interpret and understand the visualization. They have to make sense out of the data in the visualization to answer the question. Because I am looking for the best way to visualize data, I measure how long it takes for someone to answer the question. Therefore each separate question has a timer. How long is needed to read and understand the specific graph, which results in an answer to the question, is measured (see fig. 13). Because of this method, it is important that the questions and graphs are not multi interpretable or ambiguous, since time required to answer the question is measured in this part of the research. Therefore five verification tests are performed.
Figure 13: Screenshot of the timer function of the quantitative online research
The first part contains fifteen questions about graphs (see example in fig. 12). The questions are closed and can be answered with 'Yes', 'No’ or 'I don’t know’ for information types 1, 2, 3, and 5. For information type 4 a certain week number has to be selected (see fig. 14).
38
Figure 14: Screenshot of one of the questions of the quantitative online research for information type 4
The second part is collecting the personal opinion of the respondents. What does the respondent thinks he or she prefers? Looking and comparing different visualizations to see what is preferred regarding quantities in graphs and amounts of numbers in figures (see fig. 15). Followed by five questions containing the possible visualization forms for a specific type of information (see fig. 16). In this part the respondents are asked directly which visualization form they prefer and consider as most clear. This way the respondents is given the possibility to compare the visualization forms, to make a personal choice. Comparing this choice of preference with their answers on the first part provides a clear view if their preferences correspond with their ability to quickly make sense out of a visualization.
39
Figure 15: Screenshot of one of the questions of the quantitative online research, focused on amount of numbers within a graph
Figure 16: Screenshot of one of the questions of the quantitative online research, focused on preference of visualization form
Continuing with part three of the online survey, where a number of statistical questions for potential correlations between answers and statistics are proposed. The questions include age, gender, job, industry, education, analytic experience. See appendix 9.2 for a more details of this online survey.
3.4.3
Credibility of research findings
The trustworthiness of the findings is enhanced by the reliability and credibility of the results. The threats of reliability have been reduced by several actions. 40
Prior to the real quantitative survey, five sample survey are conducted to check if the questions are not multi interpretable or ambiguous. The type of questions and the approach of this survey could be uncomfortable or unfamiliair for respondent, because it could be seen for the first time. It may occur that respondents start up slowly on the first questions of the survey because of this unfamiliarity of the questions or the visualizations that they see. Therefore a total randomizing-tool is applied on all thirty-four questions with graphs from the first part of the survey. The first part contains thirty-four questions. But taking the time of the survey in account I had to minimize this amount, because otherwise the step to participate in this survey would be too high and my concern is that it would decrease the response rate for the survey. Considering that I want the survey to be a maximum of ten minutes to complete this survey, I used the randomizing-tool to propose every respondent fifteen out of the thirty-four questions. Because only when I acquire enough respondents, I have enough data to satisfy all questions. To prevent learnability (enable the user to learn how to use something, or in this case answer the question, when they see it multiple times) all the questions are phrased slightly different but with the same meaning. The construction of the sentences is always consistent. Also the graphs are slightly different as well to prevent lapsing into rote responses. I used the randomizing tool for all questions which means that every respondents is provided with an authentic set of fifteen randomized questions and also the order of these questions. Otherwise all the respondents would have started with one of the first question, and that could have results for the measured interpretation time, if there was a possible effect of patterned or rote responses. Graphs with two metrics could contain a combination of visualization forms, for example a combination of bars and a line. These are double includes in the survey. For example the first time metric one is visualized as bars and metrics two as a line, and the second time with metric one as line and metric two as bars, two examples are shown in figure 17a and 17b.
Figure 17a: Screenshot of barline1
Figure 17b: Screenshot of barline2
41
For the target audience a non-probability sampling is used in order to derive a sample out of the audience in the online sphere that is professionally active in online performance.
3.4.4
Data Filtering
Focusing on the quantitative research, it is essential that I work with a clean and focused dataset. I gathered all the data obtained out of the online survey from the online tool Qualtrics. Because I want to keep the target audience focused, I begin with filtering the dataset by looking at the type of respondents who participated in my survey. The target audience are people who are professionally involved with the use of online dashboards and have a certain education degree. Therefore I first filter out the respondents without a Higher Professional Education (in Dutch translated to Hoger Beroeps Onderwijs), or an advanced academic degree in education (in Dutch translated to Weterschappelijk Onderwijs), which results in three respondents. Respondents with a Higher Professional Education degree or higher, but without a direct connection with online dashboards are checked on their analytical ability. If they have answered question ‘How do you rate your analytical ability’ below a rate of three out of five, I try to identify their professional background, if they are involved with online dashboards in an indirect manner If they are not involved with online dashboards I excluded them from the dataset, which resulted in two respondents.
Secondly I focused on the remarkable extreme results that show up occasionally; these I excluded from the dataset. Only complete surveys shall be included in the first place. Looking at the results within these complete surveys, there are remarkable extreme results that pop up occasionally in the data results, varying of questions that are answered within eighty to two hundred seconds, which is obviously not very convincing. Firstly I compare all the results of a specific question with each other (vertical in the datasheet), extreme results pop up automatically during this comparison. By checking and comparing these extreme results manually with the other results from the same respondent but for other questions (horizontal in the datasheet) provides a judgment if they are normal for this respondent or really extreme. Framing all those extreme results provides a clear summary. This results in the decision to delete all results above sixty seconds aka one minute. The basis for this decision is that more than eighty percent of extreme results are above seventy seconds. Therefore the few results between sixty seconds and seventy seconds are not extreme but high results, but the results under sixty seconds I consider as credible results which I shall use for my research. I have 42
decided to exclude these high and extreme results above sixty seconds, and shall only work with the results under sixty seconds for a validated research. The reason for deleting the high and extreme results is that I believe that these results are the consequence of two things: first, the respondent does not understands this specific graph properly, caused by the question, the visualization, or the answers. And secondly, the respondent is likely distracted from the survey and therefore cannot focus on answering the question properly. I filtered the high and extreme data out of the dataset. In my opinion they could evoke a distorted result of the quantitative research. Therefore I continue with the filtered dataset to interpret the results of the quantitative research.
3.5
Data interpretation
In this section I explain the way of interpreting the obtained results of the quantitative research, because all the different parts need a different approach of interpretation. The interpretation of the qualitative research is not discussed, because those results are summarized into key aspects in section 4.1, results of the qualitative research. The first data gathered on the first part of the survey, is the metric time. Containing ‘Timing-First Click’, ‘Timing-Last Click’, and ‘Timing-Page Submit’. The second data that is gathered are the answers on the questions about the visualizations. The questions can be answered in two ways, where the first option is for information type 1, 2, 3, and 5, where the answers are divided into ‘Yes’, ‘No’, and ‘I don’t know’ (see fig. 12). The second option is for the question belonging to information type 4, those need to be answered with a specific week in a dropdown menu (see fig. 14). For the subset belonging to information type 1, 2, 3, and 5, I want to know how much time it took for a respondent to interpret the visualization. To figure this out, I look at ‘Timing-Last Click’, because this is the last click before submitting the page and therefore this is also the click of the final answer of the question. By summing the time spend on answering the question of all respondents that got this question, and dividing through the number of respondents that got this question, I proved which visualization is interpreted the quickest. With the questions of information type 4, the method for answering is different and therefore I have to approach these questions differently. Because these questions are answered with a week number, I used a dropdown method in the online survey. I noticed that this dropdown function does not always measure the ‘Timing-First Click’ or ‘Timing-Last Click’. ‘Timing-Page Submit’ however is always measures. Therefore I want to focus on ‘Timing43
Page Submit’, and look at this time, because this is more secure and provides more data to base the results on. My claim is that the visualization that is answered the quickest, is interpreted the most quickly. My prediction is that this interpretation is caused by the fact that this visualization is the most clear. The process of visualizing data/information, towards the step of interpretation is most optimal in this type visualization for this type of information. The time difference between ‘Timing-Last Click’ and ‘Timing-Page Submit’ could also been measured for information type 1, 2, 3, and 5. This measures the time that people need from the moment they interpret the question and the moment of being all secure of this answer by submitting it. The time measured here could give an indication of how sure people are about the question. High scores here could indicate that the question or visualization is difficult to interpret or even ambiguous. Also measured in part one are the answers. Critical is also to have a look at the correct interpretation of the visualization. Are the answers answered correctly or incorrectly? Therefore I looked at the numbers and percentages of the amount of questions answered right and wrong and compare them with each other. Because I am searching for the best visualization for interpretation, I have to find out which graph provides the most correct interpretations.
Continuing with the questions in the second part of the online survey, which were questions about the clearest visualization in the eyes of the respondents. The respondents could click one visualization they like and find the most clear and also one they dislike and find the least clear (see fig. 16). This question gives the opportunity to look for relations or contradictions between the opinions of the respondents and the perceptual and interpretation ability of the respondents by looking at the interpretation time (Time-Last Click). Is the reactions and interpretation time on the visualizations corresponding or different from what they say that they prefer? Also essential in part two of the quantitative research was to look at the general or overall dashboard. How do the respondents feel about the layout of the whole dashboard? Making use of a ranking tool, they could drag the dashboards in their preferred order. Looking at the ranking order, I show the results of their opinions to answer this question. As well as the question about the amount of numbers. Because number are included in all the dashboards of Conversion Company. They are included within the graphs, under the title and above the visualization. When questioning the visualizations in part one, I left the number out on purpose. I did not want them to be a distraction for the visualization. Later in 44
part two a question focuses on numbers. Also here they could drag their amount of numbers in the preferred order. To interpret this question I look at the ranking order that the respondent used, while answering this question.
I elaborated on the total research approach and the methodology, with a special section on the data filtering and the interpretation approach of the data. It has become clear how I shall approach the obtained results from the online survey.
4
Results
4.1 Results qualitative research Below I demonstrate my qualitative data results, gathered from conducted interviews (see appendix 9.3). First I present the basic use of online dashboards. Continued by the type of dashboards that the interviewees use and the objectives of online dashboards in their opinion. Followed by the results on the level in the organization, where dashboards are being used. The next section is about the dashboards that are presented to the interviewees and the opinions about these four different dashboards. I conclude with the insights of a word-cloud, containing all the interviews together.
4.1.1
Basic use of online dashboards
If I look at the results of the qualitative research, the basic principles of an online dashboards stand out. It is used as a report with a summarized view on the performance of a website. The data, containing performance based results of a website, is obtained through programs such as Comscore, Omniture or Google Analytics. These programs measure the digital traces and online performance on websites and provide abstract datasets based on those measurement which then can be analysed. All this data contains information and insights, which both can be used for example for making changes or making decisions about elements on a website or in a company. “Most dashboards are summaries, or a report to show how you have been doing over a recent period” (Interview 1). The dashboards are a summary which presents an overview on how the website is doing and the company should get the feeling that when they look at the dashboard they can see their business in one overview. “The online dashboard should reflect the activities of the organization and should form an overview of all activities in an organization and how the channel has performed during that specific period”
45
(Interview 4). The online dashboard is interpreted by its users. This interpretation forms a support for decisions that need to be made.
4.1.2
Objectives and types of dashboards
In the results of the qualitative research it becomes clear what should be presented and how this should be presented. According to the interviewees it is very important that you see the most important results on a weekly basis. The dashboard should be focused on the main Key Performance Indicators (KPI) which are the main elements for controlling the online channel by the organization. Examples of KPIs are measuring the pageview per page section or measuring the amount of new visits on the website. There are hundreds of KPIs, because everything that happens on a website could be measured by one of the analytical programs. Organizations focus on a few key elements in an organization, which can be for example more visitors on the website or more orders. Those key elements are translated in KPI and on these key points the whole online channel is directed. Therefore their online dashboard needs to be focused on these specific KPIs. Based on the obtained measurements on the KPIs, the organization takes action and makes decisions. An online dashboard is considered a tool to take decisions with and to maintain control. The online dashboard is used as an indicator, because it indicates how it goes with the state of affairs in an organization. It should "Inform and provide insight into the performance on a limited number of KPIs" (Interview 2). There should be a limited number of KPIs because of the importance of the focus in a dashboard. The information in an online dashboard should be clearly translated into actual insights preferably on one single page. How these KPIs should be presented in the online dashboards are by means of the use of the key objectives. There are three key objectives stand out and are most frequently mentioned by the interviewees, which are ‘focused’, ‘clear’, and ‘immediate’. These three words are translated from the Dutch interviews and therefore I shall explain them more specifically. With a focused dashboard is a dashboard meant that is concentrated on a few key points and it should not be too big with lots of information, but focused on only the most important information. Clear is the second key objective and by that is meant that the information that is visualized should be obvious and understandable. The last objective is immediate and by that is meant that the information should be readable at a glance. The insights in a dashboard should be clear right away. Now I explained the key objectives, it is
46
clear what is meant by them and I use these three translations in the rest of this thesis for these three key objectives. One of the negative experiences with online dashboards is, according to the interviewees, that they tend to expand enormously. It is a challenge to present a compact and focused dashboard: "If there are three metrics on a dashboard, then you are doing a great job, you can see at a glance if it goes well with the organization or not" (Interview 4). It should show only the most important numbers that give an indication on how the organization is doing compared to the chosen targets and goals. That is why it is important that the trend is included in online dashboard, to see if you making progress and reaching the targets. It should be clear and easy to read so it can be well understood what is actually shown on the dashboard. The insights should stand out, so it becomes easy to analyze these insights and make improvements, based on these analyses, to perform better. The internal interviewees of Conversion Company show that there is a desire to get a general type of dashboard like a standard online dashboard. Similarly the external interviewees mentione a desire for a general dashboard: "[...] looking for a total dashboard, where all the data is included, so you can see what happened in the past, just with one push on the button, and which goals are completed that week in total" (Interview 5). The interviewees of Conversion Company work with weekly dashboards, while interviewees of their clients also work with monthly or daily dashboards, but indicate that weekly online dashboards are most used. The context around the results that are shown on a dashboards is also very important, you should have the opportunity to compare different elements on the dashboards, according to all the interviewees. Therefore not only numbers, but also graphs should be comparable with each other to see if things are going good, not that good, or even bad. "It is a challenge when you are coping with a lot of data, to visualize that in such a way that clearly shows the right insights and that is looks nice" (Interview 3). "If something is good visualized, then you can get a lot out of this visualization" (Interview 5) and that is the challenge.
4.1.3
Level in organization and tasks
When I look at the responses to the question about the level in the organization where the online dashboards are being used, actually all interviewees responded that it is used by almost everyone. "Anyone who has a direct relation to the online channel" (Interview 2). A dashboard can be used internally at Conversion Company, but also externally by the clients of
47
Conversion Company or between those two parties by the channel managers (the function of channel manager is positioned at a client on behalf of Conversion Company, to control the clients channel). The dashboard is used by all the levels in the organization, but when I generalize this broad perspective, the interviewees agree that is it most often used between the operative level and the management level. “It is about accountability upwards and controlling the lower levels downwards” (Interview 5). From the operational employees to the employees who make (important) decisions such as a marketing manager, and even further to the highest level in the organization which are interesting insights for my research. Three interviewees explain that dashboards for lower levels are more detailed and dashboards that are presented to the management level contain more general information and are more visual accents and fancy designs. Three of the interviewees indicate that at management level, it would be ideal if you can show a general overview of what have been the insights, what are the actions that have been performed, and what is the result that is ultimately yielded. This is usually more textual than dashboards for lower levels in the organization which are more visual. Therefore the dashboards for the very high levels in the organization should be designed with nice visuals or just textual, in the form of an overview. And the lower level dashboards contain mainly visualizations, but on a more detailed level. The different dashboards need to fit well with the various activities of every level of the organization.
4.1.4
The dashboards
Below I provide a summary about the opinions of the interviewees when the four different dummy versions of the dashboards are proposed to them.
Dashboard 1
48
Figure 18: Screenshot dummy dashboard 1
In figure 18 a screenshot of the first online dashboard that is proposed to the interviewees is presented (see appendix 9.1.2 for the full dashboard). All the interviewees indicate that this dashboard is packed. There are too many graphs in this online dashboard. "Although it is one single page, when I look critically at this dashboard, a lot could be left out" (Interview 2). Through the amount of graphs and numbers, you cannot see which graph is the most important and which should be focused on. There is no overview anymore: "It is better if you include fewer graphs and really know what you want to focus on to control your organization" (Interview 4). The interviewees present a solution for the crowded dashboard, and that is with the use of different layers. The most important graphs should be included in a more 'general' dashboard. If you want to create a deeper analysis, another ‘deeper’ layer should provide more detailed information. The choice about the amount of graphs that should be included in an online dashboard, can be a very difficult decision according to one interviewee. Because some clients say they want to know everything and they want all the information included in the dashboard. "It's a process [...] what the objectives are or what the most important KPIs are, some customers claim that there are a lot and that all are important" (Interview 2). But the interviewees still think that there must be more focus on the most important KPIs. This dashboard contains also many numbers and percentages with a lot of information, but this complicates the interpretation and the comparison. It is not clear enough what all these numbers and percentages mean. To what are these numbers compared? Are they compared to a target, with the week before, or the average number? A clearer explanation 49
would solve this problem. Two interviewees think it is definitely useful if there is an extra explanation or legend included in the dashboard for example in the form of axes. Good elements about the dashboard are the columns. Because each column represents its own part of information, and that makes it simple to interpret. Colors work very well. Especially the use of red and green is considered pleasant by two interviewees. The period is clearly indicated in this dashboard, but one of the interviewees found the decision to report ten weeks arbitrary. “Why ten weeks, why not twelve or one year?” (Interview 2). As seen in figure 19, the graph with multiple bars is considered unclear by all interviewees. You cannot distinguish the different colors of the bars properly and also the different heights of bars are difficult to read. The interviewees find the colors of the bars not corresponding with the numbers above the bars, which is very confusing. "[...] you can barely see what the differences are. While sometimes five percent can be a very high achievement, it is impossible to read in this bar chart” (Interview 3).
Figure 19: Screenshot of one graph of dashboard 1, containing multiple bars
When summarizing these results of the opinions on dashboard 1, I noticed that including four-times-nine graphs in one dashboards does not count as a focused dashboard. Although the choice of how much graphs should be included is difficult, this amount is considered as too much and provides a lack of focus. The users cannot immediately see which graphs or numbers are the most important ones. Without the support of a legend or explanation this dashboard is not perceived as a clear one. Only the use of columns provides some explanation about how to read this dashboard.
Dashboard 2
50
Figure 20: Screenshot dummy dashboard 2
In figure 20 a screenshot of the second online dashboard that is proposed to the interviewees is shown (see figure 5 for the full dashboard). The tabs at the top provide a pleasant experience. According to the interviewees, you can navigate through the dashboard itself and you can search by using the tabs. It gives more control, what causes a peaceful feeling. Also the fact that the informative is divided into blocks makes it easy to read. This also gives a clear opportunity to compare. Different blocks "[…] you can really take the time to explore by clicking around. So first an overview and then zoom in with the use of the tabs"(Interview 5). Another positive point is the focus per site that makes it very clear. This shows a clear focus on KPIs. This dashboard gives a good overview. There is a clear connection within the total dashboard. But there are also some negative aspects about this dashboard. The lack of explanation makes it hard to interpret and compare the graphs and numbers. Also the scales are completely missing. Neither an x-axis nor y-axis have been included. At the top you can find the number of the week of this dashboard. However, in the graph itself it is unclear if it contains just a week or several weeks? What about the numbers above the graphs? What are they? Are they compared to an average, or to the previous week; it is not clear at all. One of the interviewees thinks that colors are generally very clear, but here: “[...] you know whether it is going better or worse. But it is still not clear compared to what.” (Interview 1). Any extra information about the element on this online dashboard is completely missing. The solution to both these problems of the axis and the numbers, would be an extra legend or explanation. Maybe it is perfectly clear for people who frequently work with this dashboard, but not for someone who independently looks at it for the first time. Without an explanation the interviewees agree that this online dashboard is very unclear. 51
Summarizing the results about dashboard 2, I conclude that the navigation function through the tabs are preferred. The option to navigate gives a positive feeling of control. The blocks that are used provide an immediate clear overview of this dashboard. The lack of explanation is totally missing in this dashboard, which makes it difficult to interpret and compare the different graphs and numbers.
Dashboard 3
Figure 21: Screenshot dummy dashboard 3
In figure 21 a screenshot of the third online dashboard that is proposed to the interviewees is shown (see appendix 9.1.2 for the full dashboard). Among the interviewees, this is the least favorite dashboard of the four. The tabs are considered as a positive explanatory element, just as in the second and previous dashboard. They provide the possibility to navigate through this dashboard. But there is generally insufficient explanation of this online dashboard. All interviewees have commented on the legend. A lot do not see it at first sigh, but if they do, they find it too small. Also, there are no axes which could provide some explanation, which makes this whole dashboard difficult to interpret. Moreover, the stacked area chart is hard to read according to all interviewees. "The trend is moving with all the different layers, so that is hard to read because you stack them" (Interview 5). Also the pie chart is not considered the most clear. But if numbers or percentages are included in the pie chart, that might bring some clarification. Also they would prefer to specify which week or month they want to compare. "The fact that you cannot 52
clearly see what it shown here" (Interview 2), does not make a successful online dashboard. They are now just "visualizations, but with minor insights" (Interview 4).
The opinions of this dashboard are partly the same as the second dashboard. The tabs are considered as clear and provide an overview, but it lacks explanation. Both the pie chart and the stacked area chart with different layers are not preferred, because they do not give the opportunity to compare the insights very well.
Dashboard 4
Figure 22: Screenshot dummy dashboard 4
In figure 22 a screenshot of the fourth online dashboard is shown that is proposed to the interviewees (see figure 6 for the full dashboard). This online dashboard attracts a diverse opinion from the interviewees. Some find it very clear in contrast to others that experience this dashboard as very unclear. This dashboard contains fewer graphs which almost all interviewees find positive. It has a general overview and could be suitable for immediate reporting. The possibility to export the dashboard to different types of documents is also a good elements. First the overview and then the tabs are handy, so you can navigate through the dashboard itself and that gives you more possibilities. This possibility to navigate makes the dashboard very user friendly. You can also choose which weeks you want to compare. Among the negative aspects of this online dashboard is the use of color. The information and colors do not correspond, which makes it difficult to read information off and causes a chaotic feeling. Also all the lines that run through the same graph, especially in the lower left corner, make it hard to read. A solution could be to "[…] choose one metric and then, if necessary, with a dropdown menu, you can select the other lines to compare" 53
(Interview 2). The interviewees get the idea that the focus of this dashboard is lacking, which makes the overview unclear. The literal dashboards, on the bottom right are not considered by all interviewees as clear. Some find them unclear because there is no clear scale. “Red and green is clear, that is good or bad. But then what is orange? A danger zone? But when is it a danger zone and when does it becomes bad?" (Interview 1). Without explanation these illustrations do not add something to the insights of this dashboard.
In this dashboard the possibility of navigating is considered positive again. Also the possibility to export the report is a good element. The use of color is perceived as chaotic and without clear explanation difficult to understand. The illustrations used in this dashboard are not very clear and therefore do not provide clear insights.
4.1.5
Word Cloud
Combining all the interviews together, I formed a word cloud (see figure 23). Below you see the word cloud in Dutch because all the interviewees I interviewed preferred to do the interview in their own native language. I excluded the main words ‘dashboard’ and ‘dashboards’ to provide an overview of key words that arose around this subject. Also I excluded the words ‘eigenlijk/actually’, ‘gewoon/just’, ‘echt/really’, and ‘wel/sure’ because these four words are typical Dutch fillers, to prevent a distorted result.
Figure 23: Word Cloud, containing all qualitative results (in Dutch).
54
Key words that stand out are (in Dutch alphabetical order): ‘data/data’, ‘duidelijk/clear’, ‘heel/very’, ‘goed/good’, ‘informatie/information’, ‘organisatie/organization’, ‘niveau/level’, ‘week/week’, ‘zien/to see’. The use of the words ‘data’, ‘organization’ and ‘week’ are definitaly concepts that are closely connected to the use of dashboards and are represented largely in the word cloud. Some smaller words such as ‘belangrijk/important’ and ‘vergelijken/compare’ are indeed corresponding with the main insights that are obtained from this qualitative research. The important data should be shown in an online dashboard to obtain a focused view of the most important KPIs and the desire to compare different insights, graphs, and numbers is also reflected. Through the word frequency the interpretation of the interviews are confirmed. This word cloud does not point towards new directions, but summarizes the qualitative research results.
Summarizing the results of the qualitative research, clear insights are provided and provide answers on the research questions. Als it provides a good support for the delimitation of the quantitative research. I obtained insights about the use of an online dashboard and the key objectives such as focus, clarity, and immediately. Also the desires on objectives, explanation, navigation, and color, provides answers on the first and second sub research question about the context, objectives and the general layout of a dashboard. Mainly elements that add something to the key objectives focused, clarity, or immediate insights, are considered positive by the interviewees of the qualitative research. With these results in mind the quantitative research is created (see section 3.4.1).
4.2 Results quantitative research Below I demonstrate my quantitative data, gathered from the online survey tool ‘Qualtrics’. After filtering the quantitative data, I now have a focused data set, and continue summarizing my results. The summary presents the results per information type. The first information type that is discussed is comparing magnitudes of independent values. This is followed by the second information type, the particular comparison with different entities. Continued by the particular comparison with equal entities, which is the third information type. The fourth type of information is the change over time and the last and fifth information type shows the overall trend. In each section of the different information types I elaborate on the time of interpretation, because in my opinion the time of interpretation is the time that a respondent 55
needs to really understand the graph, because I believe that the time a respondent needs to provide an answer is the same time a respondent needs to understand the graph. I also compare the last click with the page submitted, because this could show how secure a respondent is about the answer that he or she gave. In my opinion the quicker one submits the page after the last click of the answer, the more secure one is about that answer. Also the different opinions on the different visualization forms are discussed. Which visualization forms are mostly preferred and which are most disliked, according to the respondents? And every section is concluded by the results of correct versus incorrect answers, because not only the interpretation is important but it should also be the right interpretation.
4.2.1
Results: Information type 1, comparing magnitudes of independent values
With this type of information, called Information type 1, the subset is created with question 5, 14 and 24. Containing visualization types ‘bar’, ‘line’, and ‘points’. Looking at the time of the last click, the visualization ‘bar’ has the quickest time and therefore I can conclude that this visualization is interpreted the quickest compared to the other two (see figure 24 and table 2). With eighteen seconds and eighty-five milliseconds the visualization with bars claims the first place. This is followed by the visualization with points on the second place, with three seconds and forty-six milliseconds more. Therefore I can conclude that a combination of comparing magnitudes of independent values with a visualization with bars is most optimal.
Figure 24: Visualization form with bars is interpreted the quickest with comparing magnitudes of independent values
Comparing the last click with the page submitted, the visualization with points is submitted the quickest after the last click (see table 2). With two and a half seconds points is the quickest submitted, but not significantly quicker that the other two visualizations.
56
Results Interpretation time Information type 1 Bar
18,85
2,7
Line
25,43
Point
22,31
0
5
10
15
20
Time last click, with actual numbers Time page submit
2,58
2,5
25
30
Interpretation time in seconds Table 2: Results interpretation time of information type 1, comparing magnitudes of independent values, in seconds
In the second part of the quantitative research the question ‘Which graph is in your opinion the most clear if you compare independent values?’ is asked and all the visualization types are shown. Seventy percent selected the visualization with bars as most clear in combination with comparing independent values (see table 3). Only three and a half percent of the respondent selected that they dislike bars the most. Second place, with twenty percent, is earned by line and with only five percent, points results in third and last place. Twentyseven percent of the respondents choose the line visualization as least clear visualization, closely followed by twenty-five percent of the respondent that selected points as least clear.
Visualizations Prefered vs Disliked 80 60 40
Preferred
20
Disliked
0 Bar
Line
Points
Table 3: Results visualizations of information type 1, preferred versus disliked
To give value at the right interpretation of the graph, looking at the answers reveals a lot (see table 4). Most of the respondents answered the questions of this information type correctly, varying from eighty-one percent at the line visualization to eighty-eight at the points visualization. In between is the bar visualization and is answered correctly eighty-four percent. No significant specific conclusion result from these answers.
57
Results Correct vs Incorrect answers Information type 1 Bar
84,21%
Line
80,70%
Points
88,46% 0%
20%
40% Correct answer
60%
15,79% 19,30% 11,54%
80%
100%
Incorrect answer
Table 4: Results correct versus incorrect answers of information type 1, in percentages (total 100%)
4.2.2
Results: Information type 2, particular comparison with different entities
With this type of information, called Information type 2, the subset is created with question 1, 7, 13, 16, 21, 25, 29 and 32. Containing visualization, ‘lines’, ‘points’, ‘barline1’, ‘barline2’, ‘barpoint1’, ‘barpoint2’, ‘linepoint1’, and ‘linepoint2’. Observing the time of the last click, the visualization lines has the quickest time and therefore I can conclude that this visualization is interpreted the quickest compared to the others (see figure 25 and table 5). With seventeen seconds and sixty-three milliseconds the visualization with lines comes first place. This is followed by the visualization with linepoint2 on the second place, with two seconds and seventy-seven milliseconds more. Therefore the combination of a particular comparison with different entities with a visualization lines is most optimal for online dashboards. Notable is that lines is ‘by far’ the best visualization for this type of information compare the results with the other visualizations, with two seconds and seventy-seven milliseconds.
Figure 25: Visualization form with lines is interpreted the quickest with a particular comparison with different entities
Comparing the last click with the page submitted, the visualization barpoints2 is submitted the quickest after the last click (see table 5). It took an average of forty-two milliseconds to submit the page, significantly quicker that the other visualizations. 58
Results Interpretation time Information type 2 Lines
17,63
2,12
Points
23,38
Barline1
22,69
Barline2
20,96
Barpoint1
21,45
Barpoint2
21,67
2,4 2,67 0,42
20,79
2,79
Linepoint2
20,4
2,73
5
10
Time page submit
2,18
Linepoint1
0
Time last click, with actual numbers
1,88
15
20
25
30
Interpretation time in seconds Table 5: Results interpretation time of information type 2 particular comparison with different entities
In the second part of the quantitative research the question ‘Which graph is in your opinion the most clear if you compare particular values?’ is proposed in combination with all the visualization types that are shown. Thirty-four percent of the respondents, selected lines as the most clear visualization of a particular comparison with different entities, while three percent find it the least clear one (see table 6). At the second place of most clear, considered by thirty-one percent, is the visualization barline2, but five percent does not agree and find this visualization the least clear. Thirdly barline1 is liked by twenty-one percent of the respondents, only two percent does not agree and dislikes this visualization. The three visualizations that resulted in the least clear visualizations are respectively points, barpoint1 and barpoint2. Remarkable is the visualization points that is disliked the most but at the same time is considered at position four of most clear visualization, but with a slightly big gap between position three and four.
59
Visualizations Prefered vs Disliked 40 30 20 10 0
Preferred Disliked
Table 6: Results visualizations of information type 2, preferred versus disliked
To evaluate the right interpretation of the graph, focusing on answers is showing that there is a lot of fluctuation between the answers of the different questions (see table 7). Remarkable is that the first question, with visualization barline1 is both answered correct and wrong the same amount of times. It is also the only question in this information type subset that is answered with ‘I don’t know’. Also the questions with visualization linepoint1 and points are wrongly answered thirty-four percent and forty percent of the total answers of that question. The rest of the questions is correct answered with eighty until ninety-six percent. The most correct answers are given at the barpoint1 question.
Results Correct vs Incorrect answers Information type 2 Lines
94,64
Points
60,38
Barline1
5,36 39,62
41,94
16,13
41,94
Barline2
80,39
19,61
Barpoint1
95,92
4,08
Barpoint2
94,12
5,88
Linepoint1
66,00
Linepoint2
34,00
81,25
0%
20% Correct answer
40% Incorrect answer
60%
80%
18,75
100%
I don't know' answer
Table 7: Results correct versus incorrect answers of information type 2, in percentages (total 100%)
60
4.2.3
Results: Information type 3, particular comparison with equal entities
With this type of information, called Information type 3, the subset is created with question 2, 6, 9, 11, 15, 19, 22, 26, 30 and 33. The subset is created with the ten visualizations: ‘bar’, ‘lines’, ‘points’, ‘stacked’, ‘barline1’, ‘barline2’, ‘barpoint1’, ‘barpoint2’, ‘linepoint1’, and ‘linepoint2’. Focusing on the time of the last click, the visualization with a stacked area has the quickest time (see figure 26 and table 8), with seventeen seconds and fifty-six milliseconds, and therefore I can conclude that this visualization is interpreted the quickest compared to the others. Closely followed by the lines visualization (see figure 27), which resulted in seventeen seconds and sixty-six seconds, meaning a difference of only ten milliseconds. Third is barline1, with eighteen seconds and ninety-four milliseconds. Only three resulted visualization finish under twenty seconds. Therefore a combination of a particular comparison with equal entities in combination with a stacked or a lines visualization would be most optimal for online dashboards.
Figure 26: Visualization form with a stacked area is interpreted
Figure 27: Visualization form with lines is interpreted the
the quickest with a particular comparison with equal entities
second quickest with a particular comparison with equal entities
Comparing the last click times with the page submit times, the visualization bars is submitted the quickest after the last click (see table 8). It took an average of one second and sixty-seven milliseconds. Followed by linepoint2 and barline1 with one second eighty-six and one second eighty-eight to submit the page. Further no significant results in this part.
61
Results Interpretation time Information type 3 Bars
27,61
Lines
17,66
2,28
Points
Time page submit
27,61
Stacked
17,56
Barline1
2,91 2,43
18,94
Barline2
1,88
22,44
Barpoint1
23,38
Barpoint2
23,18
Linepoint1
20,56
Linepoint2
20,32
0
5
Time last click, with actual numbers
1,67
2,56 2,64 2,05 2,69 1,86
10 15 20 25 Interpretation time in seconds
30
35
Table 8: Results interpretation time of information type 3 particular comparison with equal entities
In the second part of the quantitative research the question ‘Which graph is in your opinion the most clear if you compare particular values’ is proposed in combination with all the visualization types that are shown. With a great majority, bars is liked by forty percent of the respondents (see table 9). In second place barline2 is considered most clear by twenty-six percent. Lines come in on third place with seventeen percent as most clear. Mostly disliked is points, with nineteen percent. Also barpoint2 and stacked are considered least clear respectively fourteen and twelve percent.
Visualizations Prefered vs Disliked 40 30 20 10 0
Preferred Disliked
Table 9: Results visualizations of information type 3, preferred versus disliked
Looking at the right interpretation of the graph, I focus on the answers (see table 10). Interesting are the answers on the first question with a points visualization, the answered are 62
more incorrect than correct; sixty-seven percent over forty-three percent. When making a review on this specific question I find ambiguity. If I read the question, the answer is not obviously ‘yes’ or ‘no’, it could be either one of them. Therefore I decide to leave this question out of the results. Because of the ambiguity of this specific question I cannot compare them to the other questions and visualizations of this information type. Continuing, the visualizations stacked and barline1, have a percentage of ninety-six and ninety-eight of correct answers. In contrast to the questions with visualizations barline2, barpoint2, linepoint1 and linepoint 2, that have between twenty-three and thirty-four percent incorrect answers.
Results Correct vs Incorrect answers Information type 3 Bars
72,09
27,91
Lines
85,45
Points
14,55
43,18
56,82
Stacked
96,08
Barline1
3,92
98,33 1,67
Barline2
76,36
Barpoint1
23,64 85,42
Barpoint2
78,72
Linepoint1
21,28
65,96
Linepoint2
34,04 75,56
0%
20%
40% Correct answer
60%
14,58
24,44
80%
100%
Incorrect answer
Table 10: Results correct versus incorrect answers of information type 3, in percentages (total 100%)
4.2.4
Results: Information type 4, change over time
With this type of information, called Information type 1, the subset is created with question 3, 10, 18 and 28. Containing visualization ‘bar’, ‘line’, and ‘points’, and ‘stacked’. Focusing on the time of the last click, the visualization with a stacked area has the quickest time, with twelve seconds and seventy-two milliseconds, and therefore I conclude that this visualization is interpreted the quickest compared to the others (see figure 28 and table 11). In second place is the bar visualization (see figure 29), which resulted in thirteen seconds and thirty-seven milliseconds, only sixty-five milliseconds. Noting there are lots of 63
results found that are zero for the last click, it could be interesting to concentrate on the page submit time. But I still find the stacked area visualization on the first place, with eighteen seconds and thirty-six milliseconds followed by the visualization with bars, one second and forty milliseconds. Therefore, with two measure points a combination of a visualization of change over time works best for the interpretation with a stacked or a bars visualization for online dashboards.
Figure 28: Visualization form with a stacked area is interpreted
Figure 29: Visualization form with bars is interpreted he second
the quickest with change over time
quickest with change over time
Looking at the last click time, compared with the page submit time, the visualization lines is submitted the quickest with an average of five seconds and six milliseconds (see table 11). This is followed by the stacked visualization with fifty-eight milliseconds more, then the fourth and fifth place bar and points, with six seconds thirty-nine and six seconds forty-nine. No significant results were extracted.
Results Interpretation time Information type 4 Bar
13,37
Line
16,41
Points
Time last click, with actual number Time page submit
6,39 5,06
17,09
Stacked
12,72
0
5
10
6,49
5,64
15
20
25
Interpretation time in seconds Table 11: Results interpretation time of information type 4 change over time
In the second part of the quantitative research the question ‘Which graph is in your opinion the most clear if you look at change over time?’ is proposed to the respondents. Forty 64
percent of the respondents liked bars the most as a clear visualization (see table 12). Five percent of the respondents did not agree and consider bars the least clear visualization to visualize change over time. The second one is lines where thirty-six percent of the respondents consider this as a clear visualization. Five percent disliked lines and considered this visualization the least clear for change over time. The visualization stacked is liked by twenty-two percent of the respondents, but is also disliked by ten percent. However points is considered as the least clear visualization by twenty-five percent of the respondents, which equals a quarter of the respondents that answered this question, and zero respondent liked this visualization.
Visualizations Prefered vs Disliked 40 Preferred
30
Disliked
20 10 0 Bars
Line
Points
Stacked
Table 12: Results visualizations of information type 4, preferred versus disliked
Looking at the correct interpretation of the graph, focusing on answers I see interesting results (see table 13). All questions are answered mostly correct. The stacked visualization is answered correctly with a percentage of eighty-two, as well as the line visualization. Followed by the points visualization with sixty-eight percent correct answers and the bar visualization with sixty-five percent.
Results Correct vs Incorrect answers Information type 4 Bar
65,00
Line
35,00 82,00
Points
68,00
Stacked
32,00 82,00
0%
20%
40% Correct answer
60%
18,00
80%
18,00
100%
Incorrect answer
Table 13: Results correct versus incorrect answers of information type 4, in percentages (total 100%)
65
4.2.5
Results: Information type 5, the overall trend
With this type of information, called Information type 1, the subset is created with question 4, 8, 12, 17, 20, 23, 27, 31 and 34. Containing visualization, ‘bars’, ‘lines’, ‘points’, ‘barline1’, ‘barline2’, ‘barpoint1’, ‘barpoint2’, ‘linepoint1’, and ‘linepoint2’. Looking on the time of the last click, the visualization bars has the quickest time, with twelve seconds and forty-eight milliseconds, and therefore I can conclude that this visualization is interpreted the quickest compared to the others (see figure 30 and table 14). This is closely followed by the barline1 visualization, which resulted in twelve seconds and sixty-seven milliseconds, meaning a difference of only nineteen milliseconds. In third place is linepoint1, with thirteen seconds and seventy-four milliseconds. Only three resulted visualization finish under fourteen seconds. Therefore I conclude that a combination of a particular comparison with equal entities in combination with a bars or a barline1 visualization is most optimal for online dashboards.
Figure 30: Visualization form with bars is interpreted the quickest with the overall trend
Comparing the last click times with the page submit times, the visualization linepoint is submitted the quickest after the last click (see table 14). It took an average of one second and eighty-two milliseconds. This is followed by linepoint2 with two seconds and six milliseconds with no significant results in this part.
66
Results Interpretation time Information type 5 Bars
12,48
Lines
2,62
14,09
Time last click, with actual numbers
2,43
Points
18,18
Barline1
12,67
2,52
Barline2
17,22
Barpoint1
16,86
Barpoint2
15,21
Linepoint1
13,74
Linepoint2
14,37
0
5
10
Time page submit
2,15
2,48 3,17
2,56
1,82 2,06
15
20
25
Interpretation time in seconds
Table 14: Results interpretation time of information type 5 the overall trend
In the second part of the quantitative research the question ‘Which graph is in your opinion the most clear if you look at the overall trend’ is proposed in combination with all the visualization types that are shown. With twenty-six percent, barline2 is chosen to be the most clear visualization to visualize the overall trend (see table 15). Bars comes in at second place with twenty-three percent of the respondents, but is also considered the least clear by seven percent of the respondents. With sixteen and fifteen percent, lines and barline1 are considered third and fourth most clear of the visualizations. With twenty-six percent, points stands out at the least clear visualization for visualizing the overall trend. This is followed by a less than seven percent for barpoint2 and the mentioned seven percent of bars.
Visualizations Prefered vs Disliked 30 20 10 0
Preferred Disliked
Table 15: Results visualizations of information type 4, preferred versus disliked
67
Focusing at the right interpretation of the graph and focusing on the percentage of correct answers there are interesting and divided results (see table 16). Six of the nine questions are incorrect answered by thirty-one until forty-two percent. Barpoint1 and bars which have the highest percentage correct answers, with eighty-two and ninety-two percent. Remarkable is that in seven of the nine questions, that women answered more incorrectly than correctly, whilst looking at men this is only in two questions the case.
Results Correct vs Incorrect answers Information type 5 Bars
92,19
Lines
79,03
Points
20,97
58,00
Barline1
42,00 64,91
Barline2
35,09
56,86
43,14
Barpoint1
82,26
Barpoint2
69,23
Linepoint1 Linepoint2
39,66 69,49
20%
40% Correct answer
17,74 30,77
60,34
0%
7,81
60%
30,51
80%
100%
Incorrect answer
Table 16: Results correct versus incorrect answers of information type 5, in percentages (total 100%)
4.2.6
Results: General Dashboards
In the second part of the online survey questions about the complete dashboard are proposed. The question that is focused on the amount of graphs placed in the dashboard, asked the preference of the respondents. Almost eighty percent of the respondents considered dashboard 1, with a layout of four graphs times four the most clear which is the smallest layout of the four proposed dashboards (see table 17). Sixty-seven percent, considered dashboard 4 the most clear, with four times five graphs, this amount gets the second place. Both eighty-three percent voted the third dashboard on the third place and the second dashboard on the fourth place. The third dashboard has four times six graphs and the on fourth place four times nine graphs. 68
Table 17: Results general dashboards, amount of graphs preferred in percentages
Another question is proposed to measure how long the respondents want to spent on an online dashboard. The majority answered this question with five, ten or fifteen minutes (see table 18). Twenty-nine percent answered ten minutes, twenty-three fifteen minutes and twenty answered five minutes. Fourteen percent would spend half an hour on an online dashboard, and only seven percent twenty minutes. Eight percent answered ‘Other’ on this question. Two answered here that ‘it depends on what your role is’.
69
Amount of time 30 20 10 0 5 10 15 20 30 minutes minutes minutes minutes minutes
other
Table 18: Results preferred amount of time spend on an online dashboard in percentages
4.2.7
Results: Amount of Numbers
I also focused on the amount of numbers, used in one graph. Thirty-nine percent considered the first graph with six numbers the most clear, while thirty-three considered the second graph with three numbers the most clear (see table 19). The third place is for the third graph with four number. Graph four with only two numbers can be considered by far the least clear, with seventy-eight percent.
Table 19: Results amount of numbers general dashboards, amount of graphs preferred in percentages
70
Summarizing the results of the quantitative research, it becomes clear that the visualizations with bars, lines, and stacked areas show the best results in combination with the five different information types. Interesting is the contradiction that pops up between the quantitative results when I look at the interpretation time and the answers on the opinion questions where I ask which visualization do they prefer and consider as most clear. Apparently what they think that is clear scores differently that how they perceive a clear visualization, shows the contradiction between the times and the preferences. With few results I can also conclude from this quantitative part how the general dashboard should look like. The participants prefer the least amount of graphs and the most amount of numbers. It also resulted in five to fifteen minutes that the respondents want to spend on analyzing the online dashboards.
5 Interpretation and Discussion Below I summarize the key findings of the qualitative and quantitative research that are meaningful for answering the research questions. Followed by a discussion where I relate my research to the literature and the broader debate. I connect the key findings to literature per different author. I conclude with presenting my new method for the most optimal online dashboard design.
5.1 Interpretation qualitative research When I summarize the results of the qualitative research it becomes clear what the objective and desires are concerning the design of online dashboards beginning with the desire for a general dashboard, containing an overview and where the keywords focused, clarity, and immediately are adequate. Those three keywords are returning in almost all the objectives of an online dashboard. A dashboard should reflect the activities of an organization and how their online channel has performed during a specific period. It should inform and provide insights into the performance on a few focused goals, in other words how the organization is doing compared to the chosen targets and goals as set by that organization. It is very important that the information in an online dashboard is clearly translated into actual insights. Without explicitly saying it, this translation of information into clear insights is actually in other words the
71
visualization process. They all desire a clear dashboard, but the only way that a dashboard will be clear is if you focus on the visualization-step of information. Moreover one of the objectives that stands out is the opportunity to compare the different insights with each other. A comparison demonstrates the performed activity of a certain period with another period. Also the possibility to navigate through the dashboard is a key objective, for example with several tabs. With the possibility to navigate the focus on the most important graphs remains, but more details can be obtained through this navigation. It gives a feeling of control and looks clear. The keywords are returning in the amount of graphs. Fewer graphs are considered as positive, because then one can work with a focused dashboard and it becomes immediately clear which insights are most important. Some explanation or legend should also be included in a dashboard, in order to make it readable for everyone who wants to work with it. An explanation or legend could provide more support to online dashboard, because it changes it immediately to a clear visualization. The colors green and red are mentioned a lot. Those two colors are considered to work really well, but implementing orange is considered to be confusing, especially without further explanation. The use of more different colors is perceived as chaotic.
5.2 Interpretation quantitative research Below, I only discuss the quantitative results that could be meaningful for the research questions or stand out of the rest of the results. As discussed in the results, I looked at the different elements that influence the most optimal visualization. Considering the interpretation time, the time between the last click and the page submit, the opinion about the different visualizations of the respondents, and the percentages of correct versus incorrect answers, the most optimal visualization form comes forward. In the sections about the best combination of visualization form with the specific information type, I only present the most important and influential elements. Information type 1: Comparing magnitudes of independent values Merging part one and part two of this qualitative research provides the outcome for the information type ‘comparing magnitudes of independent values’. With the quickest interpretation time and a selection of seventy percent of the respondents, bars provides the best combination if you want to compare magnitudes of independent values in online dashboards. 72
Information type 2: Particular comparison with different entities Combining the first and the second part of the quantitative research provides the outcome for the information type two, which is the particular comparison with different entities. The quickest interpretation is made with the visualization with lines and this visualization is also considered as the clearest visualization for this information type. As well as visualizations barline1 and barline2, in second and third place, but have forty-two and twenty percent incorrect answers. With a ninety-five percent result of correct answers, in combination with the quickest interpretation and very preferable, lines results in the best combination when visualizing a particular comparison with different entities in online dashboards. Information type 3: Particular comparison with equal entities Merging part one and part two of the quantitative research show interesting results for information type ‘particular comparison with equal entities’. In the first part, results show that the quickest interpretation is made in combination with the visualization stacked, and has a correct answer percentage of ninety-six. However in the second part, the visualization stacked is considers as third least clear visualization (out of ten). Lines is secondly quickest interpret and is considered third best and clear visualization. With the contrasting results noticed at the stacked visualization, the save option is to choose lines for the optimal combination between visualizing and a particular comparison with equal entities. Information type 4: Change over time A combination of part one and part two of the quantitative research for information type four, which is change over time, shows that the quickest interpretation is made in combination with visualizations stacked area or bars. With two measure points, they provide the quickest interpretation. They are also submitted the quickest. Bar is chosen by forty percent of the respondents to be the most clear visualization for change over time. Stacked is liked twentytwo percent, but also disliked ten percent. Notable is that the stacked visualization has the most correct answers with eighty-two percent. The bar visualization had the least correct answers, with sixty-five percent. This contradiction of the quickest interpretation in combination with the least amount of correct answers, does not make the bar visualization very convincing. Looking at the results, stacked is interpret after bar the quickest and has the most correct answers, but is not chosen by the respondents as very clear. The contradiction of what the respondents prefer and how they result in the first part of the research is a 73
remarkable outcome. Therefore choosing stacked as the optimal combination for visualizing change over time, since I consider part one of the research more trustworthy. Information type 5: The overall trend Merging the first part and the second part of the research provides interesting results for information type five, visualizing the overall trend. The quickest interpretation is made in combination with visualization bars, and has the highest percentage of correct answers with ninety-two percent, and on top of that is chosen as second most clear visualization in combination with visualizing the overall trend. Followed by barline1 that is secondly interpreted the quickest and thirdly chosen as most clear by the respondents. Therefore I can conclude that a visualization with bars of a combination of bars and lines is the most optimal for visualizing the overall trend in online dashboards. A side-note that I want to make for information type five, the overall trend, is the rather high percentage of incorrect answers on all nine questions. With exception on the visualization of bars. A possible reason is the inexperience of the respondents with this type of question and visualization. There is a plausible opportunity that this information type or graph is not read correctly as a cumulative graph, but interpreted differently and therefore wrongly.
General Dashboard The obtained results of the general layout of a dashboards show that a less amount of graphs is preferred by the respondents. This corresponds with the desire of an focused dashboards, where only the most important insights and information is shown. With a divided opinion on the time spend on a dashboard, most of the respond preferred to spend five or ten minutes on a dashboard. But considering the opinions of the qualitative part, it really depends on your function and how much your work is related to the online dashboard, for the time that should be spend on an online dashboard.
Amount of Numbers Looking at the amount of numbers that should be included in a graph, the merge of the qualitative and quantitative research provide a combined result. Numbers are considered as extra, yet focused information, and therefore as positive. But there are preconditions to this fact. The numbers should be clear immediately, with the support of an explanation or a legend. For example to what are the numbers compared? Also the numbers should be comparable with other numbers, and again this should immediately be clear. 74
5.3 Discussion In this section I connect the key findings of my thesis with the wider debate and literature. Below I present the discussion, beginning with elaborating on big data and online dashboards in general. This is followed by the seventh principle of Tufte that I added to his principles as a guidance during the design process. Continuing with the sixth basic principle of Mazza that needs to be added for the general information visualization process. Then I elaborate on Card, Mackinlay, and Shneidermans with their two principles to amplify cognition that should be included into the visualization process. This is followed by how the information seeking mantra of Shneiderman that can be applied on the layout of a general dashboard.
Big data and online dashboards As described by danah boyd and Kate Crawford it is critical how people deal with the possibilities of big data. Current decisions could have a great influence on our future, they claim. That is why this research is so important. More empirical research should be done, focused on data visualization, because that is the best solution of dealing with all the data that could be obtained automatically. Just like Bollier claims, visualization is the best tool to identify meaning out of all the data. This research, where I focus on the visualization form online dashboards, shows that these dashboards are developed mainly in practical ways, and this does not correspond with a lot of objectives and desires. Therefore this research is a contribution to the wider debate because it shows more focus points that could and should be developed. Looking at the discovered focus points in a visualization such as an online dashboard, improving them provide better correlations and better ground rules for exploring the visualizations in a way that could uncover significant meaning hidden in big data, which are important features according to Bollier. With the outcomes of the research I provide a new method which could be used to reveal patterns and insights and show information in a way that is better understandable for many people in the age of big data. This again is an important focus point according to danah boyd and Kate Crawford, just like the possibilities to bring data into new relations. Keeping in mind that this research is focused on the visualization tool of online dashboards, this research can be seen as a new method with basic principles for a more complete and optimal online dashboard design. Tufte’s seventh principle: As guidance during the design process of data visualization 75
Where Tufte describes his six main principles for a controlled comparison and as guidance during the design process of data visualization, I want to add a seventh principle. The six principles are definitely important and support the design process, but there should be a last principle added at the end of this list of six principles and focuses on sense making and understanding. The seventh ground rule is crucial in the process of statistical evidence and should not be left out in the designing process of graphics, which could be applied on online dashboards. Tufte claims that a visualization should reveal its information, therefore a strong focus on the interpretation of this visualization is required. As resulted from this research people could think that they experience a specific visualization as a clear visualization, but when they try to interpret that visualization, different results are shown. Like the remarkable negative opinion about the stacked area charts in the qualitative research section and opinion questions in the quantitative research, but with notable high percentages of interpretation time and also high percentages of correct answers. This shows that what they say that they prefer is not always the most optimal way of visualizing. Therefore a seventh principle that focuses on perception and the ability of cognition, sense making and understanding should be added to the list of six principles of Tufte. It is important to focus on this part during the design process because the way how the visualization is interpreted is crucial. The interpretation reveals the information hidden in the visualization. Without this last principle it is risky to use visualization for making important decisions, like the decision made for the launch of the space shuttle in 1997 with horrible following results. If visualizations are used for decision making, which they are in the case of online dashboards, the seventh principle that focused on the interpretation and perception cannot be left out. Mazza sixth basic principle: important for the general information visualization process A part of the definition that Mazza gives for information visualization is “The use of computer-supported, interactive visual representations of data to amplify cognition”, which shows that he agrees on the importance of cognition that needs to be amplified. But when looking at his five basic principles that need to be considered at the information visualization process, there is little attention for this element. ‘To amplify cognition’, is the main purpose of visualization, but these five principles mainly focuses on the goal of the visualization and type of data. The last principle focuses on the representation, but missing is the step after that, which is the interpretation, sense making, and understanding step. In the field of online dashboards this step is essential. Otherwise the information-flow of data and information that is interpreted, what leads to a certain result or action, lacks attention on the interpretation part. 76
But I believe that also in a broader perspective of information visualization this step cannot be left out. How well a visualization is interpreted indicates how clear the visualization is and to what extend this information visualization process is succeeded. Mazza quotes that the purpose of visualization is to display facts in such a way that others could understand them better, therefore it is crucial to include this sixth principle in the visualization process of Mazza. Card, Mackinlay, and Shneiderman their ways to amplify cognition in online dashboards When talking about amplifying cognition Card, Mackinlay, and Shneiderman write about six ways how to perceive this. They claim that the goal of information visualization is to provide insights, discovery, decision making and explanation. Exactly the key aspects of online dashboards. Two of the six ways to amplify cognition are especially beneficial if you include them in the visualization process of online dashboards. The first of those two is focus on cognition. Grouping information together makes it easier to find information, keeping in mind the use of chunks of visual data in the working memory where Card, Mackinlay, and Shneiderman write about. And the second one is the detection of patterns in the visualization. Both ways amplify cognition and ensure that the information and knowledge hidden in the visualization is perceived and understood. It makes the visualization immediately clear, because of the focus of the design. Therefore by including these two tricks of Card, Mackinlay, and Shneiderman in the visualization process, ensures the key objectives of online dashboard that are obtained out of the research of this thesis. Ben Shneiderman’s Information Seeking Mantra in a General dashboard Combining the qualitative and quantitative results, I can conclude that the respondents prefer fewer amounts of graphs over dashboards with more amounts of graphs. Focus is one of the key elements. This claim is supported in theory, in the outcomes of the qualitative research, and in the quantitative results. Notable is the tendency to expand enormously, while at the same time focus is one of the main objectives. It reminds me of the history of online dashboards, where a lack of focus was found and attention lay more on special features to attract the clients. Currently a tendency is noticed towards expanded dashboards, caused by the desires of the clients. However this is what is thought. Clearly this research shows that both sides, a company that is offering online dashboards and clients that are using these dashboards, actually desire a very focused dashboard. Both results of the qualitative and quantitative dashboard reveal this desire. Sometimes an overview is just enough, but most of 77
the times a need to zoom in on specific deeper details is necessary to understand underlying results better. For example a manager spends ten minutes, as a channel marketer could spend more than an hour. Therefore there is a need for different dashboards layers for different groups, because it depends on how far someone wants to down drill the data. Beginning with a general dashboard layer with the overview and with a tab-option that shows more data and can be used to navigate through the dashboard and find more detailed information and insights. Looking at this need for different layers within the dashboards I see a strong connection with the Information Seeking Mantra of Shneiderman. The mantra is one of the strengths of information visualization and therefore should be applied on online dashboards more often in my opinion. To begin with an overview, like a general dashboard, and continue by zooming, for a deeper analysis on the information. After this analysis details on demand could be found in the analytical database. To respond on the resulted desire, this powerful information seeking mantra should be included in the design process of online dashboards. Ideal design for an online dashboard would be a combination of the theories Tufte, Mazza, Card, Mackinlay, Shneiderman and Few. All wrote very interesting theories such as the guidance during the design process, principles for the information visualization process, how to amplify cognition, the information seeking mantra and general information about online dashboards. However the power lies in combining them. Currently, in my opinion, there is no perfect theory that is scientifically tested, but with merging different theories with research, I provide one. The gap between practical experience, theory and scientific testing could be closed by combining the theories of Shneiderman, Tufte, Mazza, Card, Mackinlay, and Few, added by the perceived results of this thesis. Together they form the basis for the design of an optimal online dashboard.
5.4 New Method for optimal online dashboard design Below I present my new method that is required to provide the most optimal visualization of abstract data in online dashboard design. First I demonstrate the requirements for the general dashboard. Including these focus points into the online dashboards design provides an optimal dashboard according to practical experience in combination with scientific testing. Continued with table 20, that is focused on the visualizations of graphs, divided into five different information types using the most optimal visualization form in combination with a suitable information type results in the most optimal interpretation. 78
Obtaining optimal visualization result if you focus on the bullet points below while designing an online dashboard: During the visualization process keep the keywords focus, clarity, and immediately in mind, these are adequate. These three words are translated from the Dutch interviews and therefore I shall explain them. With a focused dashboard is a dashboard that is concentrated on a few key points meant and it should not be too big with lots of information, but focused on only the most important information. Clarity is the second key objective and by that is meant that the information that is visualized should be obvious and understandable. The last objective is immediate and by that is meant that the information should be readable at a glance. The insights in a dashboard should be clear right away. Follow the six principles of Tufte, with one additional principle, for a guidance during the design process of data visualization. o The first is the documentation of the sources and characteristics of the data. o The second is the focus on an adequate comparison. o This is followed by the demonstration of the cause and effect with comparable mechanisms. o And back in return those mechanisms should be expressed quantitatively. o The fifth attention point is the acknowledgement of the multivariate nature of analytic problems. o And the last sixth principle is the inspection and evaluation of the alternative explanations. o Don’t forget to focus on the seventh principle that is added, that focuses on the ‘making sense’ of the visualization, by using table 20. Taking one step further into the direction of information visualization, different elements are important in the building process. Listed below are the five basic principles by Mazza that need to be considered during the information visualization process, concluded with one additional step: o The problem. This relates to what has to be presented, demonstrated, or found.
79
o The nature of the data. There are different data types, data could be numerical (a top-5 list), ordinal (non-numerical data having a conventional ordering, such as days of the week), and categorical (data with no specific order, like cities). o Number of data dimensions. Depending on the number of dimensions, representations can be univariate- (one dimension), bivariate- (two dimensions), trivariate- (three dimensions), and multivariate data (four or more dimensions). We perceive our world in three spatial dimensions, therefore interpreting up to three dimensions is rather easy. But things with more than three dimensions however, are very frequent in real world situations and represent one of the most challenging tasks in Information Visualization. o Structure of the data. This could be linear (data coded in plain data structures, like arrays or tables), temporal (data which changes during the time), spatial or geographic structure (like maps or something physical), hierarchical (like structures in organizations), network structure (representing relationships between two nodes). o Type of interaction. Whether the resulting graphical representation is static (like a print or a static image on a display screen), transformable (features like zooming or filtering), or manipulable (users may control parameters during the process of image generation). o These five basic principles are focused on the goals of the visualization and the type of data. The last principle focuses on the representation, but missing is the step after that, which is the interpretation, sense making, and understanding step. In the field of online dashboards this step is essential. Amplify cognition with the two principles of Card, Mackinlay, and Shneiderman, which are supporting the last principle Mazza. With these tricks visualization could amplify the cognition and support the perception and leads one from abstract data to more specific and concrete insights and knowledge. o Make it easy to find information by grouping it together. During the visualization process this would be helpful to amplify the cognition. o Make use of the detection of patterns; recognition instead of recall is an example to amplifying cognition Taking into account that recognition works in a more beneficial way than recall for amplifying the cognition, should definitely be considered during the design process. 80
There should be a possibility to navigate through the dashboard, by using the Information Seeking Mantra of Shneiderman. The best way is to use the mantra is to begin with an overview, like a general dashboard, and continue by zooming, for a deeper analysis on the information, for example with tabs. After this analysis details on demand could be found in the analytical database. To respond on the resulted desire, this powerful information seeking mantra should be included in the design process of online dashboards. Then I would like to continue with some key desires, obtained from of the qualitative and quantitative research results: o An online dashboard should reflect the activities of an organization o An online dashboard should provide insights into the performance on only a few focused goals. Fewer graphs are considered as positive, because this focus makes sure that the insights become immediately clear. o There should be an opportunity to compare the different insights with each other, keep that in mind while designing the different elements of a dashboard. o An explanation or legend should be included Usage of color: Only use meaningful colors for numbers within graphs: the colors green and red are preferred. Other or more colors (such as orange) are experienced as chaotic. Usage of numbers: Numbers are considered as extra information. They should only be used within a graph, if they are clear immediately, with the support of an explanation or a legend
Below you find the table for the design of graphs, focusing on five different information types and their most suitable visualization form:
81
Information
Recommended
type
visualization
Information type 1
Comparing magnitudes of
Bars
independent values
Lines Information type 2
Particular
Reason
Example
Quickest interpretation and preferred
Quickest interpretation and preferred
comparison with different entities Combination Preferred, but possible bars and line incorrect interpretation Quickest interpretation
Stacked Information type 3
Particular
and high percentage correct answers, but not preferred
comparison with Second quickest
equal entities Lines
interpretation, third preferred visualization
Quickest interpretation, Stacked
Information type 4
high percentage correct answers, but less preferred than bars
Change over time
Second quickest interpretation, Bars
preferred, but rather low percentage correct answers Quickest interpretation,
Information type 5
The overall trend
Bars
highest percentage of correct answers and preferred
Table 20: The most optimal combination of visualization forms with a specific information type
82
6 Conclusion I closed the gap between practical experience, theory and scientific testing, with a new method to design an optimal online dashboard and by giving an answer to my main question: ‘what is the optimal visualization of data in online dashboard design?’ By the literature review and the convenient methodology on the research approach, the outcome is a combination of elements which results in the new theory. Combining the theories of Tufte, Mazza, Card, Mackinlay, Shneiderman and Few, added by the perceived results of this thesis. Together they form the basis for the design of an optimal online dashboard.
In addressing this, I based my findings on qualitative and quantitative empirical data which I developed through the my research. Below I elaborate on each part of the key questions is answered, and continue with a summary on how my work relates to the key debates. I conclude with presenting the limitations of this research and possibilities for further research. The main question is divided into three additional questions, which I identify as ‘subquestions’ to obtain a focused answer. In this context, the first question is: ‘What is the context and what are the requirements of an online dashboard?’ The results of the qualitative research part provides answers to this first sub-question. The online dashboards are used as report with a summarized view on the performance of a website. They provide insights and information that support decisions that need to be made in the organization. Many different people in organizations work with online dashboards and use it primarily by the upper of higher organizational levels to control the lower levels in the organization and for the upwards flow from the lower levels to the higher levels in the organization for the accountability. The second sub-question is: ‘How do you design the layout of a general online dashboard?’ By splitting up this question in two detailed question, a more focused answer could be provided. The detailed questions focused on the amount of graphs in a general dashboard, the amount of numbers in a graph, and other remaining elements. Both results of the qualitative and quantitative research provided results for this sub-question. The expressed desire for a general dashboard and the objectives of online dashboards provide an answer on the first sub-question. Focus, clarity and immediacy are the key objectives in a general dashboard that stands out. An explanation or legend should be
83
provided to support these objectives. Keeping these key objectives in mind, an online dashboard should present a focused amount of graphs which clearly relate to the expressed goals and targets of an organization. Numbers can be included to support and clarify the graphs but only with a clear explanation or legend. The remaining key desires are the opportunity to compare different results and the possibility to navigate through the dashboard. The third question is ‘what is the best visualization per graph?’. To provide a clear focused answer, this question could also be divided into five more detailed questions, that focus on a different information type. The results of the quantitative research provide clear answers on each of these questions. The first information type is comparing magnitudes of independent values and a visualization in which bars can be used to enhance the design and utility. Using bars provides the quickest and most correct interpretation. For the second information type with a particular comparison with different entities, the visualization with lines provides the best combination. Combining the interpretation and the opinion concludes in the best combination for this information type if you use the line visualization. It could also be considered to use a combination of bars and a line in second best place. The third sub-question focuses on a particular comparison with equal entities. This information type provides the best results in combination with a stacked visualization. Together they enable the user to come to the quickest interpretation with a very high percentage of correct answers. However, in this case, the opinion of the respondent is more negative about the stacked visualization and the respondents considers the use of lines as a better visualization. And because lines have the second quickest interpretation, this could be a fallback option to choose as visualization for this information type. Continuing with the fourth question that focuses on the most optimal visualization in a graph if you visualize the change over time, the quantitative results show that this type of information provides the best interpretation with a stacked or bar visualization. Although I observed a contradiction concerning the stacked visualization; the percentage of correct answers is the highest, but the opinion of the respondents shows that they not consider a stacked visualization as one of the clearest visualizations. The visualization bars is considered the clearest, but has the least amount of correct answers. This makes the bar visualization not convincing and I recommend to use the stacked visualization to visualize a change over time in online dashboards. The last information type contains the overall trend, which is a very important information type according to the results of the qualitative research. The optimal combination is formed with visualization with bars because 84
this combination had the quickest interpretation and highest percentage correct answers. The second preferable combination is formed with bars and a line, where the bars visualize the target and the line the actual numbers.
Summarizing all the new insights that are obtained out of this thesis, I started with the literature review, and found gaps within different theories on visualization. With the combination of lacks within certain literature principles, such as Tufte and Mazza, and the additional literature, such as Card, Mackinlay, Shneiderman, and Few, a convenient research approach was designed. With the division of qualitative research and a quantitative research, the methodology was most optimal to provide answers on the stated research questions. The findings of both the qualitative research as well as the quantitative research provided strong argumentations and answers for the gaps that were found between the literature, the practical experience, and the scientific testing. With the obtained findings, the new method for the design of an optimal online dashboard can be developed, which is a great addition to the field of online dashboard design and could be even seen broader in the field of information visualization design. The new method provides a comprehensive list of principles that could be applied on the information visualization process, with additional principles that are specifically focused on online dashboards design.
There were also some limitations to my research where a different approach or further research could provide more insights and answers beginning with the element of numbers. While numbers could be included in the graphs, there are no significant results on how they should be presented. Further research provides more insights in this element and could be a contribution to the new method on designing the most optimal online dashboard. Secondly the focus on the variable color is left out in the quantitative research on purpose although the use of color is mentioned in the qualitative research by all the respondents. But with only minor results, the only significant outcome is that the colors green and red are preferable in combination with numbers. Other colors, such as orange for a ‘danger-zone’ are only perceived as clear if they are presented in combination with explanation. In order to make claims or provide a judgment on these opinions about color, further research is required, where different colors versus each other or color versus no color could be tested. A research focused on the use of color in online dashboards could be a great additional part for the optimal design in online dashboards.
85
Notable at the third information type which is a particular comparison with equal entities, is the choice of the respondents for the clearest visualization. The visualization stacked resulted quickest and has the highest percentage correct answers, these quantitative results contradict the opinions of the respondents. The same contradiction is found in the information type that focuses on change over time. The fastest interpretation and the highest percentage correct answers are resulted in combination with a stacked visualization. But the results on the opinions of the respondents do not match. The respondents prefer the bar visualization over stacked, but bars do not show the same high results. There are no explicable reasons found for these discrepancies. Could these results be caused by the recognizability and familiarity of visualization with bars? There is a possibility to find this reason in a more extended research. Looking at information type five, the overall trend, I am not convinced of the results that show the most optimal visualization. Taking into account the high percentages of incorrect answers makes it plausible that there could be a mistake in the approach of the questions, the graphs, or something else concerning this information type. Therefore I could recommend a larger or different research approach to find out what the most optimal visualization is in combination with this type of information that shows the overall trend.
But moreover with three complete answers, I now know what the context and requirements in online dashboards are, how the layout of a general dashboard should look like and what the best visualization is within the use of graphs with focus on five different information types. When I merge these answers with insights and theory, I have my answer on the main question ‘What is the optimal visualization of data in online dashboard design’: With a combination of my perceived results based on my observations and research, the useful elements of Shneiderman’s mantra, two applicable principles of Card, Mackinlay, and Shneiderman and the modified theories of Tufte and Mazza, a new method with basic principles is created which can be used to design the most optimal online dashboard. With this new method I seal the gap and form a bridge between practical experience, theory, and scientific testing. The contribution of this thesis to the wider debate is important, because it shows important elements that could be applicable on the larger field of the information visualization design process. The results of this research, the mantra of Shneiderman, the principles of Card, Mackinlay, and Shneiderman and the modified theories of Tufte and Mazza are useful in the design of every information visualization. Though this research is focused on online dashboards, looking at the bigger picture, this combination of principles provides a new 86
method that also could be applied on information visualization. With further, enhanced research, this could be the new method that is used in the future to design the most optimal information visualization.
87
88
7 Bibliography Bollier, David. The Promise and Peril of Big Data. Washington, DC: Aspen Institute, Communications and Society Program, 2010. Print.
Boyd, Danah, and Kate Crawford. "Critical Questions for Big Data: Provocations for a Cultural, Technological, and Scholarly Phenomenon." IEEE Communications Society Information 15.5 (2012): 662-679. Print.
Card, Stuart K., Jock D. Mackinlay, and Ben Shneiderman. Readings in Information Visualization: Using Vision to Think. San Francisco, CA: Morgan Kaufmann, 1999. Print.
Cleveland, William S. Visualizing Data. Murray Hill, NJ: At & T Bell Laboratories, 1993. Print.
"Data, Data Everywhere." The Economist (US) 27 Feb. 2010: n. pag. Print.
Feteke, Jean-Daniel, Jarke J. Wijk, Van, John T. Stasko, and Chris North. "The Value of Information Visualization." Information Visualization: Human-Centered Issues and Perspectives. Vol. 4950. Berlin Heidelberg: Springer, 2008. 1-18. Print.
Few, Stephen. Information Dashboard Design: The Effective Visual Communication of Data. Beijing: O'Reilly, 2006. Print.
Few, Stephen. Now You See It: Simple Visualization Techniques for Quantitative Analysis. Oakland, CA: Analytics, 2009. Print.
Manovich, Lev. "Trending: The Promises and the Challenges of Big Social Data." Debates in the Digital Humanities. Ed. Matthew K. Gold. Minneapolis: University of Minnesota, 2011. 460-675. Web.
Mazza, Riccardo. Introduction to Information Visualization. London: Springer, 2009. Print.
89
Tufte, Edward R. Visual Explanations: Images and Quantities, Evidence and Narrative. Cheshire, CT: Graphics, 1997. Print.
Shneiderman, Ben. "The Eyes Have It: A Task by Data Type Taxonomy for Information Visualizations." IEEE Xplore. IEEE Computer Society Washington, 1996. Web. 10 Mar. 2013.
Ware, Colin. Information Visualization: Perception for Design. San Francisco: Morgan Kaufman, 2000. Print.
"What Is Big Data? - Bringing Big Data to the Enterprise." IBM Big Data Platform for Enterprise - Bringing Big Data to the Enterprise. N.p., n.d. Web. 20 Jan. 2013.
90
8 Appendix In this section the total interview of the qualitative research and the online survey of the quantitative research can be found in section 8.1 and 8.2. In section 8.3 the results of the qualitative research can be found and in 8.4 the results of the quantitative research. The recommendation report for Conversion Company can be found in section 8.5.
91
8.1 Interview qualitative research Below the interview of the qualitative research, subdivided into the interview questions and the dummy dashboards one to four. 8.1.1
Interview
Preparation: What organization are you working for? What is your role in the organization? What kind of dashboards are they using? For which kind of level in the organization are these dashboards? (Working floor) What is your responsibility in regards to the dashboard? (distributing, interpreting, analyzing, decision making) Questions in regards to dashboard in general (part 1): -What are, in your opinion, the objectives of a dashboard in general? what is/should be the purpose? -How are the dashboards you received in your organization used? (decision making, or persuade others, higher level, daily management)? -Does this meet the objectives? If not why not? -What are the requirements for a successful dashboard? -Which elements should a dashboard include? -What should it look like? -What kinds of variables are important for you in dashboards? Looking at the dummy dashboards (part 2): -What kind of grade would you give this type of dashboard? -Did you experience difficulties or problems with previous dashboards? -Do you think that some elements could be shown differently or more clearly? -Are there any extra elements that you would find useful in a dashboard? -Which metrics do you consider as organized and clear relating to the information on dashboards? -What kinds of variables are important for you in dashboards? -Do you expect a clear dashboard? -What do you expect to be clear? -What is clear in your opinion? -Do you expect that you quickly see and interpret the information? -What do you quickly want to see? -What is quick in your opinion? (by yourself, or some explanation) General questions (part 3): Name: 92
Age: Male/Female Company: Function: Education/Field of study Level of experience with analyzing data: 8.1.2
Dummy dashboards
Below you find the first and third dummy dashboard that are (together with the dummy dashboards in figure 5 and 6) used in the qualitative research. The second dummy dashboard you see is the complete version of figure 18 and the complete version of the fourth dummy dashboard in figure 21.
Figure 31: Dummy dashboard 1, used in the interview of the qualitatitve research
93
Figure 32: Dummy dashboard 3, used in the interview of the qualitatitve research
8.2 Online survey quantitative research Below all the selected visualizations per information type. The full online survey can be found on the CD-ROM under ‘Qualtrics online Survey.docx’
Figure 33: Visualization forms of information type 1
94
Figure 34: Visualization forms of information type 2
Figure 35: Visualization forms of information type 3
Figure 36: Visualization forms of information type 4
95
Figure 37: Visualization forms of information type 5.
96
8.3 Results qualitative research Below are the results of the qualitative research presented. The outcomes of all the interviews are presented in the native language of the interviewees, which is Dutch. The results used in the chapters results and interpretation are literal translations from Dutch to English. Interview 1 Voor welke organisatie werk je en wat is jouw rol in de organisatie? Conversion Company, ik bouw gewoon dashboards, de wekelijkse dashboards, rapportages, ja ‘you name it’, dus ja, voornamelijk dashboards. Ik moet ervoor zorgen dat de data klopt. En het opsturen als dat nodig is, checken op een reguliere basis om te kijken of het allemaal nog goed werkt. En als er aanpassingen erbij moeten, dan pak ik dat op. Wat voor soort dashboard gebruik je? Op dit moment heb ik een hele boel type dashboards. En het idee is eigenlijk wat we nu aan het doen zijn is om al die dashboards dan in één soort dashboard te kunnen krijgen. Dus een standaard type dashboard. Dus als je dan echt wil weten wat voor type dashboard we gebruiken, dat hangt van de klant af. Wat we kunnen of wat voor info we gaan opnemen. En voor welk level van de organisaties worden deze dashboards dan gebruikt? Dat verschilt ook per klant. Dus het kan zijn dat het gewoon voor binnen het bedrijf is, dus gewoon intern, maar het kan ook voor extern zijn, voor de klant of dus gewoon management bij de klant, of dashboard voor de klant of alleen voor de channel-managers. Dus dat verschilt echt per klant. Wat zijn voor jou de doelstellingen van een dashboard? Gewoon overzichtelijk, in één keer moet iemand ernaar kunnen kijken en zijn of haar conclusies eruit trekken. En wat ik daarmee bedoel is dat het niet echt groot hoort, dus dat je dan vier of vijf of tien of twintig pagina’s hebt, maar wat ik eigenlijk het prettigst vindt is eigenlijk gewoon één pagina. Één A4tje waar je alles erop hebt staan en ook de informatie die je nodig hebt, dus niet data diagrammen waar je geen behoefte aan hebt, maar echt dingen waarop je echt eigenlijk aan het sturen bent, zoals tabellen. En dan zit duidelijk er ook gelijk bij, het moet goed duidelijk zijn wat je eigenlijk aan het tonen bent, dat iemand naar bepaalde elementen van het dashboard aan het kijken is, wat gebeurt er eigenlijk hier, maar dat je dan gewoon in één keer kan zeggen ‘Aaah, ok ja ik weet precies wat er allemaal gebeurt.’ Ja in een overzicht, ‘that’s it’. En waarvoor worden de meeste dashboards gebruikt die jij dan opstelt? De meeste dashboards zijn om een samenvatting, een rapportage te geven van een afgelopen periode en die periode verschilt ook gewoon tussen weken en maanden, hangt er vanaf welke periode dat is. Maar de meeste zijn de weken waarop we rapporteren. En hoe worden de dashboards dan meestal gebruikt? Ik gebruik het niet voor decision making, of om mensen over te halen. Nee helemaal niet, want wat ik eigenlijk doe is het zorgen voor dat de dashboard gewoon klopt. Dat je data erin klopt. En die gaat dan verder naar iemand die dan contact heeft met de klant. En die kan er met de klant eraan zitten en kijken wat er allemaal gedaan moet worden. Dus het wordt meer gebruikt om de trend te laten zien, wat is er eigenlijk gebeurd, zodat ze daarop kunnen werken, van okee wat gaan we nu doen. En als je kijkt naar de variabelen in een dashboard? Kan je er dan een paar noemen wat er voor jou uitspringt wat het zeker moet hebben?
97
In de meeste dashboards in een bepaalde periode rapporteren, bijvoorbeeld tien weken in een periode, dan zou je wel graag een trendlijn willen zijn. Waarom trendlijnen, want dan zie je gewoon echt wat er allemaal is gebeurd in de afgelopen periode en dan kan je bij sommige ook een voorkeursstijl opzetten waar je dan kan bekijken wat er gebeurt of over een bepaalde periode, waar zit je eigenlijk. Dashboard 1 Onderweg naar een 8, nog net niet een 8. Een 8-. Ze zien er allemaal hetzelfde uit, al die blokken zien er hetzelfde uit. In dit geval niet goed omdat het er echt een heleboel zijn. Ik vind het eigenlijk te veel, dat erop te veel gerapporteerd wordt, in deze. Voordelen bij deze is sowieso dat elke kolom heeft zijn eigen onderdelen eigenlijk. Bij de eerst kolom bijvoorbeeld traffic, dan gaat het alleen om traffic. Dan weet je meteen dat je dat de rest niet in de andere kolommen kan vinden. Dus als iemand meteen iets van content wilt weten, dat weet hij precies dat hij dat nooit in de eerste kolom gaat vinden. Dat is sowieso heel goed. En kleuren enzo, die corresponderen met elkaar. En even denken, je kan heel goed zien met welke periode je bezig mee bent. Wat ik wel mis met periode is wat de begin periode is, je hebt wel nu tot welke ik periode ik aan het rapporteren bent, maar je weet niet precies wat de eerste week is in de grafiek. Want je hebt rechtsboven de data, de week. Als je kijkt naar de visualisaties, dan klopt het wel, want het gaat bijna overal om verschillen type matrix, want je hebt aantallen en percentage en als je die dan met elkaar wilt combineren dan zit je beter in twee verschillende grafieken, dus een lijn of een staafdiagram. Dus dat ziet er sowieso heel goed uit hier. En het is gewoon duidelijk. En als ik nog iets zou toevoegen. Dan het aantal grafieken op het dashboard zelf eigenlijk, dat vind ik wel een beetje veel. De knelpunten zijn niet gelijk duidelijk, en het wordt niet gelijk aangegeven wat die cijfers betekenen. Als het dan vorige week is of van de hele periode in de dashboard of van alle data die ze beschikbaar hebben. En het verschil geeft ook niet aan wat het verschil is ten opzichte van de periode ervoor, of verschil ten opzichte van het gemiddelde, dat wordt allemaal niet aangegeven. Dus misschien duidelijkere uitleg misschien wel gewoon als je de eerste keer hebt aangegeven, dit is het dan weet men dat. Maar als iemand dit nog nooit heeft gezien dan zit hij wel te denken wat is die verschil en wat zijn de cijfers en dat soort dingen. En qua toelichting. Met of zonder? Kan beide, maar omdat deze dan zoveel van die grafieken heeft, van die blokken, dan maakt het ook lastig om meer info erin te zetten want je hebt gewoon geen ruimte meer. Anders worden die grafieken gewoon klein en dan daar heb je niet zoveel aan. Maar het is altijd gewoon goed om een legende of uitleg erbij te hebben, van wat er eigenlijk gebeurd. Wat je eigenlijk ziet. Diegene met meerdere staven daar wordt het gewoon een beetje lastig, want je moet ten eerste goed gaan kijken naar de kleuren en omdat je dan verschillende schalen en bepaald kleur gebruik, dus drie verschillende kleuren blauw, is ook moeilijk om dan naar die cijfers te gaan kijken. Want die cijfers die hebben niet de kleur van de staaf. Dus dan moet je gewoon goed weten waar je nou moet kijken. Dus de cijfers hebben niet dezelfde kleuren, de titels erboven wel.En als ik kijk naar de visualisaties. Ja dat is wel overzichtelijk, kijk hier is het geen probleem want het is allemaal gewoon trend, dus de assen hoeven er niet bij te zijn om echt te zien waar ben ik nou eigenlijk naar aan het kijken. Welke hoort bij welke, maar als het meer dan een wordt, dan twee matrix, dan wordt het lastig om die dingen te vinden en dat is precies bij die derde. Als je die drie staven bij elkaar hebt, dan wordt het een beetje lastig. Dashboard 2 Hij is kleiner dan de vorige, dat is mooi. En ik snap wat er allemaal onderdelen, gedeelte naast elkaar zetten. Verschillende tabjes links gezet, waardoor je door de dashboard kan schakelen. En dat is beter, dan heb je meer de ruimte nodig. En je hebt ook dat je zelf kan kiezen welke weken je met elkaar wilt vergelijken. Dus goed dat je zelf kan aangeven wat je wilt zien. Dat weet ik, dus dan wil ik niet met die week vergelijken, maar met twee andere weken vergelijken, die eigenlijk bijvoorbeeld zonder campagnes zijn geweest. Dan weet ik zeker dat ik hetzelfde aantal bezoekers naar mijn site
98
heb. Ik zou hem een 8+ geven. De goeie is van die weken wat je zelf kan kiezen en ik kan mijn data meteen exporteren als ik die nodig heb. Wat er niet goed is, zijn die wijzers, want ik heb geen idee waarom die daar eigenlijk staan. Er staan gewoon cijfers daar maar waarom eigenlijk met die schaal? Wanneer is het eigenlijk goed en wanneer is het eigenlijk fout? Dat zie ik niet in een opslag. Dashboard 3 Deze is ook bijna hetzelfde als die tweede, want hier kan je ook zelf kiezen wat je eigenlijk allemaal wilt bekijken. Dus het aantal elementen binnen het dashboard wordt gepresenteerd, wat je wilt zien. De tapjes in het blauw bovenaan. De elementen zelf zijn gewoon lastig, want ik heb geen idee welke weken het zijn. Wel boven aan, maar niet daar welke weken het zijn, of het weken zijn of niet. Dus de schaal zou ik graag zien. De cijfers bovenaan, wat zijn die? Is het totaal is het average, is het van vorige week, dat weet ik niet. Ik neem aan dat de groene en rode percentages een verschil is, maar met wat eigenlijk? Net zoals bij die eerste dashboard eigenlijk. En als ik er uitleg bij zou krijgen? Ja dan wel, want je ziet gewoon het is een trend, en als ik er uitleg bij krijg wat eigenlijk alles nou betekend dan ja. Dashboard 4 O nu zie ik het, je hebt de trend, dat is wel goed om de trend te zien van alle kanalen en daarnaast is dit november. En dan heb je dus dit voor een bepaalde periode en dan voor een andere periode dan. Maar dan weer hier kan ik niet selecteren. Zou fijn zijn als ik mijn eigen periode kan selecteren net zoals bij dat andere dashboard 2. En de data die ernaast ligt is wel echt handig, want soms heb ik niet zoveel aan die chart maar wel aan de data. Maar als die data in de pie-chart zou zitten, misschien zou het wel handig zijn. Misschien zou het wel wat meer overzicht geven. Percentages of getallen erbij. Wanneer vind je iets duidelijk? Als iets duidelijk is, dan heb ik gewoon geen vragen. Ik moet dat dashboard gewoon in een keer kunnen lezen. Dus wat ik daarmee bedoel, kijk hier hebben we geen assen ofzo, en als ik naar deze zou kijken dan weet ik niet, zoals ik net zei, of er het nou de week ervoor was of een periode of een gemiddelde of totaal. En als dat allemaal nou in een keer duidelijk was, zou ik dat wel fijn vinden. Moet het dan ook snel geïnterpreteerd worden? Kan, maar hoeft niet. Ja snel sowieso, nou niet snel maar als het snel is dan weet je gewoon wat er eigenlijk allemaal gebeurd is. Maar als je iemand bent die meer wilt weten en echt de diepte in wilt, dan wil je wel gewoon meer informatie in een elementen. Dus het hoeft niet, maar het hangt weer af van de behoefte van de klant. Als ze meer willen weten in een keer. Naam: Kinsley Roosburg Leeftijd: 25 Bedrijf: Conversion Company Functie: Information Designer Field of study: Informatica (globaal) Niveau met analyseren: weinig ervaring, ik maak ze maar analyseer ze niet.
99
Interview 2 Wat is jouw rol in de organisatie? Ik ben channel manager, en dat betekent, wat ik nu doen ik ben e commerce manager bij Telfort. Daar ben ik verantwoordelijk voor de sales en de online service verlenging en online branding voor residential, dus voor de verkoop van alle vaste producten van Telfort. zoals internet tv en bellen. dat doe ik voor interim basis en daarom heb ik ook data nodig om goed te kunnen sturen op het een of ander. maar als je even terug gaat naar channel manager, dat kan betekenen e-bussiness strategien schrijf of een website bouwen voor de klant. Maar bij de functie van channel manager kan je functie elke keer een andere invulling krijgen. Maar ik zit dus voornamelijk tussen klant en ons bedrijf Conversion Company. Met wat voor type dashboards werk je voornamelijk? Ik werk met wekelijkse dashboards en ze worden gebruikt om te sturen, om beslissingen mee te maken. meeste op hoofdlijn, op de belangrijkste KPI’s, niet op detail, daar gebruik je geen dashboards voor. Maar voornamelijk om de belangrijkste KPI’s in de gaten te houden, om dus controle over het kanaal te houden en te kunnen sturen. Bespreek je dat dan met de klant? Ik ben eigenlijk de klant, ik stuur het kanaal aan en daar heb ik die data voor nodig. En ik word daarmee geholpen door de webanalyse afdeling hier bij Conversion Company. Ik ben een van de steakholders die het dashboard daadwerkelijk ook gebruikt. Dus ik lever het niet af aan de klant maar ik gebruik het gewoon echt om mn werk voor hun te kunnen doen. Maar aan de andere kant zijn er ook mensen bij Telfort die daar werken, zoals de marketing directeur of de marketeers bijvoorbeeld de achterliggende keten, de leverinsketen, bestuurd. Die wilt ook graag weten wat er aan de voorkant allemaal binnenkomt elke week. Die maken ook gebruik van hetzelfde dashboard. Die sturen wij dat ook. Over welk niveau in de organisatie praat je dan voor het gebruik? Marketing, IT, eigenlijk iedereen die een direct belang hebben bij het online kanaal. Dat zijn uitvoerende mensen zoals marketeers, of mensen uit mijn team, maar ook mensen die beslissingen nemen zoals de marketing manager. De klant-process manager, dus die de leverings keten doet. Dus zowel operationele uitvoerende mensen gebruiken het, als mensen die beslissingen nemen de boel aansturen. Dus echt werkvloer en management niveau. Wat zijn volgens jou de doelstellingen van een dashboards? Informeren en inzicht geven in performance op beperkt aantal KPI’s. Je merkt nogal eens at dashboards de neiging hebben om op het niveau van metrics dus alle dingen die interessant zijn iets te zeggen richting je KPI’s. Maar het heeft nogal eens de neiging om onwijs uit te breiden. Voor je het weet heb je een dashboard met achtenveertig tabbladen, waar ontzettend veel informatie in staat maar wat niet heel relevant meer is voor heel veel mensen. Dat merk je soms wel, dat als een dashboard soms een week niet verstuurd wordt, dat van de veertig ontvangers er slechts twee roepen ‘he waar is het dashboard?’ En dat is dus een beetje het risico. Voor mij moet het compact zijn en op een beperkt aantal KPI’s inzicht geeft in de performence. Hoe worden dashboards in de organisatie gebruikt? Het is om beslissingen te kunnen nemen. En wat je vaak merkt als je het dashboard ziet, is dat vaak antwoord geeft met paar belangrijkste getallen op een de belangrijkste vragen, namelijk hoe doen we het? En vervolgens roept het weer een heleboel nieuwe vragen op. Stel we doen 20% te weinig, waarom is dat? En dan is het vaak weer aanleiding tot nieuwe analyse met als doel het inzicht boven water te krijgen, waardoor we de juiste beslissingen kunnen nemen. Dus voor mij is het echt een belangrijke beslis-tool. Ik hoef niet per se andere te overtuigen om bepaalde beslissingen te nemen. Ik zie wel een tweedeling tussen enerzijds dashboards en wekelijkse dingen zoals de belangrijke cijfers en anderzijds heb je enorme bakken met data en we hebben enorm veel analysten zitten dus je kan allerlei verschillende verdiepingsvragen stellen om iets in gang te zetten of om iemand mee te overtuigen of wat dan ook. Maar een dashboard an sich gebruik ik daar niet echt voor, het zijn meer
100
verdiepingsvragen, die ik dan aan een van de analysten stel om dan iemand in de organisatie ergens van te overtuigen. Wat zijn de elementen van een succesvol dashboard? niet groter dan 1 a4tje, want het wordt vaak te groot. Dan is het gewoon niet meer relevant als het voor iedereen relevant probeert te zijn. Maar wat het belangrijkste is dat het echt zuiver op je performance indicatoren zit. Dat dat ook getallen zijn waar iedereen het over eens is en dat we daar dan met z’n allen aan werken. Want die getallen moeten ons vertellen hoe we op weg zijn naar onze doelstellingen. Het moet breed gedragen zijn aan relevantie en compact en klein zijn. En de opmaak, welke variabelen vind je belangrijk? Je wilt weten hoe je het deze week doet en daar wil je context bij. Hoe we het doen versus plan of versus vorige weken. Week op week bijvoorbeeld. Dus je wilt de moment opname van hoe we het doen maar je wilt dit moment ook in context hebben van wat je probeert te bereiken en hoe je het in het verleden hebt gedaan. Dashboard 1 Ik geef het een 6. Hier probeert het heel veel te zijn. Het is 1 a4tje maar als ik hier kritisch naar kijk kan er heel veel weggelaten worden. Het punt is dat het overzichtelijk moet blijven. Ik kan het bijna niet lezen. Daarnaast dwingt dat je ook om heel kritisch te zijn. Ik kan zo nog vier interessante grafiekjes verzinnen maar dan kan je eindeloos doorgaan. Het voelt als of iemand nog wat kritischer had kunnen zijn. En volgens mij is dit meer dan alleen de kritieke cijfers die we willen hebben die hier staan. Of zitten hier dingen bij die interessant zijn om te weten maar niet kritiek. Maar misschien moet je er gewoon een ander dashboard voor bouwen. Voor iemand die alleen maar met traffic bezig is bijvoorbeeld, dat je er nog een extra laag onder hebt zeg maar. Maar ik zou eerder denken okee, ik kan hier twee grafieken heel interessant vinden en echt gebruiken. En van de grafieken, ja dit is heel standaard conversion company. Zo’n tien weeks is vrij arbitrair, waarom tien weken en waarom geen jaar? Voor sommige wil je gewoon en jaar beeld hebben namelijk, zodat je de Year to Date duidelijk op het plaatje hebt, dat lukt hier natuurlijk niet. Voor sommige metrics werkt het wel zo’n tien weekse met een target line. Maar ik zie ook nog twee verschillende assen. Zo’n grafiekje zou goed zijn, maar zoals we het bij alles op de zelfde manier probeert toe te passen schiet het z’n doen denk ik een beetje voorbij. Ik denk dat je het op een andere manier veel sprekender kan maken. Maar ik zou ook niet precies weten hoe dat er dan uit komt te zien hoor, daar heb ik kennelijk niet voldoende creativiteit voor. Maar het ligt ook een beetje aan de klant. Want sommige klanten zeggen dat ze ontzettend veel willen weten en dat zijn typische klanten die roepen dat ze duizend dingen willen weten. Dat is een veel voorkomend process, als je met die klant om de tafel gaat zitten om vast te stellen wat belangrijk voor hun is. Wat de doelstellingen zijn of wat de belangrijkste KPI’s zijn, dan roepen sommige klanten zo ontzettend veel. En ik vind echt dat wij dan kritischer moeten zijn, als organisatie, want ik vind dat we daarin te veel meegaan. Wij zeggen te snel okee als je dat ook wilt weten dan doen we er nog een grafiekje bij. Zonder te vragen maar wat doe je daar dan eigenlijk mee, met dat extra grafiekje? Ik krijg het idee dat ze wel van alles willen maar dat ze er uiteindelijk niet per se iets mee doen. Van tevoren willen ze van alles, lekker makkelijk geef me het maar. Maar op het moment dat ze het krijgen, dat ze merken maar ja wat moet ik er eigenlijk mee. En nog even die weken, waarom tien waarom niet acht of twaalf. Welke variabelen zijn belangrijk voor jou? Beperkt aantal, dus focus op de belangrijkste KPI’s. Als ik denk aan wat vind ik belangrijk elke week, dan is een handvol variabelen. Als hij er op staan is het voor mij heel nuttig. En qua snelheid. En met kleurtjes werken vind ik ook fijn. Als we het groen is dan zijn we op target en rood betekent niet op target. Het dashboard waar ik mee werk is verdeeld in drie kolommen. En dat maakt het overzichtelijk. Met alles wat je visueel maakt, dat is een process. Je denkt als ik het zo neerzet dan is het duidelijk, maar dan kan het zijn dat je er na een paar weken achter komt van onee het kan toch ook anders. Bijvoorbeeld hier heb ik belangrijke getallen, maar eigenlijk zit ik hier nu al dat ik het anders misschien ook fijn zou vinden. Elke keer bijsturen. [feedback]. Volgens mij wilt de afdeling webanalytics boven, die hebben natuurlijk veel handmatig werk en willen zo veel mogelijk automatiseren en vastzetten voor je. Maar ik kan elke week wel weer iets anders willen, als iets me
101
niet bevalt of als ik anders gevisualiseerd wil. Dus eigenlijk denk ik niet dat er een gestandaardiseerd dashboard kan komen. Maar het moet wel tot op een bepaalde hoogte gepersonaliseerd blijven. Het ligt gewoon heel erg aan de klant waarmee je werkt en het bedrijf. Dashboard 2 Deze vind ik beter. Dit geeft per site, dus die focus per site zit hier echt in, meer op de KPI’s. Waar men targets op heeft en waar men op afgerekend wordt. En de tabbladen, ja als er nog meer informatie achter zit, ja dat lijkt me beter. Dit geeft me meer rust. Als dit mijn baan zou zijn, meer richting social, dan lijkt hij me wel erg goed ja. Dashboard 3 Totaal overzichtelijk en zegt mij helemaal niets. Omdat ik geen tijden zie, de trend waar is die? Waar is de legende? Het feit dat je hier niet duidelijk ziet wat wat is. Hier zou ik niets mee kunnen. Het punt van deze, is dat als je het aan iemand geven die er weinig verstand van heeft, maar niet perse verstand heeft van het kanaal. Dan snapt hij niet gelijk wat hier staat denk ik. En dat is wel de kracht van een dashboard, dat je gelijk ziet wat er staat, zonder al te veel uitleg. Dat is voor deze echt totaal niet zo. En de as verdeling? Dashboard 4 Die metertjes zijn we denk al sinds 2001 een beetje klaar mee. Ik ken deze en dit is niet mijn smaak. Die zeggen uiteindelijk niet zoveel. Wanneer is iets oranje en wanneer is het rood? Kijk groen en rood begrijp ik wel, maar vanaf wanneer is het dan oranje? En niet goed en niet slecht? Gevarenzone, maar vanaf wanneer. Week op week is wel een goede optie hier. Dat je het zelf kan aangeven, kijken waar uitschieters zitten. Maar geen context verder over meerdere weken enzo. dit soort lijntjes door elkaar heen van verschillende metrics, ze hebben wel wat met elkaar te maken maar eigenlijk ook weer helemaal niet. Dit maakt een grafiek onoverzichtelijk dus uiteindelijk relevant. Ik zou één zo’n metric pakken en dan desnoods met een dropdown om de verschillende te selecteren. Want dit is wel veel met elkaar vergelijken. Maar kies tussen visits of visitors, niet beide, je moet je ergens op focussen en daarvoor kiezen. En die andere zijn ook relevant maar zou ik even wat dieper rapporteren. En ik zie er een paar op de grond liggen. maar als ze nul zijn, waarom rapporteer je ze dan? Dan zou ik het interessanter vinden om andere uitschieters vinden. Ik denk hier dat er gewoon niet goed is afgesproken wat de belangrijkste getallen zijn. Dat stoort me. Maar als ik naar die metrics kijk, vind ik ze niet inzichtelijk omdat er geen focus is. Wel goed dat die schalen duidelijk zijn weer gegeven, dat vind ik belangrijk. Maa niet overzichtelijk. Dus wat vind jij belangrijk? Overzichtelijk vind ik een zeer beperkt aantal getallen. En hoe dat dan in weer gegeven ligt aan dat getal. Dat maakt me niet zoveel uit. Het is een beetje kijken wat werkt per metric, dat weet ik zelf ook niet zo goed. Bij mijn eigen dashboard vind ik dat we dat heel goed doen en bij sommige ook helemaal niet. Maar dus focus is overzicht. Alleen maar belangrijke getallen, dan wilt iedereen het zien. Als er minder interessante getallen bij staan dan vertroebelen de belangrijke getallen. Onbelangrijke halen de kwaliteit van een dashboard naar beneden. En qua snelheid, als ik een dashboard zie en ik begrijp het gelijk. Dan betekent dat dus dat iemand het dus kennelijk goed heeft weergegeven. Dus ik weet niet of het belangrijk is, maar het is wel een indicatie of het een goed dashboard is. Name: Douwe Sterrenburg Age: 32 Organization: Conversion Company Function: Channel Manager Field of Study: Bedrijfskunde Level of analysing data: Ik gebruik het elke dag, erg ervaren.
102
Interview 3 Wat is jouw rol in de organisatie? Ik ben een van de twee eigenaren. Mijn werkzaamheden zijn heel verschillend, ik ben verantwoordelijk voor opdrachten en de opdrachtgevers. Mensen coachen en begeleiden, zorgen dat ze de gewenste kwaliteit leveren. Zorgen dat we opdrachten binnen halen, de juiste mensen aannemen. Het financiële gedeelte in de organisatie, de cijfers, dat we goed kunnen sturen, precies weten hoe het gaat. Dat doe ik vooral. Komt u ook veel in aanmerking met dashboards? Ja, met name de opdrachtgevers, daar leveren we ook vaak dashboards op, zodat ze goed kunnen zien wat de resultaten zijn. En op basis daarvan kunnen ze dan het kanaal aansturen, dus heel veel verschillende. Over welk niveau in de organisatie praat u dan over? Soms is het op het aller hoogste niveau, is heel data gedreven. Of juist op het niveau van mensen die echt het kanaal aan het aansturen zijn. Of een onderdeel van het kanaal, zoals online marketing, dus dat is heel verschillend. En gebruikt u hier dan ook verschillende dashboards voor? Ja, bij de hogere niveaus wil je het vooral koppelen aan een paar dingen waarop een bedrijf vooral aanstuurt, vaak één of twee of drie of vier dingen waarop ze het bedrijf op aansturen en daar moet je je dashboard heel erg aan koppelen. Dat je duidelijk de relaties laat ziet van waar zij de hele dag mee bezig zijn. Die gaan niet snel in op de inhoudelijke op het kanaal aansturen, dat is meer voor de niveaus daaronder. Maar dat is gelijk twee of drie management niveaus lager, maar dat is het niveau waarop mensen het kanaal dagelijks sturen. Dan ben je meer naar funnels aan het kijken en de conversie. En wat is jouw verantwoordelijkheid inzake die dashboards? Als het om dashboards gaat, kan ik de organisatie uitleggen waarom het belangrijk is om ze te hebben. En de uitvoering ligt dan meer bij andere mensen. Wat zijn de doelstellingen van een dashboard? Doelstellingen zijn dat je aan de ene kant weet wat er gebeurt, dat je een analyse kunt uitvoeren waar verbeter mogelijkheden liggen. Terwijl een dashboard is meer een signaal functie over hoe het gaat. En als je echt de diepte analyse in wilt, ga je meer in de analytics systemen en databases kijken waar alle informatie bij elkaar komt om meer diepte analyses te doen. Een dashboard heeft meer een signaal functie, waar je dan uiteindelijk juist analyses wilt uitvoeren. Want het opleveren van een dashboard kost alleen maar geld maar daarna op basis van die inzichten kun je dingen gaan verbeteren en ook realiseren. Dus dat je business impact maakt, want dat is waar het uiteindelijk om gaat. Zodat je kunt laten zien wat het nut is van zo’n dashboard en het meten en dan ga je dingen meer gestroomlijnd krijgen. Wat maakt een dashboard succesvol? Dat is heel erg verschillend per stakeholder, soms is het meer visueel. Als het een dashboard is voor de raad van bestuur, dan moet het er echt gelikt uitzien, dat je het meer visueel maakt. Als je dan ook wat verder van de materie zit, kan je het toch heel goed koppelen aan business doelstellingen. Hoe lager je gaat hoe meer het ook gewoon cijfertjes mogen zijn. Maar bij een dashboard moet je wel in een keer kunnen zien of het wel of niet goed gaat. Dus het moet niet alleen cijfertjes aangeven, maar moet ook aangeven of die cijfertjes goed of niet goed zijn. Als er ergens veertig procent vol blijft houden in een funnel, is dat dan veel of weinig. Je moet het ergens aan kunnen relateren en dat is belangrijk in een goed dashboard. De kunst is vaak als je heel veel data hebt, om dat op de juiste manier te visualiseren, zodat je de juiste inzichten eruit krijgt maar ook dat het er goed uitziet. Wat niet goed is als het dashboard ontworpen is in één keer, dat het dan vervolgens een jaar hetzelfde blijft, de stakeholders die ermee werken die moeten erin getraind worden om de informatie op de juiste manier te verwerken en informatie te genereren. Want de ene informatie roept weer de
103
volgende informatie op, dus daarom moeten dashboards dus ook flexibel zijn, dat ze continue geüpdate worden en steeds meer toe groeien naar waar de organisatie naar op zoek is. Dashboard 1 Dit dashboard heeft heel veel informatie, dus eigenlijk voor iemand die daar waarschijnlijk elke week mee werkt. Ik vind eigenlijk dat er veel te veel informatie op staat. En die procenten, wat wordt daar nou precies mee bedoeld? Dat is onduidelijk. Ik mis de y-as vaak. Als ik dit zie dan is mijn eerste vraag, wie is de doelgroep van dit dashboard. Want dat is belangrijk, wie is de doelgroep en wat is de behoefte van de doelgroep. En waar wordt deze doelgroep door gedreven. En op basis daarvan kan ik beoordelen of dit een goede rapportage is. Maar los daarvan, als ik in het algemeen kijk vind ik dat er veel te veel op staat. Dit is echt veel te veel informatie, dan kan je beter onderverdelen in een aantal blokken over een bepaald onderwerp. En de graphs, die staafjes naast elkaar je kunt nauwelijks zien wat de verschillen zijn. Terwijl soms vijf procent heel knap kan zijn om dat voor elkaar te krijgen, maar met deze staafjes kun je dat onmogelijk zien. Verder heb je overal dezelfde balkjes en trendlijnen, maar ik zie geen relatie met vorig jaar of met targets. Hier heb je een verdeling, maar is dat een bepaalde trend? Dat haal ik er niet uit. Dus te weinig uitleg erbij. En is dit echt informatie die je moet weten, in het dashboard moet zetten, om moet diegene dan gewoon even in het webanalyse systeem kijken? Dashboard 2 Deze is niet zo interessant voor mij, dit zegt me helemaal niet, het gaat er mee om wat het allemaal oplevert uiteindelijk. Reach, maar wat is het doel. Van opzichte van wat, vorige week of maand? -12 procent, is het ten opzichte van wat, is het cumulatief, dat is allemaal volkome onduidelijk. je wilt weten wat levert het op. Nee dit vind ik niks Dashboard 3 Dit is niet te lezen, dat moet je erbij hebben staan. De stacked haal je helemaal niets uit. Dat is volkome waardeloos, als het hier hoger is, dan snap je toch niet meer wat er met paars is gebeurd. En in december en november, maar wat zijn de verschillen. O dat staat hier wel uitgelegd. Maar daar heb je volkomen niets aan. November ten opzichte van december, dat is wel leuk, maar als december een piekmaand is, ja dan is het interessant om te weten. Maar dan is het interessanter om te weten hoe ze het doen ten opzichte van december vorig jaar. Dus dit i gewoon slecht. Over de tabbladen, ja dat is op zich wel goed. Maar die tab indeling is niet waar ik naar op zoek ben. Je wilt kijken naar je KPI’s, je belangrijkste conversie funnels, je online marketing wil je zien. Dashboard 4 Wat goed van deze is, dit is duidelijk eentje die altijd doorontwikkeld had moeten worden. Goed aan deze is, zijn de meters, zodat je weet hoe je het doet. Hier kun je makkelijk een vergelijking invullen. En deze twee tabellen moeten anders. Bij die lijnen moet je gewoon keuzes maken, niet alles neerzetten. En deze, die zitten op nul, zie je niet welke nou bij welke hoort. Dit vind ik goed dat je die weken ziet. En het kleurgebruik is prima. Rood, geel, groen dat is wel duidelijk, waarschijnlijk ten opzichte van je target. Dit is de week en vergelijken met de week ervoor. Wat is duidelijk voor u in een dashboard? Dat ligt aan de doelgroep dat is heel verschillend. 20minuten Dit is de week, dus ja dit kan je wel vergelijken. Wat is duidelijk voor u? Dat is heel verschillend voor wat de doelgroep is voor wie je het gaat doen. Als je het voor een hoger niveau in de organisatie doet dan wil je het heel erg koppelen aan hele simpele dingen zoals wat was er deze week, wat was het ten opzichte van vorige week. Of wat is het cumulatief van het jaar, het cumulatief ten opzichte van target. Kijk je naar de vaste harde KPI’s, dan heb je een vaste manier van rapporteren vaak in een organisatie, waarmee het moet aansluiten. Dat mensen bij een rapportage online gelijk het format herkennen, dat is heel belangrijk. Ze hebben vaak niet de tijd om een half uur erop te gaan studeren hoe het nou in elkaar zit. Het is belangrijk dat het aansluit op bestaande rapportages. Als je hoger in de organisatie zit moet je duidelijk aangeven wat goed en wat niet goed
104
is. Dat doe je door orders aan targets te koppelen of aan het jaar daarvoor. Je moet goed rekening houden met de dynamiek van de business, bijvoorbeeld als je in een seizoen gebonden business zit, dan is het belangrijk om naar de zelfde week maar het jaar daarvoor te kijken. Als je meer in een business zit die meer met concurrentie te maken heeft, of weersinvloeden, of voorraad, of dat soort dingen, dan zijn er meerdere afzet kanalen, dan moet je het goed koppelen aan de channelmix. Dus online ten opzichte van de andere kanalen. Daar heb je business inzichten voor nodig om dit erin te zetten. Ik vind echt belangrijk dat een dashboard het juiste business inzicht laat zien. De aspecten waarop een organisatie op wordt aangestuurd, die moet je laten terug komen in een dashboard. Je moet in een oog opslag kunnen zien of het wel of niet goed is. Bijvoorbeeld als het online omlaag gaat en je ziet dat het offline ook slechter gaat, dan kan het aan het seizoen liggen, maar als je ziet dat je bij het online kanaal slechts een beetje omlaag gaat, dan doe je het helemaal nog niet zo slecht. Dat is echt de grootste uitdaging, want je hebt hier technische mensen aan werken, analytische mensen en je zou nog meer de business gedreven mensen hier bij moeten betrekken. Dat gebeurt in mijn opzicht te weinig. Als je dat voor mekaar hebt, dan heb je de basis. En dan moet je het er nog beter uit gaan laten zien. En ga je kijken wat daar weer uit komt. Name: Edgar Mul Organization: Conversion Company Age: 42 Field of study: Econometrie Level of analysing data: 7.5
105
Interview 4 Wat voor een soort organisatie werk je voor? Ik werk echt voor een food retail organisatie, voor Ahold. Wij zitten in met de afdeling in een bepaald onderdeel daarvan, het online onderdeel. Wat wij hier doen is die food retail proberen naar online te vertalen en dus echt het e-commerce stuk voor de organisatie op te pakken. Wat is jouw functie in de organisatie? Webanalist, dus ik vertaal de data die verzamelen online, naar bruikbare inzichten voor de organisatie. Dat ze dat kunnen gebruiken om de waarde van de organisatie te vergroten en sneller beslissingen te nemen. Wat voor soort dashboards gebruiken jullie? Heel veel verschillende dashboards. Vooral ook legacy, want we zijn nog maar een half jaar serieus bezig met de web analytics afdeling. Dus we hebben in eerste instantie dashboards gebouwd en we gaan nu echt naar de afdelingen. Zodat zij een dashboard hebben die aansluit bij hun werkzaamheden. Die ook de functies biedt die relevant is voor hun werk. Het platform geeft bijvoorbeeld veel inzichten. En inzichten in de verschillende campagnes. Het heeft wel een beetje dezelfde layout, maar daar zijn we vooral nog mee bezig. Omdat we pas aan het opzetten zijn. Nu gaan we over van de oude dashboards naar de nieuwe. Het uiteindelijke doel is ook dat het management team alle dashboards naast elkaar kan leggen om een goed overzicht te krijgen van alle activiteiten. Er is ook een algemene in de maak, daarnaast is er een soort analytics pack, waarin we in eerste instantie we letterlijk vertellen dit zijn de inzichten geweest, dit zijn de acties die we hebben uitgezet daar en dit heeft het uiteindelijk opgeleverd. Dat is meer voor het management team. Dus meer tekstueel, dit is wat we hebben gevonden, dit gedaan en dit heeft het opgeleverd. En daarachter hangt alle visuele informatie zoals alle dashboards en een algemeen dashboard. Dat is dus meer een visie die we hebben, maar dat is er nu nog niet. Voor welke niveaus van de organisatie wordt het gebruikt? Door iedereen. Van hoofd e-commerce tot online marketeer worden de dashboards gebruikt. Denk dat we echt willen bewegen naar een data gedreven organisatie. Dus het is niet zo dat in dat dashboard alle informatie staat ook voor het laagste niveau, want hoe lager je zit in de organisatie hoe gedetailleerder het is, je bent dan met meer gedetailleerde zaken bezig. Maar in dat dashboard staat dan wel het stukje informatie waarop de manager kan sturen, als dat ophoog gaat dan heb ik m’n werk goed gedaan. Daarachter hangt natuurlijk allemaal detail informatie voor analyse. Waarvoor gebruik je de dashboards? Belangrijkste is dat we ervoor zorgen dat ie dashboards ook daadwerkelijk gebruikt worden en dat ze ook aansluiten op de behoefte van de mensen in de organisatie, de verschillende afdelingen en verschillende managers binnen de organisatie. Daarbij hoort natuurlijk ook dat je je houdt aan een aantal hygiëne factoren, het moet altijd op tijd verstuurd worden, je moet er voor zorgen dat er accurate informatie op staat, daarnaast wil je ook echt dat ze het gaan gebruiken. Daarom is het voor ons ook een stukje vertalen van die informatie naar daadwerkelijke inzichten. Heel vaak komt er vanuit het dashboard een bepaalde vraag want de detail informatie staat er niet op want het is meer een tool of je het goed doet, minder goed doet of gewoon slecht doet. Daaraan is het alleen maar een graadmeter of het goed gaat of niet en dat kan je vervolgens gebruiken om diepere analyses te doen. En mijn taak is dan om er voor te zorgen dat die analyses ook gedaan worden. Ik doe die analyses die er achter hangen. Wat zijn de doelstellingen van een dashboard volgens jou? Ervoor zorgen dat de organisatie weet of wat ze doen ook daadwerkelijk waarde oplevert voor de organisatie. Wat zijn de voorwaarden van een goed succesvol dashboard Ik zou willen, hoe beter je erin bent om het heel erg simpel te houden. Je moet het simpel houden hoe minder metrics hoe beter. Als er drie metrics op staan dan ben je goed bezig dan kan je in een
106
oogopslag zien of het goed gaat met de organisatie. Je ziet toch wel vaak dat het meer uitgebreid wordt en dat er heel veel detail informatie in komt te staan. Daar ben ik minder fan van. Daarnaast vind ik het belangrijk dat je naast het getal ook de verhouding weet. Je hebt het aantal en het percentage van het totaal. En je moet trends erin kunnen zien. Niet een nummertje maar ook zien wat je ervoor hebt gedaan en een van de belangrijkste dingen is dat je het target hebt op het dashboard. Dus niet dat je alleen vergelijkt met wat je eerder hebt gedaan maar ook echt dat je een doel hebt waar je naartoe werkt. Dashboard 1 Ik zou deze een 7 geven. Het geeft goed inzicht. Ik zie er nog niet echt uit dat het helemaal gelinkt is aan de organisatie. Het is ook moeilijk af te lezen wat hier nou de belangrijkste metric is, wat is nou eigenlijk de KPI. Heel veel getallen maar als ik een manager was, dan zou ik me echt afvragen waarop ik me nou zou moeten focussen waarop moet ik sturen, wat is het belangrijkste voor mij als manager. Niet gelinkt aan de organisatie bedoel ik dat het meer dingen linken met elkaar. Ja ik weet ook niet precies wat het doel is natuurlijk. Maar ik zie heel weinig conversie punten terug. Ik ben wel een fan van staaf en lijn diagrammen en dat zie je hier in terug. Dat is volgens mij wel de beste manier van weergave. Maar het is wel allemaal heel klein, dat is ook wel een nadeel. Als je minder metrics hebt en echt weet waarop je je organisatie wilt sturen is dat beter. Nu zijn er gewoon heel veel metrics neergezet. Ik denk als je weet waarop je wilt sturen dan je zeker de helft weg kan laten. Ik denk dat de helft detail informatie is en niet echt voor een manager handig is om op te sturen. Er loopt toch wel veel door elkaar. Als je dit nog nooit hebt gezien lijkt dit me toch wel echt een lastig dashboard. Dashboard 2 Social dashboard zie ik. Dit vind ik lastig, ik zou ook een 7 geven. Het is ook weer veel informatie. Ik vind deze misschien toch fijner, omdat je per content blok één stuk informatie. Hij is toch simpeler. Toch een 7.5. hij is dus simpeler en dan kijk ik naar de content blokjes en het is een herhaling dus je kunt heel makkelijk de verschillende sites op hoofdlijnen vergelijken. Waar kan ik snel een overzicht krijgen van mijn organisatie dan zie ik dit sneller bij dashboard 2 dan bij 1. Het verschil met de vorige week staat er wel bij, maar dat je kunt bijna niet aflezen, okee maar waar begint en stopt de week bijvoorbeeld in de lijn diagram, dan liever een staaf diagram. En ook het week nummer ontbreekt. Is wel echt een minpunt van deze. Als ik niet kan zien in welke week we zitten..? We hebben wel reporting period, dus ik weet niet eens dat we hier naar weken of maanden zitten te kijken. Misschien toch een 6. Dashboard 3 Dit geef ik echt een 4. De stacked area chart vind ik echt heel onduidelijk. Dat het zo stapelt, ik kan geen totalen zien of iets. Maar waar is de legende. Okee klein daarboven. In de tweede kolom een piechart maar daar zijn de verhoudingen ook erg onduidelijk. Als je dat goed wilt weergeven wat zijn de percentages ervan. Nu zijn het gewoon visualisatie maar weinig inzicht. Pie-chart zijn meestal echt lastig om te interpreteren. Houd je vaak voor de gek, als je zit te vergelijken, welke is nou groter. Ik vind het maar lastig. Je denkt dan dat het elkaar opvolgd, maar dat hoeft niet. Bij een bar-chart zie je dat bijvoorbeeld veel beter en duidelijker. Maar stacked area chart en pie-chart zijn niet mijn favorieten. Managers vinden het vast mooi, maar als je er echt data uit wilt aflezen is het veel lastiger. Qua data visualisatie is het niet een heel duidelijk dashboard, de cijfertjes helpen dan weer wel. Dan kan je het zelf berekenen. Jammer dat er geen percentages bij staan. Dashboard 4 Ja ook geen fan van dashboards en dan letterlijk tellertjes neerzetten. Je ziet eigenlijk maar een cijfer en ik vind het heel moeilijk om van deze informatie af te lezen. Eigenlijk alles vind ik onduidelijk. Het correspondeert helemaal niet bij elkaar. De kleuren en het is onoverzichtelijk. Dan heb je conversions, allemaal op 1 grafiek weergegeven. Maar sommige hebben veel hogere aantallen dan andere, dus dit kan ik helemaal niet aflezen. Als het een paar honderd zijn, en andere duizendtallen. Dat maakt het lastig en het correspondeert niet. En wat is dit? Weken vergelijken, o dat kan je aangeven dat is wel handig. En het tab-systeem vind ik ook wel handig. Dat je extra informatie daar kan opzoeken. Maar de indeling zou ik wellicht iets anders maken. Met de tab, denk ik wil dieper de informatie ingaan, maar dan weet ik niet zeker welke tab ik dan moet aanklikken. Maar een
107
dashboard waarin je lekker kan klikken is goed. De weken, de tabjes. De navigatie is wel erg fijn altijd in een dashboard. Verder qua informatie er snel uithalen vind ik deze minder. Wat is duidelijk? Als je de trend eruit kunt halen. Het aantal maar ook de verhouding tot iets. Stel je hebt een metric op registraties, hoe veel registraties heb ik deze periode gedaan. Hoe is dat in verhouding met ervoor, en wat is de conversie verhouding. Hoeveel hebben hem binnen gehaald en hoeveel echt geregistreerd. Als je dat in een blok kan visualiseren is dat wel heel knap. Dat is bij dashboard 1 wel een mogelijkheid. En snelheid is dat belangrijk? Ja is belangrijk, het moet duidelijk zijn wat het precies is. De dingen die we nu hebben bekeken zijn nog best wel makkelijk te begrijpen. Maar attributiesystemen, dat is wel ingewikkelder. Hoe sneller iemand iets begrijpt hoe beter je visualisatie is. Hoe sneller hoe beter vind ik gewoon. En qua kleur. Je moet het dan wel gefocused houden op de nummers. En niet te veel kleur op de staven. Name: Michel Spalberg Age: 29 Organisation: Ahold Function: Web analyst Field of study: Bedrijfskunde Level of analysing data: Zeer ervaren
108
Interview 5 Voor welke organisatie werk je en wat is je rol daar? Bij KPN.com ben ik binnen het segment Residential dus dat is internet tv en bellen verantwoordelijk voor de online media. Dus eigenlijk het verkeer naar de website toe. Dus het aansturen van de Google partners, voor de search engines optimizations and advertizing, display, affiliate, email en dergelijke. Dus om er voor te zorgen dat er zoveel mogelijk kwalitatief verkeer naar de site komt en dat dat uiteindelijk gaat converteren. En daarbij als het gaat om dashboards en dergelijke natuurlijk heel erg om één inzicht te hebben van welk kanaal komt het bezoek en wat levert het uiteindelijk op als het gaat om orders. Met de partners waar ik mee samenwerk krijg ik natuurlijk per partner weer een ander dashboard of overzicht te zien, hoe goed dat kanaal heeft gepresteerd. Dus dat zijn veel verschillende. Voor display zijn dat er twee, voor search twee, in totaal denk ik wel vijf verschillende overzichten per week. Sommige hebben ook hun eigen dashboard met hun log in, maar waar ik uiteindelijk naar op zoek ben is één totaal dashboard, waar al die data in wordt geladen, zodat je met één druk op de knop kan zien wat is er de afgelopen week opgeleverd in totaliteit. In plaats van wat komt daar vandaan en wat komt daar vandaan. Zodat je beter kan vergelijken. Hoe zien die dashboards eruit die jij gebruikt? Ja een aantal offline, dat zijn voornamelijk in pdf of in Excel, waarbij je dus gewoon de data ziet op week niveau, of op dag niveau, op basis van een aantal variabelen. Met ook wel een aantal grafieken daarin. En in pdf eigenlijk ook, dus op zich wel heldere lijnen als het gaat om traffic, orders en dergelijke. En ja de online dashboards, die zijn eigenlijk wel hetzelfde, maar daar maak ik eigenlijk niet heel veel gebruik van, want dat is meer omdat je per dashboard weer een andere log in nodig hebt en daar ben ik best wel slecht in. En het is maar één netwerk is. En de data die eruit komt niet altijd het totaal behelpt. Maar visueel is dat wel redelijk maar nog te weinig in geschiedenis. Het afstemmen van de ene week op de andere week. Welk niveau van de organisatie worden de dashboards gebruikt? Dashboards die ik gebruik, eigenlijk alleen binnen het team. Waar ergens wel het probleem inzit is dat het nog ontbreekt aan één groot dashboard, dus dat ik alle data van de verschillende dashboards moet samen voegen in één Excel bestand en daar weer mijn dashboard moet uithalen. En dat is eigenlijk iets wat ik op wekelijks niveau aan het management presenteer maar ook aan mijn team. En daar blijft het wel bij. Dus zeg maar het verantwoorden naar boven en hoe ik het dan weer met mijn team ga aanpakken. Niet veel verder dan dat. Waar wel behoefte aan is, om daar ook een split in te hebben. Dat ik een dashboard voor mijzelf heb, om echt op detail niveau te kunnen zoeken, als ook een dashboard die top level is met grafieken voor managers die niet alles hoeven te weten op detail niveau of iets dergelijks. Dus ideaal zou zijn een overzicht met de daarachter de details in tabellen of iets dergelijks. Ja dat ik zelf vanuit mijn specialisme, daar dieper in zou kunnen duiken, en bepaalde verbanden kan zoeken ter optimalisatie van bepaalde activiteiten. Maar ook graag één overview waarbij je met één druk op de knop kunt zeggen, “kijk hier manager, hier dit is eruit gekomen.” Welke doelstellingen zijn belangrijk in een dashboard? Qua output, wat belangrijk is, is dat je de belangrijkste inzichten die je wilt hebben er zelf uit kunt halen. Dat het niet voor gedefileerd is maar dat je het zelf kunt aanpassen. Dat het in een opslag wel helder moet zijn dat waarop je op kijkt, dat je dat er ook wel gelijk uit moet kunnen halen in de vorm van visuals, dus grafieken of tabellen en trendlijnen. Je kan wel van de ene week op de andere week kijken maar ook wil ik zien hoe doen we het het afgelopen half jaar bijvoorbeeld. Is het een stijgende lijn of een dalende lijn.
109
Hoe moet het er uitzien? Als je de mogelijkheid hebt om verschillende dashboards uit te draaien, zoals in pdf ofzo, ja dan kan het handig zijn om er een bepaalde elementen in te hebben zoals bepaalde teksten of logo’s die er gewoon in vast staan. De gebruiksvriendelijkheid is wel een belangrijk natuurlijk. Het moet wel overzichtelijk zijn, maar huisstijl zou voor mij niet heel belangrijk zijn. Maar de output die je wilt delen die is wel belangrijk. Dashboard 1 In een oogopslag is goed te zien, qua cijfers, de veranderingen week op week. De staven is lastig om verschil te zien omdat het allemaal beetje op hetzelfde niveau is, maar de trendlijn is wel handig om aan te geven wat de verschuivingen zijn geweest. Het groene en rode werkt wel heel prettig. Dan weet je of het beter of slechter is gegaan dan de week ervoor of waarmee je het ook vergelijkt. Ook zonder dat je de data van de vorige week erbij ziet. Dus dat is wel overzichtelijk. Lijkt wel een beetje op mijn eigen Excel sheet, dat je met staafdiagrammen per week aangeeft hoe het ervoor staat en wat de resultaten zijn. Ja deze vind ik duidelijk. Als je kijkt naar puur hoe het er uit ziet, je zou bijvoorbeeld ook iets in een pie-chart kunnen visualiseren bij een bepaalde verdeling, maar meer op basis van het totale verkeer. Dan zie je de verdeling goed. De verhouding is hier best moeilijk zichtbaar, omdat het qua zoek of order kan het week op week niet heel veel verschillen, dus zou je iets meer moeten inzoomen eigenlijk op deze verschillen, die data per week. Je ziet hier wel vrij veel metrics. Dit zou op zich als je echt de diepte in wilt, dan is het wel prettig wat er per sectie gebeurd. Maar wat gebeurt er op hoofdlijnen, op traffic en content. Dus uiteindelijk heb je drie, vier of vijf grafieken nodig om de totale week te beoordeling. En dat zou dan eerder een management summary zijn, dus dat wil je wel zoveel mogelijk beperken in redelijk een oogopslag zien wat daar uitkomt. En andere grafieken zou je dan eerder gebruiken om op bepaalde dingen dieper in te gaan. Bijvoorbeeld om vergelijkingen te trekken. Dit is wel vrij veel om op weekbasis te bekijken maar ik kan me voorstellen dat als je niet wekelijks maar op gegeven moment wat dieper in wilt gaan op de materie, ja dan is dit wel heel prettig. Als die inzichten er zijn is dat goed, maar op hoofdlijnen zou het een stuk minder kunnen. Dashboard 2 Ja wat ik hier ontbreekt is de aantallen aan de zijkant en onderkant, welke weken het zijn en het niveau van de cijfers. Kan me voorstellen dat dit wel praktisch is, de tabjes, dat je wat dieper in kan gaan op verschillende niveau’s. ik denk dat dat wel sterk zou werken dat je echt kan verdiepen door te klikken. Dus eerst een overzicht en dan via de tab inzoomen. Ik denk dat het wel goed is om zoveel mogelijk kanalen hierin in door te meten. Door uiteindleijk inderdaad ook naar facebook en twitter te kijken en dit ook te vergelijken. Die trend kan je vergelijken is wel fijn. Ook hier met rood en groen werken, dat geeft ook wel in een oogslag weer of het goed of niet goed is gegaan ten opzichte van andere dagen of het gemiddelde, maar ja de enkele lijnen wordt wel lastig omdat je het wel met elkaar wilt vergleijken en ook de dadta ernaast wilt zien. Maar de assen ontbreken hier wel echt. De mogelijkheid om per tab die inzichten te krijgen is el ech tgoe,d dan kan je echt die diepte analyse maken. Maar het toale verband is hier ook goed terug te zien. Dashboard 3 Dit maakt het wel makkelijker om te onderbouwen omdat hier ook alle cijfers bijstaan en de data en de verschillen. Maar ook de combinatie met het visueel weergeven ervan. Ja als je dit vergelijkt met december met november dan zie je weinig verschil in de grafieken terug terwijl dat er wel is maar dat is maar een paar procent, de piechart zegt dan ook niet heel veel. Dat kan je op maand dan ook minder goed gebruiken, omdat op maand niveau het altijd een beetje hetzelfde zal zijn. Dus dan kan je daar beter geen piechart voor gebruiken. Plus dat hier ook ontbreekt wat wat is, o daarboven zie ik
110
een legende maar die is nogal klein. Dat zou je misschien, dat is een beetje afhankelijk voor hoe vaak je rapporteert maar het is misschien goed om een keer in het halfjaar aan te geven hoe iets wordt gebruikt. Als het gaat om traffic om dit geval. Om voor mensen die er wat verder vanaf staan. Qua data is het veel meer op het niveau van verschillende trafficbronnen met elkaar te vergelijken en optimaliseren. Dus als mediamanager kan ik daarop wel goed bijsturen. De verbanden zien en daarop inspelen is wel echt belangrijk. Dus meer als gebruiker dan op management niveau. Het rode en het groene en de verschillen maand tot maand zou misschien een toevoeging zijn. Om te kijken of een kanaal structureel minder wordt, dus de wat bredere verbanden. Ja met de stacked area chart vind ik het een beetje lastig, hoe je dit moet aflezen. Het formaat is ook een beetje klein, de trend die beweegt mee met alle verschillende lagen, dus dat is wel lastig af te lezen, omdat je m stapelt. Per kanaal kan je het heel lastig afzonderlijk zien. Dan zou ik eerder met lijnen gaan werken. Dashboard 4 Wat sterk hier is, is dat je een general overview hebt en direct zou kunnen gebruiken in een week rapportage. Hoe is het afgelopen week geweest, of weken met elkaar kan vergelijken of een enkele week zou kunnen uitdraaien. Op de hoofdlijnen, op basis van KPI’s of iets dergelijks. Het ziet er zo in eerste instantie heel overzichtelijk uit. De metertjes vind ik ook wel goed, lopen we voor of niet, lopen we op target of niet, vind ik wel goed aangegeven. De trend kan je goed zien het ene kanaal ten opzichte van de andere. Dus ja ik denk dat deze het gebruiksvriendelijkst en op managementniveau het meeste laat zien. De mogelijkheid om er dieper op in te gaan, met tabjes is ook fijn om dieper te spitten in wat kan beter en wat kan aangepast worden en wat niet. De export mogelijkheid is ook altijd erg fijn. Maar ook als dat naar pdf zou kunnen, dan kan het met een druk op de knop doorgestuurd worden. Of per email misschien, dan heb je er zelf niet zoveel werk van. Het is gewoon erg overzichtelijk. Wat is duidelijk? Als iets goed visueel is dan kan je daar wel veel uit halen. Ook met rood, geel en groen, de vergelijkingen met vorige weken, of het goed gaat of niet. In zoverre is het overzichtelijk dat je het in een oogopslag is het duidelijk. Dus niet te veel hoeft te zoeken, wat betekent dit en wat betekend dat. Als je iets redelijk snel ziet. Maar wat duidelijk is, het belangrijkste is de kleuren denk ik en de verschillende lijnen. En mogelijkheden om te vergelijken. Door de kleuren zie je het echt gelijk, dus dat vind ik heel fijn. Maar dat zal ook wel een beetje gewenning zijn, wat is duidelijk. En belangrijk dat je het snel ziet? Ja dat denk ik wel, voor iedereen wel. En als je echt zegt op basis van die data gaat optimaliseren, raak je er wel aan gewend hoe het werkt. Binnen KPN, het hogere management, die willen gewoon een paar regels met wat gaat goed wat gaat fout. Verder hebben ze niet echt de tijd of kennis om daar op in te gaan, dus dan hebben ze aan een halve minuut wel voldoende. Dus op management niveau is dat wel echt belangrijk. Dus op niveau van hoe gaan we er op sturen, is de gewenning van toepassing. Dan heb je meer behoefte aan vergelijken van de data en het spelen met de data. Dan heb je ook de niveau’s daaronder. Name: Erik de Jong Age: 36 Organization: KPN.com Function: Online media manager KPN.com (internet, tv, bellen) Field of study: Bedrijfs Communicatie en Digitale Media Level of analysing data: High (7.5)
111
8.4 Results quantitative research All results of the quantitative research can be found on the CD-ROM under ‘Results quantitative research.csv’
8.5 Recommendation Report for Conversion Company
Recommendation Report Conversion Company To:
Conversion Company
From:
Jaimy Quadekker, research intern Conversion Company
Date:
June 2013
Subject:
Master thesis research: Data Visualization in Online Dashboards: Optimal visualization of abstract data in online dashboard design
In this recommendation report I present my findings of my master thesis research ‘Data Visualization in Online Dashboards: optimal visualization of abstract data in online dashboard design’. I developed a new method and provide an optimal approach for the design of online dashboards.
112
Recommendation Report Conversion Company Data Visualization in Online Dashboards Optimal visualization of abstract data in online dashboard design
Jaimy Quadekker Graduation School of Humanities Media Studies University of Amsterdam June 2013
Warmoesstraat 83b, 1012 HZ Amsterdam Phone: 0031618040448 Email:
[email protected]
113
Abstract: In the data-driven culture that we are living in today, visualization is a great tool to display big and complex datasets. Using information visualization for abstract data in the most optimal way represents a different way for people to view and better understand data and the underlying structure of the data. Online dashboards are a specific form of information visualization and can be seen as visual summaries that provide insights on online performance. Key purposes of these insights are discovery, decision making, and explanation. With an empirical research, divided into a qualitative and a quantitative part, I found an answer on the main question: ‘What is the most optimal visualization of abstract data in online dashboard design?’ In this thesis I develop a new method for the most optimal visualization design in online dashboards. Focusing on the main question draws on a two part empirical approach, combining qualitative and a quantitative methods. The findings
show that an effective dashboard should be focused on the most important information and should provide immediate and clear insights to the user. For different information types, different visualization form should be used, containing mainly bars, lines, or stacked areas. With the results a bridge is created between practical experience, theory and scientific testing. Combining the seeking mantra of Shneiderman, the adjusted principles of the data visualization design process of Tufte, the adjusted principles for the general information visualization process of Mazza, two key point to amplify cognition in online dashboards by Card, Mackinlay, and Shneiderman, and the general background information of Few, added by the perceived results of this thesis, create a new method. Together they form the perfect basis for the design of an optimal online dashboard.
114
Table of Contents
1
INTRODUCTION
116
2
METHOD
117
3
RESULTS
119
4
RECOMMENDATIONS
123
5
CONCLUSION
128
115
1. Introduction The current design of online dashboard can be improved, changing the current practical approach into a more empirical, scientifically tested approach. The key objective of my thesis is to provide an improved method which could be applied on online dashboard design and that Conversion Company can enhance their online dashboards with a more scientific approach and scientific results. This new method provides a bridge between practical organizations and scientific theory. My empirically based research results are based on an empirical research that is subdivided into a qualitative and a quantitative research part, which, I believe, provides a new method for designing the most optimal dashboard, including the most optimal visualizations of data in online dashboards, based on information type. In this recommendation report I elaborate on the used method and explain the obtained results of the qualitative and quantitative research. I continue with the recommendations and this report closes with the conclusion.
116
2. Method As a result of the research problem, the theoretical framework and the exploratory nature of the research, different questions are proposed to develop this new method. The main question is ‘What is the most optimal visualization of abstract data in online dashboard design?’ To answer this main question, three sub-questions need to be asked. Beginning to obtain a clear view on online dashboards, based on practical experiences, the question ‘What is the context and what are the requirements of an online dashboard?’ should be stated. The next sub-question is ‘How do you design the layout of a general online dashboard?’ Looking at the general layout, this question again could be subdivided into three questions ‘How does the design of a general online dashboard look like if you look at the amount of graphs?’, ‘How does the design of a general online dashboard look like if you look at the numbers?’, ‘How does the design of a general online dashboard look like if you look at the remaining elements?’ The last sub-question focuses on the graphs and therefore is ‘What is the best visualization per graph?’ Here I decided to focus on five different information types which provides the questions ‘What is the best visualization in combination with comparing magnitudes of independent values?’, ‘ What is the best visualization in combination with particular comparison with different entities?’, ‘ What is the best visualization in combination with particular comparison with equal entities?’, ‘ What is the best visualization in combination with change over time?’, ‘What is the best visualization in combination with and the overall trend?’ With these five questions, a clear focus per graph per information type is drawn. Looking at the nature of the question, I decide to break the research into two different parts, a qualitative and quantitative part. First I use a qualitative approach to mainly focus on the first sub-question about the context and requirements. With empirical approach containing personal interviews I try to provoke answers on the other questions, based on the practical experience of the interviewees. The second research is a quantitative approach, where the research questions are scientifically tested with a group of 148 respondents. Merging the two research parts together results in an extensive answer on the main question.
117
In each section of the different information types I focus on the time of interpretation, because in my opinion the time of interpretation is the time that a respondent needs to really understand the graph. Because I believe that the time a respondent needs to provide an answer is the same time a respondent needs to understand the graph. I also compare the last click with the page submitted, because this could show how secure a respondent is about the answer that he or she gave. In my opinion the quicker one submits the page after the last click of the answer, the more secure one is about that answer. Also the different opinions on the different visualization forms are discussed. Which visualization forms are mostly preferred and which are most disliked, according to the respondents? And every section is concluded by the results of correct versus incorrect answers, because not only the interpretation is important but it should also be the right interpretation. Below I only present the most important elements that have an influence on the best combination of visualization form with the specific information type.
118
3. Results Below I summarize and sensitize the key findings of the qualitative and quantitative research that are meaningful for answering the research questions. When I summarize the results of the qualitative research it becomes clear what the objective and desires are concerning the design of online dashboards beginning with the desire for a general dashboard, containing an overview and where the keywords are focused, clear, and immediately are adequate. Those three keywords are returning in almost all the objectives of an online dashboard. A dashboard should reflect the activities of an organization and how their online channel has performed during a specific period. It should inform and provide insights into the performance on a few focused goals, in other words how the organization is doing compared to the chosen targets and goals as set by that organization. It is very important that the information in an online dashboard is clearly translated into actual insights. Without explicitly saying it, this translation of information into clear insights is actually in other words the visualization process. They all desire a clear dashboard, but the only way that a dashboard will be clear is if you focus on the visualization-step of information. Moreover one of the objectives that stands out is the opportunity to compare the different insights with each other. A comparison demonstrates the performed activity of a certain period with another period. Also the possibility to navigate through the dashboard is a key objective, for example with several tabs. With the possibility to navigate the focus on the most important graphs remains, but more details can be obtained through this navigation. It gives a feeling of control and looks clear. The keywords are returning in the amount of graphs. Fewer graphs are considered as positive, because then one can work with a focused dashboard and it becomes immediately clear which insights are most important. Some explanation or legend should also be included in a dashboard, in order to make it readable for everyone who wants to work with it. An explanation or legend could provide more support to online dashboard, because it changes it immediately to a clear visualization.
119
The colors green and red are mentioned a lot. Those two colors are considered to work really well, but implementing orange is considered to be confusing, especially without further explanation. The use of more different colors is perceived as chaotic. Below, I only discuss the quantitative results that could be meaningful for the research questions or stand out of the rest of the results. As discussed in the results, I looked at the different elements that influence the most optimal visualization. Considering the interpretation time, the time between the last click and the page submit, the opinion about the different visualizations of the respondents, and the percentages of correct versus incorrect answers, the most optimal visualization form comes forward. Below I only present the most important elements that have an influence on the best combination of visualization form with the specific information type. Information type 1: Comparing magnitudes of independent values Merging part one and part two of this qualitative research provides the outcome for the information type ‘comparing magnitudes of independent values’. With the quickest interpretation time and a selection of seventy percent of the respondents, bars provides the best combination if you want to compare magnitudes of independent values in online dashboards. Information type 2: Particular comparison with different entities Combining the first and the second part of the quantitative research provides the outcome for the information type two, which is the particular comparison with different entities. The quickest interpretation is made with the visualization with lines and this visualization is also considered as the clearest visualization for this information type. As well as visualizations barline1 and barline2, in second and third place, but have fortytwo and twenty percent incorrect answers. With a ninety-five percent result of correct answers, in combination with the quickest interpretation and very preferable, lines results in the best combination if you want to visualize a particular comparison with different entities in online dashboards.
120
Information type 3: Particular comparison with equal entities Merging part one and part two of the quantitative research show interesting results for information type ‘particular comparison with equal entities’. In the first part, results show that the quickest interpretation is made in combination with the visualization stacked, and has a correct answer percentage of ninety-six. However in the second part, the visualization stacked is considers as third least clear visualization. Lines is secondly quickest interpret and is considered third best and clear visualization With the contradiction results noticed at the stacked visualization, the save option is to choose lines for the optimal combination between visualizing and a particular comparison with equal entities. Information type 4: Change over time A combination of part one and part two of the quantitative research for information type four, which is change over time, show that the quickest interpretation is made in combination with visualizations stacked or bar. With two measure points, they provide the quickest interpretation. They are also submitted the quickest. Bar is chosen by forty percent of the respondents to be the clearest visualization for change over time. Stacked is liked twenty-two percent, but also disliked ten percent. Notable is that the stacked visualization has with eighty-two percent the most correct answers. The bar visualization had the least correct answers, with sixty-five percent. This contradiction of the quickest interpretation in combination with the least amount of correct answers, does not make the bar visualization very convincing. Looking at the results, stacked is interpret after bar the quickest and has the most correct answers, but is not chosen by the respondents as very clear. The contradiction of what the respondents want and how they result in the first part of the research is a remarkable outcome. Therefore choosing stacked as the optimal combination between visualizing and a particular comparison with equal entities, since I consider part one of the research more trustworthy. Information type 5: The overall trend Merging the first part and the second part of the research provides interesting results for information type five, visualizing the overall trend. The quickest interpretation is made in combination with visualizations bars, and has the highest percentage of correct answers with ninety-two percent, and on top of that is chosen as second most clear 121
visualization in combination with visualizing the overall trend. Followed by barline1 that is secondly interpreted the quickest and thirdly chosen as most clear by the respondents. Therefore I can conclude that a visualization with bars of a combination of bars and lines is the most optimal for visualizing the overall trend in online dashboards. A side-note that I want to make for information type five, the overall trend, is the rather high percentage of incorrect answers on all nine questions. With exceptions of the visualization with bars. A possible reason is inexperienced of the respondents with this type of question and visualization. There is a plausible opportunity that this information type or graph is not read correctly as a cumulative graph, but interpreted differently and therefore wrongly. Amount of Numbers Looking at the amount of numbers that should be included in a graph, the merge of the qualitative and quantitative research provide a combined result. Numbers are considered as extra yet focused information, and therefore as positive. But there are preconditions to this fact. The numbers should be clear immediately, with the support of an explanation or a legend. For example to what are the numbers compared? Also the numbers should be comparable themselves with other numbers, and again this should immediately be clear. There are no significant results on the amount of numbers and how they should be presented.
122
4. Recommendations Below I present my new method that is required to provide the most optimal visualization of abstract data in online dashboard design. First I demonstrate the requirements for the general dashboard. Including these focus points into the online dashboards design provides an optimal dashboard according to practical experience in combination with scientific testing. Continued with table 20, that is focused on the visualizations of graphs, divided into five different information types using the most optimal visualization form in combination with a suitable information type results in the most optimal interpretation.
Obtaining optimal visualization result if you focus on the bullet points below while designing an online dashboard: During the visualization process keep the keywords focus, clarity, and immediately in mind, these are adequate. These three words are translated from the Dutch interviews and therefore I shall explain them. With a focused dashboard is a dashboard that is concentrated on a few key points meant and it should not be too big with lots of information, but focused on only the most important information. Clarity is the second key objective and by that is meant that the information that is visualized should be obvious and understandable. The last objective is immediate and by that is meant that the information should be readable at a glance. The insights in a dashboard should be clear right away. Follow the six principles of Tufte, with one additional principle, for a guidance during the design process of data visualization. o The first is the documentation of the sources and characteristics of the data. o The second is the focus on an adequate comparison. o This is followed by the demonstration of the cause and effect with comparable mechanisms. o And back in return those mechanisms should be expressed quantitatively. o The fifth attention point is the acknowledgement of the multivariate nature of analytic problems. 123
o And the last sixth principle is the inspection and evaluation of the alternative explanations. o Don’t forget to focus on the seventh principle that is added, that focuses on the ‘making sense’ of the visualization, by using table 20. Taking one step further into the direction of information visualization, different elements are important in the building process. Listed below are the five basic principles by Mazza that need to be considered during the information visualization process, concluded with one additional step: o The problem. This relates to what has to be presented, demonstrated, or found. o The nature of the data. There are different data types, data could be numerical (a top-5 list), ordinal (non-numerical data having a conventional ordering, such as days of the week), and categorical (data with no specific order, like cities). o Number of data dimensions. Depending on the number of dimensions, representations can be univariate- (one dimension), bivariate- (two dimensions), trivariate- (three dimensions), and multivariate data (four or more dimensions). We perceive our world in three spatial dimensions, therefore interpreting up to three dimensions is rather easy. But things with more than three dimensions however, are very frequent in real world situations and represent one of the most challenging tasks in Information Visualization. o Structure of the data. This could be linear (data coded in plain data structures, like arrays or tables), temporal (data which changes during the time), spatial or geographic structure (like maps or something physical), hierarchical (like structures in organizations), network structure (representing relationships between two nodes). o Type of interaction. Whether the resulting graphical representation is static (like a print or a static image on a display screen), transformable (features like zooming or filtering), or manipulable (users may control parameters during the process of image generation). o These five basic principles are focused on the goals of the visualization and the type of data. The last principle focuses on the representation, but missing is the step after that, which is the interpretation, sense making, and understanding step. In the field of online dashboards this step is essential.
124
Amplify cognition with the two principles of Card, Mackinlay, and Shneiderman, which are supporting the last principle Mazza. With these tricks visualization could amplify the cognition and support the perception and leads one from abstract data to more specific and concrete insights and knowledge. o Make it easy to find information by grouping it together. During the visualization process this would be helpful to amplify the cognition. o Make use of the detection of patterns; recognition instead of recall is an example to amplifying cognition Taking into account that recognition works in a more beneficial way than recall for amplifying the cognition, should definitely be considered during the design process. There should be a possibility to navigate through the dashboard, by using the Information Seeking Mantra of Shneiderman. The best way is to use the mantra is to begin with an overview, like a general dashboard, and continue by zooming, for a deeper analysis on the information, for example with tabs. After this analysis details on demand could be found in the analytical database. To respond on the resulted desire, this powerful information seeking mantra should be included in the design process of online dashboards. Then I would like to continue with some key desires, obtained from of the qualitative and quantitative research results: o An online dashboard should reflect the activities of an organization o An online dashboard should provide insights into the performance on only a few focused goals. Fewer graphs are considered as positive, because this focus makes sure that the insights become immediately clear. o There should be an opportunity to compare the different insights with each other, keep that in mind while designing the different elements of a dashboard. o An explanation or legend should be included Usage of color: Only use meaningful colors for numbers within graphs: the colors green and red are preferred. Other or more colors (such as orange) are experienced as chaotic.
125
Usage of numbers: Numbers are considered as extra information. They should only be used within a graph, if they are clear immediately, with the support of an explanation or a legend
Below you find the table for the design of graphs, focusing on five different information types and their most suitable visualization form:
126
Information
Recommended
type
visualization
Information type 1
Comparing magnitudes of
Bars
independent values
Lines Information type 2
Particular
Reason
Example
Quickest interpretation and preferred
Quickest interpretation and preferred
comparison with different entities Combination Preferred, but possible bars and line incorrect interpretation Quickest interpretation
Stacked Information type 3
Particular
and high percentage correct answers, but not preferred
comparison with Second quickest
equal entities Lines
interpretation, third preferred visualization
Quickest interpretation, Stacked
Information type 4
high percentage correct answers, but less preferred than bars
Change over time
Second quickest interpretation, Bars
preferred, but rather low percentage correct answers Quickest interpretation,
Information type 5
The overall trend
Bars
highest percentage of correct answers and preferred
127
5. Conclusion With my research I found the answers on the questions about the context and requirements, the layout of a general dashboard and the best visualization within the use of graphs with a focus on five different information types. When I merge these answers with insights and theory, I have my answer on the main question: ‘What is the optimal visualization of data in online dashboard design?’ With a combination of my perceived results based on my observations and research, the modified theories of Tufte and Mazza, two applicable principles of Card, Mackinlay, and Shneiderman, and the useful elements of Shneiderman’s mantra, a new method with basic principles is created which can be used to design the most optimal online dashboard. With this new method I seal the gap and form a bridge between practical experience, theory, and scientific testing. With the recommendations in section 4 this recommendation report can be used to design the most optimal online dashboard. For extra information and details, I would like to refer to the complete version of my master thesis
128
