In the pressure to meet customers’ changing needs, we in libraries are increasingly adopting user-centred design (UCD) approaches to the development of our services. Innovative experiments on storytelling and other ethnographic methods (Tilley, 2016) or libraries as living labs (Culén & Gasparini, 2013), for example, encourage us to increase our competence in inspiring design methodologies. On the strategic level, new approaches help in envisioning and leading libraries to the future (e.g., LIBER, 2017). An example of sharing experiences of design approaches with peer librarians is the Frilux initiative, developed by the Oslo University Library. Frilux aims to disseminate knowledge and understanding about design thinking and user experience (UX) particularly among Scandinavian library professionals.1 Echoes of similar enthusiasm on designerly approaches can be heard over Europe (Priestner & Borg, 2016).
However, is it worthwhile to invest librarians’ time and effort in UX or UCD? In the pressure of limited resources, library directors and managers who have embraced the “UX State of Mind” and are enthusiastic about the design thinking approach, have to be able to justify the allocation of resources to this activity. There are competing targets for resource investments, and convincing evidence of the benefits of user-centred design activity is needed.
For decades, libraries have developed and used performance indicators that show the quality of their resources and services (Hernon, Altman, & Dugan, 2015; Poll & te Boekhorst, 2007; Renard, 2007). The international standard ISO 11620:2014 (International Organization for Standardization, 2014) provides a framework for monitoring and assessing the success of library activities, both in the physical and digital realm. Along with new user-centred approaches, libraries have expanded their means of collecting qualitative and quantitative data of their customers (Priestner & Borg, 2016). Still, we seem to be missing a framework with which we can evaluate the design performance, the activity of designing. This situation evokes a second question: What is essential when evaluating and managing the UCD performance in libraries?
To address these questions, we present a study that examines the performance management of UCD activity in the context of libraries. Our focus is on digital library services, because of the penetrating effect of information and communication technology (ICT) in the contemporary library service landscape. The purpose of our study is to examine which are the conceptual elements of evaluating and managing user-centred design performance in the library context. We are also interested in the relations between the elements.
User-centred design (UCD) is treated here broadly as a wide spectrum of design activities and approaches. The concept incorporates ideologies and meanings from the domains of human-computer interaction (HCI), service design (SD) and user experience (UX) design, bridging different concepts such as participatory design, co-design and usability. The common denominator for these varying approaches is the aim to take a human-centric approach to library services’ design and development.
The purpose of this paper is, first, to contribute to the academic discourse on libraries’ performance management and value assessment. To this end, we reflect the existing knowledge about digital library services and design management against the recent theoretical understanding of public sector performance management. We complement the conceptual elaboration further with the theories of temporality, which is an emergent area of interest in public services design. The speed of change and prompt modes of participative design have drawn practitioners’ and scholars’ attention to the aspects of time (cf. Culén & Stuedahl, 2017). By joining these perspectives, we aim at capturing some essential aspects of the operating environment, where libraries design and provide their services. Figure 1 illustrates the scope of this study in the intersection of the application domain and theoretical perspectives.
Grounded on the examination of existing literature and theoretical contemplations, we propose a new conceptual model for value-based evaluation and management of UCD performance in libraries. The model guides our way to the second purpose of this paper, which is also our primary motivation: We aim to contribute to the library practices and enhance the position of UCD in organisation cultures via developing methods of justification. The conceptualisation is one step in the direction of constructing instruments and established routines of UCD performance management. Also, we provide empirical findings from two cases where we tested the new conceptual model.
The rest of the paper presents the study as follows. In the second chapter, we explore the existing literature and synthesise it into an integrated framework that includes the key concepts for further elaboration. After identifying the gaps in the existing knowledge, we pose our research question and describe the research process. As our main contribution, we propose a new conceptual model for evaluating UCD activities in libraries. We also describe how others could apply the conceptual model to real-world situations. To test the model, we apply it to two digital library cases. After a brief discussion, we conclude the paper with final notes that offer directions for future research.
2. Theoretical Perspectives
The knowledge of user-centred design in the context of digital libraries is growing. To expose some characteristics of the contemporary service provision landscape, we examined the literature from two theoretical perspectives. First, the performance management perspective explains why and how performance evaluation and measurement is considered necessary for public service providing organisations, including libraries. The second perspective examines the phenomenon through the concept of time, which is an integral element of performance management and measurement, but which is often bypassed without a proper conceptualisation.
2.1. Performance Management Perspective: Evaluation of Public Values
Performance management is about controlling the relationship between effort and delivery of the desired outcomes (Van Dooren, Bouckaert, & Halligan, 2015). All organisations, private and public, have to be able to show how well they are fulfilling their purpose. The purpose is relative to the viewpoint of interest and the changes in the environment (op cit.). Libraries, having the essential role in facilitating information to people, are influenced by the attitudes and norms addressed to public services.
Since the late 1980s, there has been an increasing demand for public sector reforms, which has affected all service providers with a public role. The New Public Management ideology has adopted elements from the private sector that aim at improving the effectiveness of public service provision (Hyndman & Lapsley, 2016; Osborne, Radnor, & Nasi, 2012). Despite some criticism (e.g., Lilburn, 2017), management and measurement in the public sector is an area of continuing interest to both scholars and practitioners (Düren, Landøy, & Saarti, 2017; O’Flynn, 2007; Stenvall & Virtanen, 2017; Talbot, 2010).
In recent years, the studies on public service management have shifted the focus from mechanistic performance budgeting towards a value-based evaluation of the performance. The Public Value theoreticians position the public value creation into the core of public sector performance (Guthrie, Marcon, Russo, & Farneti, 2014; Scott, DeLone, & Golden, 2016; Talbot, 2010). This new paradigm emphasises the role of public managers in securing, balancing, and advocating public values (Moore, 2014; Spano, 2014). Moreover, Public Value management is not just about controlling the resources, but also about controlling the created outcome, i.e., the values and benefits for the citizens and all other stakeholders of public services (O’Flynn, 2007). This approach is not far from the performance and impact evaluation that libraries are familiar with: the “fitness for purpose” evaluation (Poll & te Boekhorst, 2007).
Libraries have experience in systematically collecting qualitative and quantitative data through web analytics, surveys (such as LibQUAL), and some more ethnographic methods (e.g., Carlsson, 2016; Renard, 2007; Sinikara, 2006). Libraries consider the Balanced Scorecard (BSC) approach an inspiring framework in the context of their services, as BSC takes a broader view of performance (Corrall, 2015; Passonneau, 2013; Tanner, 2012).
However, it seems to us that the existing evaluation models, frameworks and standards are focusing on the value of libraries’ collections and services rather than on the value of their operations. Besides, design activity has its unique characteristics that require consideration in the management and evaluation of operations (Harpum, 2004). We do not, therefore, consider these models and frameworks applicable to the assessment of UCD performance as such.
To address this gap, we apply the Public Value approach and, specifically, Moore’s (2014) Strategic Public Management triangle (Figure 2), to the performance evaluation of library services. According to Moore (2014), strategic management of public services consists of three conceptual entities: 1) legitimation and support, 2) operational capacity and 3) public value. Each element is in dynamic relation to the others; stakeholders provide an organisation with legitimation and support to operate. Through operational programs, policies and procedures organisations build their capacity to create value. The outcome of this value-generating process is the public value, which, according to Moore (2014), takes the form of individual and social values.
2.2. Temporal Perspective: Evaluation for the Future
The cultural-psychological time is an utterly essential element influencing our actions and world-view. There is a growing body of literature on temporalities in the organisational settings (e.g., Rubin, 2007b; Whipp, Adam, & Sabelis, 2002), and particularly, in future studies (e.g., Voros, 2003, 2006). In the scholarly discourses of design or library studies, temporalities have remained under-theorised. Nevertheless, aspects of the time, i.e., past, present and future, are interwoven on our everyday discourses.
Temporal information gives managers a basis for decisions concerning the efficient and effective use of design resources. They can use existing data from past projects or estimation data for future projects in prioritisation between individual design projects (Best, 2010; Karat, 2005). Ex-ante estimations and ex-post calculations can also help with choosing between various design procedures and methods (Mayhew & Tremaine, 2005).
In the context of design, comparisons with temporal data may be more challenging. Good design can bring positive results, which were not anticipated in the initial goals and, therefore, not taken into consideration in the evaluation (Hirsch, Fraser, & Beckman, 2004; Whicher, Raulik-Murphy, & Cawood, 2011). On the other hand, managers who are responsible for the development may not be accountable for future costs affected by the lack of design (Siegel, 2003). Thus, there can be significant temporal gaps between inputs and outcomes of design. From a holistic and value-oriented viewpoint, the power of design lies in moving from short-term problem solving to long-term improvements (Cockton, 2006, 2008; Design Council, 2015; Friedman, Kahn, & Borning, 2008; Sikorski, 2008).
The internet has redefined our perceptions of time by increasing the speed of communication. It is also vital for libraries to envision the different futures, the possible as well as the preferable ones (cf. Voros, 2003). Studies on libraries’ ability to meet the requirements of the future society have warned that the traditional approaches to strategy and management may become disconnected from the rapid societal and technical change (Batt, 2015; Kallinikos, Hasselbladh, & Marton, 2013). The concepts of time can help managers to better frame and make sense of organisational change (Wiebe, 2011). In our current society, the temporal structures of work time are seen to become more layered than linear (Rubin, 2007a). When discussing different temporal aspects of time, it is important to use flexible ‘timescapes’ and allow context-dependent interpretations (Keenoy, Oswick, Anthony, Grant, & Mangham, 2002).
Building on these notions, we argue that the influence of time (in the psychological meaning) should be explicitly expressed in the conceptualisation of UCD performance management. Furthermore, we embrace the orientation to the future and consider it essential to take a foresight approach to performance management and evaluation.
2.3. Integrated Framework and Empirical Key Concepts
As described in previous chapters, our theoretical examination builds on the Public Value approach, complemented with the theoretical perspective of temporality. The Strategic Public Value triangle by Moore (2014) serves as a primary framework, but it lacks the aspect of time, which we considered essential to conceptualise for contemporary performance management. By extending Moore’s triangle with the concept of time, we construct a strategic public value diamond that consists of four interrelated theoretical concepts: 1) legitimation and support, 2) temporality 3) public value and 4) operational capacity.
Figure 3 illustrates the Strategic UCD Management diamond, which constitutes the integrated theoretical framework for our study. We bind the theoretical framework in the context of our research with four intermediary empirical key concepts. First, the digital library Stakeholders give the legitimation and support for the UCD activities. Second, the design performance is managed and evaluated through different temporal Phases. Third, the public value gained in design can be operationalised through Benefits that are received from the different outcomes of design. Finally, Design activities, which can have a variety of possible approaches and processes, constitute the operational capacity of design. Table 1 displays the empirical and the theoretical key concepts of the integrated model, and their appearance in the literature.
|Theoretical key concept||Empirical key concept||Publications|
|Legitimacy and support||Stakeholders||Moore, 2014; Van Dooren et al., 2015|
|Temporality||Phases||Keenoy et al., 2002; Rubin, 2007a,b; Wiebe, 2011|
|Public value||Benefits||Cockton, 2006; Friedman et al., 2008; Moore, 2014|
|Operational capacity||Design activities||Best, 2010; Hirsch et al., 2004; Law, Hvannberg, & Cockton, 2008|
The ability to evaluate and compare operations lies in the core of management functions. However, some design theoreticians pose strong arguments against justifying incremental benefits of different design activities (Hirsch et al., 2004; Siegel, 2003). Instead, they point out that design should be considered more as a culture that, if persistently cultivated, accrues significant benefits in the future.
For this research, we decided to treat the empirical Design activities with a similar “black-box” approach. Thus, the concept has its role in the framework as a strategic entity, but not as a target of in-depth empirical examination. This approach allows using the framework in the context of diverse design approaches, independent of their processual and methodological differences.
3. Research Question and Process
In search of a precise research question for this study, we took an overview of the existing literature on the aspects relevant to our topic. We used the theoretical perspectives as a filter to identify conjunctive and potentially complementary elements in the literature. The outcome of the first phase was a framework that integrates the two theoretical perspectives and defines the key concepts, both theoretical and empirical, in a suitable composition for our purpose.
Placing the integrated framework in our research context, i.e., the design of digital library services, directs us to the research question:
RQ: How do different stakeholders assess the temporally aggregated benefits of user-centred design (UCD) for digital library services?
This question leads us to the second phase of our study: setting and defining a conceptual model for UCD performance evaluation and management. We build the model on the integrated framework and the key concepts. Our purpose is to construct a conceptual model that falls into the intersection of design and science. Our model serves as an independent epistemic tool (scientific model, cf. Knuuttila, 2010), as well as a creative artefact (designed model, cf. Krippendorff, 2006). By a generous use of visual representations, we aim at reinforcing the understanding of presented arguments (cf. Nelson & Stolterman, 2012).
The construction of the model requires operationalisation and concretisation of the empirical concepts. The first step is the identification of the stakeholders involved in the UCD. Secondly, we must define the benefits of the UCD of digital services. Finally, there is the temporal aggregation of UCD benefits to be established. In the fourth chapter, we define these elements and their relations.
To empirically test the model in the context of our interest, i.e., the digital library service development, we apply the model to two cases in chapter five. We gather data of these cases from the existing project documentation and conduct semi-structured interviews of the key informants. Overall, our study consists of three phases. Figure 4 outlines the phases of this study and depicts the core content of each phase.
4. The 360-Degree Temporal Benefits Model
In this paper, we propose a new conceptual model for assessing the benefits of design activities. We based the new model on the integrated framework described in chapter 2.3, and it elaborates on the empirical key concepts defined in the framework.
We call the new conceptual model 360-Degree Temporal Benefits (360°TB Model, in short). The name reflects the core ideas. First, the model takes a comprehensive 360-degree view of all stakeholder groups. Second, it acknowledges the temporal aspects of performance. Third, it captures the benefits of the design activity. Our objective is to depict the key elements and their relations that constitute the design activities in the contemporary library service landscape. The model is applicable specifically in the context of designing digital services. The model aims to direct the evaluation and management of the UCD performance towards better understanding and control of the design activities.
4.1. Engaging Stakeholders in the Evaluation
The first element of the conceptual model is the Stakeholder, which is related to the theoretical concept of Legitimacy and support in the Strategic UCD Management diamond. By including this element in the conceptual model, we give an active voice to all people who contribute to or have interest in the design of a service.
In the context of digital library services, the benefits and value of the design activities are not only relevant to the organisation responsible for the development of the service, but also to various internal and external actors. As Tanner (2012) emphasises, only stakeholders can transform the understanding of output into a significant outcome. We argue that this also applies to the outputs of UCD activity. Therefore, our model engages stakeholders in the joint evaluation and promotes open and public conduct of the evaluation process.
Identification of relevant stakeholders requires explicit positioning of the centre of dominant interests. Most often, stakeholder analysis is organisation-centric (Friedman & Miles, 2006), and therefore it may take extra effort to objectively identify stakeholders from the viewpoint of the service or the benefits it accrues. Applying these notions to the classic definition of stakeholders (Friedman & Miles, 2006), a stakeholder in our model is an individual or a group who can affect or is affected by the activity of user-centred design (UCD).
In the interest of feasibility, the range of identified individuals should be segmented into purposeful stakeholder groups. The grouping should be context sensitive, and therefore we recommend relying on the empirical material or some existing stakeholder map (e.g., Eskerod & Jepsen, 2013). Design theories or guidebooks also provide feasible classifications for design-centred analysis of involved people (Bryson, 2004; Löwgren & Stolterman, 2005). Figure 5 illustrates exemplary stakeholder groups, who can be engaged in the evaluation.
4.2. Categories of Benefits
The second element of our conceptual model is Benefits, which is related to the concept of Public Value in the theoretical framework. Literature implies that individuals and groups representing various fields or domains may have conflicting values and interests about public services (Botero, Paterson, & Saad-Sulonen, 2012; Wijnhoven, Ehrenhard, & Kuhn, 2015). Thus, we can assume that we may also find differences in their perception of values and benefits of design. The element Benefits in our model aims to capture Stakeholders’ divergent views of the goals of design performance.
The literature suggests a plethora of possible benefits and values which organisations can pursue in their design activities. The objectives of design can and should be analogous to the objectives of the service itself (Krippendorff, 2006). Altogether, we identified from the literature nineteen distinct benefit categories for UCD, such as cost-savings, efficiency, innovation and trust. These categories are partly overlapping, some of them apply better for a particular type of services and in a particular context. Moreover, the literature presented some of them only for private sector services, e.g., the value of intellectual property rights. Still, we considered all benefit categories as potential in the context of digital library service provision and therefore included them in our examination. Table 2 presents the nineteen categories of benefits of UCD derived from the literature.
|ID||Benefit category||Benefit definition||Phase||Publications|
|C1||Cost-savings||UCD diminishes the costs of redesign and redevelopment later in the process||During implementation||Bias & Mayhew, 2005; Hirsch et al., 2004; Lockwood, 2009; Mantei & Teorey, 1988; Rajanen, 2011|
|C2||Cost-savings||UCD decreases the costs of supportive operations (e.g., user-support, training and documentation)||During implementation, operation and maintenance||Bias & Mayhew, 2005; Donahue, 2001; Hirsch et al., 2004; Rajanen, 2011|
|C/L||Competence/ learning||UCD increases the competence of the organisation/people involved in the design activities||Indirect benefit, in undefined future||Best, 2010; Design Council, 2015; Lockwood, 2009; Siegel, 2003|
|CO||Commitment||UCD increases the commitment of people involved in design activities (end-users and employees)||Indirect benefit, in undefined future||Bias & Mayhew, 2005; Hirsch et al., 2004; Lockwood, 2009|
|E1||Efficiency||Design approach speeds up the production process and diminishes delays when problems are detected early in the process||During development and implementation||Bias & Mayhew, 2005; Donahue, 2001; Lockwood, 2007, 2009; Mantei & Teorey, 1988; Service Design Network, 2016; Siegel, 2003|
|E2||Efficiency||UCD increases staff efficiency and productivity (e.g., time spent on a task diminishes)||In use||Bias & Mayhew, 2005; Donahue, 2001; Hirsch et al., 2004; Rajanen, 2011; Siegel, 2003|
|E3||Efficiency||UCD increases end-users’ efficiency and productivity||In use||Bias & Mayhew, 2005; Mantei & Teorey, 1988; Rajanen, 2011|
|I1||Innovation||UCD reveals new opportunities and innovation in the organisation and the developer community||Indirect benefit, in the undefined future||Lockwood, 2007, 2009; Siegel, 2003|
|I2||Innovation||UCD reveals new opportunities and innovation among end-users||Service Design Network, 2016|
|IPR||Intellectual Property Rights||UCD produces valuable IPRs||Indirect benefit, in the undefined future||Lockwood, 2007, 2009|
|M||Marketing value||UCD increases the marketing and public relations (PR) value of the organisation or the service||Production process, in the undefined future||Best, 2010; Bias & Mayhew, 2005; Donahue, 2001; Lockwood, 2007, 2009|
|N||Normative obligations||UCD enables that the service/product meets normative obligations (lower risk of litigation)||Indirect benefit, in undefined future||Best, 2010; Bias & Mayhew, 2005; Donahue, 2001|
|Q1||Quality||UCD produces quality (e.g., good usability and user experience, accessibility) for the target group(s)||In use||Best, 2010; Bias & Mayhew, 2005; Cockton, 2006; Law et al., 2008; Lockwood, 2007, 2009; Rajanen, 2011|
|Q2||Quality||UCD produces quality of the product/service to the broad group of end-users and citizens||In use||Best, 2010; Bias & Mayhew, 2005; Design Council, 2015; Lockwood, 2007, 2009|
|Q3||Quality||UCD increases employees’ job quality and satisfaction||During implementation and maintenance||Bias & Mayhew, 2005; Lockwood, 2009; Rajanen, 2011|
|R||Revenue/ productivity||UCD increases economic profitability, revenue and the competitive edge of the organisation or nation||In the long-run (future)||Bias & Mayhew, 2005; Design Council, 2015; Donahue, 2001; Hirsch et al., 2004; Lockwood, 2009; Rajanen, 2011; Whicher et al., 2011|
|SP||Social problem-solving||UCD enables identifying and solving social problems in the society||During development, in the future||Lockwood, 2009; Service Design Network, 2016|
|T||Trust||UCD increases trust in the organisation producing the service, and in the society||Indirect benefit, in the undefined future||Bias & Mayhew, 2005; Service Design Network, 2016|
|U||Use||UCD increases the attractiveness of the product/service, which generates more use (among existing and new end-users)||In use||Bias & Mayhew, 2005; Donahue, 2001; Hirsch et al., 2004; Lockwood, 2007; Mantei & Teorey, 1988; Rajanen, 2011|
4.3. Predictive Evaluation
The third and final key element of our conceptual 360°TB Model is Phases, which concretises the theoretical concept of Temporality. This element builds better understanding of the temporal connection between the design efforts and their outcomes. In other words, we argue that if the desired benefit can be located in a particular phase of the life-cycle, it can thus be more precisely evaluated and managed.
Another aspect of temporality found in the literature concerns the relation between past, present and future. Design activity is considered to be dominantly future-orienting, and it may disrupt present stabilities (Krippendorff, 2006). The outcomes of design activity can increase value and benefits soon or long after the outcome has been achieved or even during the implementation process. Table 2 displays the temporal aspects of each benefit category that we interpreted from the literature. Some benefits are considered to become apparent during a particular phase in the life-cycle of the service, while others can accrue variously along the process.
Applicability in the context of service provision requires splitting continuous time into useful phases. We identified from the literature three more or less overlapping periods of time, when different benefits of user-centred design may be perceived: during the process of developing or producing the service, during the use of the service, or in the undefined future (cf. Table 2). Encouraged by these notions in the literature, we divide the element Phases into three distinct parameters that point from the present to the future: process-time, use-time and (undefined) future.
To better comprehend the parameters in the context of digital public service development, we map them to the life-cycle of service development. Alter (2008, 2013) has captured the different conditions of service systems into the Work System Life Cycle (WSLC) model. The model represents the dynamics of work systems in four phases from the practitioners’ viewpoint: 1) initiation, 2) development, 3) implementation, and 4) operation and maintenance. User-centred design can take place in each phase of the life-cycle. The WSLC model acknowledges the iterative and repetitive nature of development processes and is to be understood as a realisation of cyclic and continuous activity.
The first two phases of the WSLC model (development and implementation) correspond to the parameter process-time of the 360°TB Model. The third WSLC phase, operation and maintenance, is parallel to the second parameter of our model, use-time. It is noticeable that use-time can occur during the WSLC phase initiation or as long the old service is still in use. The third parameter of our model, future does not have a counterpart in the WSCL model, because the latter model ends with the extinction of the service. As the theory and literature indicate, design of public services can derive benefits long after the service has ceased to exist (e.g., Service Design Network, 2016). Thus, we consider it essential to include the aspect of undefined future in the conceptual model. Figure 6 illustrates the comparison of the temporal elements of the WSCL model and the 360°TB Model on a single timeline.
Figure 7 illustrates the three key elements of our 360 Degree Temporal Benefits Model. We can best describe the relations between the elements through a narrative sentence: A Stakeholder anticipates Benefits of the design in different Phases. The evaluation process is predictive since it takes measures from the present moment in order to derive benefits for the future in three phases: process-time, use-time and (undefined) future.
4.4. Using and Applying the Model
The 360 Degree Temporal Benefits Model serves as an empirical conceptualisation of the integrated theoretical framework that we depicted in Figure 4. The purpose of the model is to capture the key elements and their relations in the public digital service provision landscape. Following the conceptual model and using the available metrics and data, design managers in libraries can have better control of their UCD efforts towards the desired goals. The model itself does not define which metrics and data sources are appropriate or adequate but recommends using material that is reasonable in the given context.
The predictive nature of the model allows using it already at the beginning of a design project, but it can also be applied later in the service life-cycle, even repeatedly. The utility of the model lies in its ability to detect similar or potentially conflicting goals of different actors and stakeholders of a service. In comparison to existing evaluation frameworks, our model suggests a 360-degree view to relevant stakeholders of a service. This approach should aid design managers in better guaranteeing the legitimation of and support for their design decisions. On the other hand, stakeholders can require that design activities are evaluated following the 360°TB Model to avoid an organisation-centric assessment. Using the model can equalise the power-balance in the strategic debate on the values and desired goals of services’ design.
Because the 360°TB Model defines the phases, when UCD benefits are anticipated to become effective, managers can use the collected assessment data for more accurate prediction and evaluation. The model can be used for UCD performance evaluation by indicating the status of current design activities (“Are we doing the right things?”) or UCD performance management by setting the future goals (“What should we do in the future?”).
The first step in using the model is the identification of relevant stakeholders for the service design case. After that, the person conducting the analysis should contact representatives of each stakeholder group and ask their opinion of UCD benefits. The list of UCD benefits derived from the literature in Table 2 can serve as a starting point, but the analyser can complement it with case-specific benefits expressed by stakeholders. Further, s/he should consider the relationship between benefits and their temporal aggregation (phasing). Depending on the case, some benefits may be relevant only in the process-time, some in the use-time, some in the (undefined) future and some at all times. Finally, we recommend that the results be opened to public discussion and stakeholders are allowed to form their joint understanding of UCD benefits. Figure 8 shows the recommended steps of using the model in practice.
To improve the rigour of the results, the benefits that stakeholders consider essential can be further concretised with appropriate metrics. The multitude of data that organisations already collect about their performance and services can be used for this purpose. Eventually, an evaluation conducted following the 360°TB Model may reveal defects in the current measurement practices.
Evaluation activities tend to consume time and effort, less in a simple case, more when the settings are complex, and the scale of the service is big. Furthermore, the frameworks of evaluation can be complex or intensive, e.g., by including many phases or procedures (cf. International Organization for Standardization, 2014; Tanner, 2012). The lack of standard definitions and parameters for the design discipline can also make the evaluation process overly time-consuming (Whicher et al., 2011). Some authors suggest that only big investments should be involved in an extensive analysis (Hirsch et al., 2004).
For these reasons, we suggest that practitioners should follow the principle of simplicity when they use the 360°TB Model in real cases. The following rules can help keep the application of the model within the constraints of time and resources:
- Limit the number of stakeholder groups to the minimum viable sample.
- Involve a minimum number of representatives per group.
- Let stakeholder representatives express only their first priority benefits for each phase (process-time, use-time and future).
Still, depending on the number of stakeholder groups, the evaluation conducted using the model can yield a multitude of benefits for the UCD of a service. If each stakeholder anticipates three different benefits (one for each phase), and these benefit-triplets vary between stakeholders, the outcome of the evaluation is a diverse set of expectations for the design activity. Figure 9 demonstrates how evaluation using the 360°TB Models may provide a multitude of benefits that stakeholders anticipate individually or share with each other.
5. Testing the Model
5.1. Two Digital Library Cases
As a tentative example of potential uses of the conceptual 360°TB Model, we applied the model to two existing cases. Both cases represent a digital library service development in which user-centred approaches were applied.
The first case is the National Digital Library (NDL), which joins together the repositories of practically all libraries, archives and museums in the country. The NDL includes a joint online user interface for all participating organisations’ materials, as well as a platform service for organisations to build their interfaces. By the end of 2016, all academic libraries and several public libraries in the country had joined the service with their collections and electronic materials. From the very beginning of the project, an in-house team of interaction designers has been responsible for the user-centred design of the service. Besides, various other stakeholders have taken part in the design activities (Hormia-Poutanen, Kautonen, & Lassila, 2013).
The second case differs from the previous one in its scale and scope. The project began as a typical library system acquisition, which included an online user interface for the end-users of a medium sized special library, serving visually impaired people in the country. Eventually, the library split the procurement and made a contract with two partners. They outsourced the interface design to a small enterprise that had expertise in the required user interface technology. During the development process, end-users have been involved in usability tests that were conducted by another firm of required expertise. The new service has been in use from the beginning of 2017.
These two projects provided the data for testing our conceptual model. The data sources were existing documentation and semi-structured interviews of key informants representing relevant stakeholder groups for the projects. Before interviewing informants, we needed some background information about the current status of each project, and particularly the goals set for user-centred design.
As the first step (see Figure 8), we browsed project documentation to identify relevant stakeholder groups for both projects. To capture stakeholders’ perception of benefits that should be received through user-centred design in the case project, we decided to interview the stakeholders. To enable more structured data gathering, we asked the informants to fill an online questionnaire during the interview (step 3, see Figure 8). We also induced informants to suggest suitable metrics for the benefits they chose as most important. For questionnaire options, we used the benefit categories (see Table 2) and temporal phases (process-time, use-time and future, see Figure 7), both identified from the existing literature (step 2, see Figure 8).
We conducted and recorded seventeen interviews altogether. Each informant filled the online questionnaire during her/his interview. Ultimately, the interviews comprised of over ten hours of recording, which we transcribed and combined with informants’ comments derived from the questionnaire. When we had conducted all interviews and compiled data, we sent each informant’s data to her/him for review and validation. Once all informants had reviewed their answers, we published the results to them (step 4, see Figure 8).
5.2. Results from the Case 1
In Case 1 we interviewed eight informants, who represent a wide spectrum of stakeholders. Three of these informants represent internal groups, who are responsible for the digital library service in different roles. Two informants represent external partners; one is a representative of a customer organisation and two of end-users of the NDL.
From all 21 available benefits, Case 1 informants chose sixteen different options altogether. Two informants considered that the given options were not entirely satisfactory and described their benefits that were combinations of the proposed benefits. Figure 10 displays informants’ choices in a pie chart on the left-hand side and in a tabular format on the right-hand side.
The results from Case 1 show a notable variation of benefits anticipated by the stakeholders. All informants picked a different first priority benefit for the process time. For the use-time, they expressed three shared benefits: quality for a more extensive group of audience or citizens (Q1), potential for innovations among end-users (I2) and increased use (U). Similarly, innovation among organisations and developers (I1) and trust to the organisation or society (T) were shared by two distinct informants as number one benefits for the (undefined) future.
The heterogeneity of aims increases the challenges of design performance, particularly for the near future. The use-time benefits/targets seem ambitious and partly conflicting: UCD of the digital library should generate more use, provide a good user experience for broad audiences as well as for specific target groups, and improve these end-users’ efficiency and enable their innovativeness. The benefits expected in the distant future are also ambitious: UCD is considered as a means for improving competence and commitment of involved persons and groups, as well as a means for enabling innovation, “common good” and enhancing trust in the public services and society.
“[Trust] …it is the social capital of trust, which means trust in institutions and public authority, and information, and social reciprocity. This is related to the innovation and the renewal of civilisation, too.” (Representative of the Ministry)
In the interest of space, we only present here the metrics/indicators, which informants in Case 1 considered applicable for measuring innovations (benefits I1 and I2). We chose this example because innovation is one of the strategic objectives for European public services, including libraries (European Commission, 2010; LIBER, 2017). Particularly open innovation, which involves various stakeholders in the design and development of services, is considered as an opportunity for the public sector organisations to improve their productivity and performance (Daglio, Gerson, & Kitchen, 2014).
Altogether, four informants chose either end-user innovation (I3) or developer innovation (I1) as their preferred benefit for proximate or distant future. The informants’ ideas for measuring achievements in innovation included quantitative and qualitative data of the service usage and users’ habits, as well as field-studies that involve end-users. One informant addressed the importance of detecting new users and new forms of using the service. Also, one informant pondered the option of measuring developer communities’ innovativeness through some gamified activities. Currently, only usage logs and usability studies generate data for this purpose. Other data sources for detecting innovation in the processes or use of the DL have not been established.
5.3. Results from the Case 2
In Case 2 we interviewed nine informants. Five informants represented internal stakeholders, two external partners, and two end-users. One external partner was also the lead designer of the digital library user interface.
The same 21 benefit options were available for the Case 2 informants, who chose 14 of them (see Figure 11). Three informants considered increased competence and learning among involved parties (C/L) as the most important benefit to the process-time. Altogether, five informants preferred the quality for a broad group of end-users and citizens (Q2) in use-time. For the future benefits, two informants prioritised societal problem-solving (SP) and three citizens’ trust to the organisation or society (T). Thus, Case 2 informants showed more similar disposition to UCD design benefits than Case 1 informants.
Because of the relative consensus among informants in Case 2, we saw here more uniformity in their first priority benefits. Further, all use-time benefits/targets focus on the end-users: UCD of the digital library should increase the use (U), improve the quality for a broad audience (Q2) or the target group (Q1) or end-users’ efficiency (E3). While the informants expressed that the process-time benefits refer to the challenges of developing the digital library in the proximate past, the future benefits/targets reflect universal values, such as social problem-solving and trust in the society.
“Use of one digital service increases one’s competence to use other digital services and one’s qualifications as a citizen […] Successful user-oriented design in one service, where the target users are somehow vulnerable people […] increases their trust in the organisation, society, and increases their trust in themselves.” (Representative of collaboration-user)
Again, we only present one example of the potential metrics/indicators expressed by the informants. In this case, we picked the most appreciated benefit, which was the quality of the service to a broad audience or citizens (Q2). The informants listed currently existing methods of gathering evidence: usage statistics, user feedback via surveys and user studies. They emphasised the importance of finding indicators for user satisfaction and user experience. Also, the informants mentioned the number of users and the extent of the user base as relevant indicators of quality.
Contemporary library services earn their legitimation first and foremost from the users of library services, but also from other stakeholders. Studies show that libraries are extending and deepening their collaboration and partnerships with other departments within their organisation (Corrall, 2014) as well as with external partners (Rowley, 2011). The organisation-centric viewpoint, also called the Institutional Paradigm (Batt, 2015), seems to dominate the theoretical and practical frameworks of operations management in libraries (cf. International Organization for Standardization, 2014). It is said to be an ethical choice, whose values or interests are reflected in the goals set to design (Holmlid, 2010). Moreover, overly organisation-centric management practices may not cohere well with the UCD approach and designerly thinking (cf. Botero et al., 2012).
For these reasons, we suggest that libraries should make the strategic choice and systematically engage all relevant stakeholders, also beyond the borders of the organisation, in the evaluation of their performance. This is an extension to the existing conceptualisations and frameworks for library performance evaluation, particularly the ISO standard (International Organization for Standardization, 2014), which already encourages gathering information from the field. Similar approaches have been tested in other application domains (e.g., the Triple Task Method by Bell and Morse, 2010).
We introduced a 360-degree model of engaging people, which is a familiar method in human resources management (e.g., Fleenor, Taylor, & Chappelow, 2008). The tentative application of the model in two cases showed us how different stakeholders are willing to assess the benefits of UCD. In both cases, the outcome was a multitude of diverse opinions, which may seem a cumbersome rather than an alluring result. Still, the informants explicitly expressed their appreciation for giving them authority in the matters of design goals. The tests further imply that the model can help people involved in a digital library design to develop their mutual understanding if they open the results to discussion.
The literature offers various values and benefits that can be achieved through design. We categorised these benefits (see Table 2) and used them in testing the model. According to the informants, it is not easy to specify the goals and benefits of the design activity. The major challenge is that the impact of design on the outcome is difficult to quantify and is often indirect (Best, 2010; Harpum, 2004). Furthermore, a design’s contribution cannot easily be isolated from the broader context or distinguished from other factors (Rosenberg, 2004). The library performance standard makes similar notices of other performance factors (International Organization for Standardization, 2014). Still, given the increasing productivity demands – particularly in the public sector – the libraries have to be able to justify also their managerial choices.
We can pursue noble goals, such as trust in the society, with the design of digital library service, but design alone cannot lead to it. Still, the informants emphasised the importance of setting high aims for design. Particularly in the public sector, design can play a role in removing and solving complex societal issues (Daglio et al., 2014; Design Council, 2015; Service Design Network, 2016). In the process of change, managers can better frame or make sense of the change through the concepts of temporalities (Wiebe, 2011). Our research, including the conceptual model and its testing, demonstrates that the temporal aggregation of UCD benefits helps to configure the relationship between the current design activity and the desired future outcome or the public value.
In the 360°TB Model, we strived to capture temporality in the design performance evaluation. During some interviews, we became aware that it is difficult for people to connect desired benefits with given phases. One explanation is that in the modern digital service development process, implementation and use co-occur (e.g., Alter, 2013). By default, the service will be used in the future, too, but also development activities continue after the service has been launched. This is apparently an area where more conceptual elaboration and empirical data are needed.
When we self-evaluate our research, we can note that it has improved our understanding of the elements of UCD performance management. Moreover, the stakeholders engaged in the tentative evaluations appreciated the approach and the outcomes. Although we did not rigorously validate the conceptual 360°TB Model in extensive field tests, the results from the tentative application indicate that the model, indeed, captures some essential elements of the digital library service production landscape. As an abstract conceptualisation, the model cannot yet serve as a mature tool for practitioners. However, our wish is to continue the development of the model into a practical and easy-to-use instrument.
We strictly targeted the application domain of the model to the UCD activities in the public sector. Furthermore, we focused our study on overly digital library services. We think that the core of our new conceptual model could be applied to other public-sector services, such as museums or e-government. Moreover, we can see no obstacles to transferring the idea of 360-degree temporally aggregated benefit evaluation to the user-centred design of physical services as well.
Most of us share the idea that, in a democratically ordered society, competing narratives and value disputes should be negotiated in a fair and just manner towards the common understanding (Held & Schott, 2006). An organisation-centric evaluation is undoubtedly the most feasible method of assessing performance. However, there is a growing demand for public sector organisations to view their service production from more intra-organisational and polycentric perspectives (Osborne et al., 2012). Our conceptual 360-Degree Temporal Benefits Model (360°TB Model) is an example of acknowledging diverse public service stakeholders’ viewpoints.
The main contribution of this research is the new 360°TB Model, which draws a picture of the most desired benefits that various stakeholders anticipate of the design activity of library service. The model allows a temporal distinction of benefits in three phases that reach from proximate processes and use of the service to the distant future.
Our model serves as an empirical elaboration of the Public Value approach to the performance management of public services as it bridges the theoretical concepts of the Public Value framework (Moore, 2014) to real-world situations through its key concepts: Stakeholders and Benefits. Also, our model incorporates the dynamic influence of time with the Public Value approach through the conceptual element Phases. Furthermore, the 360°TB Model extends the scope of existing evaluation frameworks by transferring the evaluation activity outside the organisation’s borders.
As part of this study, we conducted a tentative exploration of the model on two test cases. Our findings implied that the set of benefits/goals for a single case could be ample indeed. Still, we argue that the temporal aggregation helps in framing the change from the reachable and measurable design benefits towards more ambitious and ambiguous public values. We recommend using the model as an instrument that helps in setting the expectations for the design (or redesign) of library service. Ultimately, this research suggests that user-centred design (UCD) may have a more significant role in building the future library than just meeting the needs of the current user base.