Determinants for Failure and Success of Innovation Projects: The Road to Sustainable Educational Innovation

The Road to Sustainable Educational Innovation

Determinants for Failure and Success of Innovation Projects: The Road to Sustainable Educational Innovation

Educational Technology Expertise Center Open University of the Netherlands Commissioned by Stichting SURF

Determinants for Success and Failure of Innovation Projects

This research was financed by the SURF Foundation, ICT and Education.

ISBN 90–358-2294-3 Copyright © 2005 Open Universiteit Nederland, Heerlen Niets uit deze uitgave mag worden verveelvoudigd, opgeslagen in een geautomatiseerd gegevensbestand of openbaar gemaakt worden in enige vorm of op enige wijze, hetzij elektronisch, mechanisch of door fotokopieën, opname, of op enige andere manier, zonder voorafgaande schriftelijke toestemming van de auteur. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form, or by any means, electronic, mechanical, photocopying, recording or otherwise, without the prior permission in writing, from the author.

2


Table of contents Managementsamenvatting ...........................................................................................I Executive summary ................................................................................................ VII Chapter 1: Literature Study ........................................................................................ 1 Abstract................................................................................................................... 3 Introduction............................................................................................................. 3 Literature review..................................................................................................... 6 Project phasing.................................................................................................... 6 Corporate / business point of view ..................................................................... 8 Educational point of view................................................................................. 13 Initiation phase and implementation phase....................................................... 13 Institutionalisation phase or sustainability….................................................... 20 Synthesis ............................................................................................................... 22 Lessons learned – Eye openers ............................................................................. 26 Chapter 2: Expert Concept Mapping Study.............................................................. 29 Abstract................................................................................................................. 31 Introduction........................................................................................................... 31 Group Concept Mapping Procedure ..................................................................... 33 Method.............................................................................................................. 36 Participants ....................................................................................................... 36 Procedure .......................................................................................................... 36 Results............................................................................................................... 39 Conclusions and Discussion ................................................................................. 47 Chapter 3: Interview and Project Documentation Study .......................................... 53 Abstract................................................................................................................. 55 Introduction........................................................................................................... 55 Describing and clustering factors ..................................................................... 55 Project classification......................................................................................... 56 Interview study ..................................................................................................... 57 Detailed results from the interviews:................................................................ 59 The clusters individually................................................................................... 62 Comparisons of highly successful projects versus less successful projects ..... 65 Project documentation study................................................................................. 68 To conclude .......................................................................................................... 70 Eye openers........................................................................................................... 71 Chapter 4: Recommendations and Checklists .......................................................... 73 SURF itself ........................................................................................................... 75 Projects ................................................................................................................. 75 Tender procedure .................................................................................................. 78 Criteria for the final proposal ........................................................................... 79 Project start ....................................................................................................... 80 Project process ...................................................................................................... 81 Prepare for institutionalisation.............................................................................. 81 Dissemination ....................................................................................................... 82 Checklist ............................................................................................................... 84


Chapter 5: Dissemination ......................................................................................... 93 Publications........................................................................................................... 95 Conferences .......................................................................................................... 95 Workshops ............................................................................................................ 96 Articles about the project...................................................................................... 96 References................................................................................................................. 97 Appendix I - SURF projects (Tender year 1999 up to 2001) ................................. 101 Appendix II - CATWOE ........................................................................................ 103 Example of a CATWOE (Brainstorm version – in Dutch)................................. 105 Appendix III - Factor ratings .................................................................................. 109 Appendix IV - Clusters........................................................................................... 117 Appendix V - Interview protocol............................................................................ 125 Appendix VI - Interviews ....................................................................................... 128 Appendix VII - Projecttoets.................................................................................... 147 Appendix VIII – Original checklists....................................................................... 149 Initiation phase ................................................................................................... 149 Implementation phase......................................................................................... 153 Institutionalisation phase .................................................................................... 154

2


Managementsamenvatting Via het Platform ICT en Onderwijs zet SURF zich in voor een beter en creatiever gebruik van ICT binnen het hoger onderwijs. Het gaat daarbij zowel om het primaire proces als ook om het vernieuwen en integreren van alle ondersteunende processen binnen de onderwijsorganisatie. In het kader daarvan stimuleert het Platform ICT en Onderwijs – via cofinanciering – educatieve innovatieprojecten bij instellingen van hoger onderwijs in Nederland. Om beter te kunnen zorgen dat deze innovatieprojecten op een nog effectievere manier kunnen worden geïnitieerd, geïmplementeerd en geïnstitutionaliseerd heeft het Onderwijstechnologisch Expertisecentrum van de Open Universiteit Nederland van SURF de opdracht gekregen om een dieptestudie uit te voeren naar succesen faalfactoren van innovatieprojecten in het hoger onderwijs, in het bijzonder van innovatieprojecten gefinancierd door SURF Educatie. In lijn met de recente constatering van de Wetenschappelijk Technische Raad van SURF (2003) dat de structurele inbedding van de projectresultaten in de instelling als geheel meer aandacht verdient, wordt in deze studie bijzondere aandacht besteed aan het bepalen van de succesen faalfactoren voor het realiseren van duurzame innovaties in het onderwijs. Dit rapport betreft het verslag van de dieptestudie naar succesen faalfactoren van innovatieve ICT projecten in het onderwijs. In de eerste fase van het project werd op basis van literatuuronderzoek bepaald wat de aard en invloed is van verschillende factoren met betrekking tot de initiatie, implementatie en institutionalisatie van onderwijsinnovatieprojecten op het succes van de onderwijsinnovatie (Hoofdstuk 1). Een indeling van de geïdentificeerde succesen faalfactoren in clusters diende als basis voor de twee volgende fasen. In de tweede fase werd door een zogenaamde ‘group concept mapping’ procedure vastgesteld welke factoren experts op het terrein van innovatie in (onderwijs)organisaties en projectmanagement van belang achten voor het succes van grootschalige innovatieve ICTprojecten in het onderwijs (Hoofdstuk 2). In derde fase werd middels diepte-interviews met de projectleiders van meer en minder succesvolle SURF Educatie-projecten van de tenders 1999, 2000 en 2001 bepaald hoe zij denken over en rekening gehouden hebben met de geïdentificeerde succesen faalfactoren (Hoofdstuk 3). Tevens werd de documentatie van de betreffende projecten geanalyseerd om vast te stellen of de succesvolle projecten anders zijn omgegaan met succesen faalfactoren dan de minder succesvolle projecten (Hoofdstuk 3). Naast de validering van eerder gevonden succesen faalfactoren, stond iedere fase ook in het teken van het identificeren van nieuwe factoren. Op basis van een vergelijking van de resultaten uit de verschillende studies worden checklists gepresenteerd en aanbevelingen gedaan die door SURF en de projectaanvragers/projectleiders in de verschillende fasen van een project gebruikt kunnen worden om de kans op succes, vooral wat betreft de duurzaamheid van de innovaties, te vergroten (Hoofdstuk 4). De disseminatie van de projectresultaten speelde in iedere fase van dit project een belangrijke rol (Hoofdstuk 5). Literatuurstudie Het eerste onderdeel van de dieptestudie, de literatuurstudie, presenteert een 3-fasen model (I3 model – Initiatie, Implementatie, Institutionalisatie). Dit model dient als leidraad voor het bepalen van beleid, het formuleren van nieuwe richtlijnen voor projectvoorstellen, het beoordelen van projectvoorstellen die zijn ingediend bij het SURF Platform ICT en Onderwijs, het bepalen van de ‘gezondheid’ van een project ter

I


identificatie van waarschuwingssignalen voor falen en het bevorderen van de verspreiding en duurzaamheid van de projectinnovaties. Het model presenteert 42 factoren die van invloed zijn op het succes van een project. Deze factoren zijn gecategoriseerd naar projectfase. De literatuurstudie heeft de volgende lessen of ‘eyeopeners’ opgeleverd: 1. Een succesvol project is niet hetzelfde als een succesvolle innovatie Er zijn twee typen succes, namelijk succes van het innovatieproject zelf en succes van de innovatie (de uiteindelijke implementatie). Deze twee typen van succes zijn compleet verschillend en geen van beide is een voorwaarde of garantie voor de ander. Bovendien betekent succes van de innovatie niet automatisch succes op instituutsniveau, maar vereist het zorgvuldige overwegingen met betrekking tot schaalbaarheid, generaliseerbaarheid, temporele flexibiliteit en financiële duurzaamheid om tot een succes op instituut- of organisatieniveau te komen. 2. Een organisatiemanager is geen projectmanager Gerelateerd aan de vorige eye-opener, bestaat er ook een groot verschil tussen een competente projectmanager en een competente organisatiemanager. Het doel van organisatiemanagement is het plannen, leiden en helpen de organisatie goed te laten functioneren en voortbestaan; met andere woorden het bereiken van continuïteit. Het doel van projectmanagement echter is het actief plannen van het einde van een project; met andere woorden het doorbreken van de continuïteit en het bereiken van verandering. Een innovatieproject moet eindigen om succesvol te zijn, terwijl een organisatie moet voortbestaan om succesvol te zijn. Verder, in tegenstelling tot organisatiemanagers moeten projectmanagers een project vanuit het niets plannen en opstarten en een projectorganisatie initiëren en tot leven brengen. Iets starten om het binnen een vastgestelde periode binnen een bepaald budget en een aantal voorwaarden te beëindigen is wat de aard van projectmanagers onderscheidt van de aard van organisatiemanagers. Het negeren van dit verschil kan leiden tot incompetent projectmanagement zowel in industriële organisaties als in de onderwijsorganisatie. Volgens een onderzoek van de Standish Group1 was de leiding van 97% van door hen succesvol onderscheiden projecten in handen vaneen ervaren projectmanager. 3. Duurzaamheid krijgt vaak pas achteraf aandacht, terwijl het een geplande strategie zou moeten zijn om de bereikte innovatie of verandering duurzaam te implementeren in de organisatie . Manieren om de duurzaamheid van een innovatie te bevorderen zijn: • De aanvragers van het innovatieproject een gedetailleerd plan voor de duurzame implementatie van de innovatie laten presenteren. • De aanvragers uitdagen om dit plan voor duurzame implementatie te realiseren na afloop van het project, dat wil zeggen nadat de subsidieverstrekking is beëindigd. • De aanvragers bewust maken en rekening laten houden met de faal en succesfactoren die van invloed zijn op de duurzame implementatie innovaties.

1

De Standish Group verzamelt en analyseert data over IT omgevingen en hun gebruik. http://www.standishgroup.com

II


• Het bekend maken van projectmanagers en teamleden met methodologiëen die gebruikt kunnen worden in de institutionalisatiefase om duurzame innovaties te bereiken. • Gebruik maken van financiële beloningen om het projectteam en de instelling uit te dagen om de innovatie te handhaven en te verspreiden naar andere instellingen. • Het plan voor een duurzame implementatie van de innovatie als voorwaarde stellen voor een gedeelte van de subsidie. 4. Voorkom de ´zes zekere ´ faalfactoren • Gebrek aan evenwicht tussen investeringen en resultaten: hoge investeringen met weinig resultaat. • Informatie politiek: macht wordt misbruikt en informatie wordt niet overgedragen. • Gebrek aan verantwoordelijkheid: onzekerheid over de verantwoordelijkheid van mensen binnen en buiten het project. • Cultuurkloof: de kloof tussen ICTers en de rest van de organisatie en tussen degenen die het onderwijs plannen (managers) en degenen die onderwijs geven (docenten). • Te grote betrokkenheid: niet weten wanneer verlies te nemen en een project te stoppen. • Alles-in-één oplossingen: proberen alles ineens te doen in plaats van verschillende projecten, stappen en fasen te gebruiken. 5. Omarm de vier kritische ‘win drivers’: • Definieer en stel de projectdoelen in een vroeg stadium vast om de betrokken partijen erbij te betrekken en een algemeen begrip en betrokkenheid te realiseren. • Bepaal de rollen en verantwoordelijkheden van alle betrokkenen (klanten, deelnemers, betrokken partijen, eigenaren). • Innovatieve projecten brengen veranderingen met zich mee en adequaat verandermanagement is noodzakelijk. • Zorg voor een systematische en grondige monitoring van controle en prestatie. 6. Kleine veranderingen hebben vaak meer impact dan gedacht wordt Hoe meer veranderingen, hoe meer factoren de kansen op succes beïnvloeden met als gevolg een toename van de risico´s van het project. Met andere woorden, een toename in interacties, leidt tot een toename in factoren, dit leidt tot een vermindering van controle, wat uiteindelijk kan leiden tot een afname van de kans op succes. 7. Innovatie is als een nucleaire reactie: Klein is fijn De ´Mass Diffusion´ theorie beweert dat voor de meeste leden van een sociaal systeem, bijvoorbeeld een onderwijsinstelling, de innovatiebeslissing in belangrijke mate afhangt van de innovatiebeslissingen van andere leden van het systeem. Als ongeveer 10-25% van het systeem de innovatie accepteert, zullen de overige leden van het systeem relatief snel volgen. Een groter percentage ‘initiële accepteerders’ kan nadelig werken voor het succes van het innovatieproject. Het streven naar directe volledige acceptatie en implementatie is nadelig.

III


8. Opinieleiders – Ken je vijanden en maak hen bondgenoten: De ´Mass Diffusion´ theorie stelt eveneens dat bepaalde groepsleden (met name opinieleiders die een groot vertrouwen van de groepsleden genieten) de innovatiebeslissing van andere groepsleden direct beïnvloeden. Opinieleiders hoeven niet noodzakelijkerwijs de hiërarchische leiders te zijn. Het op je hand krijgen van een erkende criticus kan belangrijker zijn dan het op je hand krijgen van een leidinggevende. Group concept mapping Ter validering van de factoren die in de eerdere literatuurstudie zijn gevonden en identificatie van nieuwe factoren, namen experts op het gebied van (onderwijs)innovatie en innovatiemanagers deel aan twee onderzoekssessies waarin gebruik werd gemaakt van een zogenaamde ´Group Concept Mapping´ procedure. De experts genereerden 220 potentiële faal- en succesfactoren. Deze factoren werden vervolgens geclusterd met behulp van een zogenaamde ´affinity´ techniek en geanalyseerd. Dit resulteerde in 13 clusters van factoren. Ten slotte werd de experts gevraagd om aan alle factoren een score toe te kennen betreffende hun belang voor onderwijskundige ICT innovatieprojecten (1 = onbelangrijk; 5 = belangrijk). Dit resulteerde in een hiërarchie van potentiële faal- en succesfactoren. De 13 clusters met hun gemiddelde score voor belang zijn: Cluster Participatie, Informatie en Communicatie Toegevoegde waarde Betrokkenheid van stakeholders Kwaliteit van institutioneel management Project- en procesmanagement Focus en complexiteit Competentie van het projectteam en haar leden Relatie tussen project en werkelijke probleem / core business Competentie van de projectmanager Middelen om het project te steunen Organisatiecultuur Positie van de innovatie binnen de organisatie Projectoptimalisatie: Begin klein en ga er dan voor!

Score 3.75 3.73 3.68 3.65 3.61 3.56 3.53 3.51 3.44 3.42 3.37 3.29 3.25

De 10 meest belangrijke (hoogste score door experts) succesen faalfactoren zijn: 1 Maak de toegevoegde waarde zichtbaar 2 Maak het voordeel van een product duidelijk (nieuw is niet altijd beter) 3 Kies een competente projectleider 4 Vier je successen 5 Management moet betrokken en competent zijn 6 De projectleider moet volledig toegewijd zijn 7 Ambitie telt! 8 Vorm een expert team en een professioneel team 9 Betrek alle stakeholders (docenten, studenten, administrateurs, directeuren) 10 Houd de cultuur open De topprioriteiten voor innovatieprojecten liggen dus op de gebieden Participatie, informatie en communicatie en Stakeholder betrokkenheid aan de ene kant en

IV


Toegevoegde waarde aan de andere kant. De twee eerstgenoemde zijn zogenaamde ´mensen´-clusters met als directe praktische implicatie het vereiste om gebruikers (cliënten) vanaf het begin te betrekken bij de analyse, ontwerp, ontwikkeling, evaluatie en implementatie van de innovatieve uitkomsten. Dit staat bekend als ´participatief ontwerp´. Het cluster Toegevoegde waarde kan waarschijnlijk worden beschouwd als het meest belangrijke cluster met factoren zoals Maak de toegevoegde waarde zichtbaar en Maak het voordeel van een project duidelijk. Een conclusie die getrokken kan worden op basis van deze resultaten is dat om falen te voorkomen er een duidelijke reden ten grondslag moet liggen aan de onderwijskundige innovatie die onderwijskundige verandering rechtvaardigt. Interviews/Scores Om te bepalen welke factoren een rol hebben gespeeld in de door SURF Educatief gesubsidieerde innovatieprojecten, werden acht afgeronde SURF Educatief projecten geselecteerd uit de tenders van 1999, 2000 en 2001. Vervolgens scoorden experts van SURF Educatief de mate van succes van deze acht projecten resulterend in de selectie van vier ‘meer succesvolle’ en vier ‘minder succesvolle’ projecten. De projectleiders van de geselecteerde projecten werden vervolgens geïnterviewd om te bepalen hoe ze zijn omgegaan met de geïdentificeerde succesen faalfactoren en ze werden gevraagd om een score toe te kennen aan de mate van belang van deze factoren, de mate waarin er rekening is gehouden met deze factoren in het project en de mate waarin de factoren van invloed waren op het succes van het project. De factoren die aan de projectleiders werden voorgelegd waren de 42 factoren uit de literatuurstudie onderverdeeld in betekenisvolle clusters. De interview/score resultaten lieten zien dat: • Met een aantal van de zeer belangrijke clusters niet echt rekening werd gehouden door de projectleiders: Projectstaf kwaliteiten, Organisationeel management, Toegevoegde waarde/gebruik/noodzaak, Verwachtingen van betrokken partijen en Schaal/complexiteit. • Hoewel de projectleiders aangaven er geen rekening mee te houden, werden twee van deze clusters (Projectstaf kwaliteiten en Schaal/complexiteit) als zeer invloedrijk voor het succes van het project beschouwd. Interessant was of de projectleiders van ‘meer succesvolle’ en ‘minder succesvolle’ projecten de factoren verschillend scoorden of implementeerden. Met andere woorden: Zijn er factoren die discrimineren tussen meer en minder succesvolle projecten? De resultaten van die analyse laten zien dat: • Vier clusters (Afbakening van de projectinhoud, Communicatie, Toegevoegde waarde en Schaal/complexiteit) altijd een hogere score kregen van de projectleiders van meer succesvolle projecten voor alle drie de maten (belang, rekening mee gehouden en invloed). • Met vijf clusters (Communicatie, Toewijding/betrokkenheid, Verwachtingen van betrokken partijen, Projectcultuur en Schaal/complexiteit) significant meer rekening werd gehouden door de projectleiders van meer succesvolle projecten. • Met drie clusters (Toewijding/betrokkenheid, Verwachtingen van betrokken partijen en Schaal/complexiteit), hoewel als uiterst belangrijk beschouwt, desondanks significant minder rekening mee werd gehouden door de projectleiders van de minder succesvolle projecten. Deze drie aspecten van innovatie en innovatieprojecten kunnen beschouwd worden als onderscheidende aspecten (discriminerende clusters van factoren).

V


De geïnterviewde projectleiders waren in het algemeen positief tot zeer positief over het aantal (succesvolle) procedures van SURF, namelijk de goede ondersteuning en hulp door SURF met betrekking tot proces- en inhoudelijke problemen, SURF en SURF Educatief management’s flexibiliteit met betrekking tot het oplossen van problemen en de zeer duidelijke betrokkenheid van SURF en SURF Educatief management bij de projecten en de projectstaf. Echter, de projectleiders noemden ook een aantal verbeterpunten voor SURF, namelijk met betrekking tot de intake aan het begin van het project, de review van de inhoud door experts en de behoefte aan een meer compacte stijl van rapporteren. Projectdocumentatie Ten slotte werd een gedetailleerde studie van de documentatie van de acht geselecteerde projecten uitgevoerd. De door de projectleiders aangeleverde projectdocumentatie werd geanalyseerd op basis van de vragen of en in welke mate ze expliciet aandacht hebben besteed aan de 14 clusters die gebruikt werden in de interviews en de scoringsvragenlijsten. Verder werd de documentatie gescand op de ´lessons learned´ en de succesen faalfactoren genoemd door de projectleiders en of de problemen genoemd tijdens de interviews ook gevonden konden worden in de documentatie. Ook werden de reviews en reviewprocedures geanalyseerd op verschillen tussen procedures die zouden kunnen aangeven of een project zich als meer of minder succesvol zou ontwikkelen. Hoewel de door de projectleiders aangeleverde project documenten compleet en de reviews grondig waren, bleken ze niet voldoende te zijn om succes of falen te signaleren! Verder kunnen er geen verschillen of determinanten voor falen of succes worden gevonden in de procedures, reviews, ondersteuning, of hulp door SURF. Met andere woorden, de documenten en hun evaluaties discrimineerden niet tussen meer en minder succesvolle projecten. Ten slotte, de lessons learned sectie van het projectdocument werd zeer gewaardeerd door de projectleiders. De lessons learned bestonden onder andere uit: • een toegewijd (financieel human resource capacity, tijd en middelen) management is essentieel, • management moet niet alleen toegewijd zijn, maar ook in staat zijn om de benodigde middelen te leveren, • management moet een verandermanagement zijn met een competente ´change agent´. Dit is consistent met de bevinding uit de literatuurstudie dat er een wereld van verschil is tussen projectmanagement en organisatiemanagement en dat daarmee rekening gehouden moet worden bij het bemensen van het project. • stabiliteit en continuïteit van de staf is belangrijk; het is vooral belangrijk om de projectleiders niet halverwege het project te vervangen. Afsluitende opmerking Hoofdstuk 4 bevat een uitgebreide samenvatting van de resultaten evenals van de aanbevelingen voor veranderingen in de tendering procedures, documentatie, en subsidierichtlijnen voor toekomstige SURF innovatieprojecten.

VI


Executive summary The Platform ICT and Education of SURF stimulates a better and more creative use of ICT within higher education. These efforts are not only focused on the primary process, but also on the modernisation and integration of all supporting processes within the educational organisation. To that end, the Platform ICT and Education stimulates – by co-financing – educational innovation projects at institutes for higher education in the Netherlands. To find out how these innovation projects can be initiated, implemented and institutionalised more effectively, the Educational Technology Expertise Center of the Open University in the Netherlands was asked by SURF to conduct an in-depth study investigation into success factors of innovation projects in higher education, in particular, of the innovation projects that are financed by SURF Education. In line with the recent observation by the Scientific Technical Council (WTR) of SURF that structural embedding of the project results in the institute deserves more attention, this study pays special attention to the determination of success factors that are important for the realisation of sustainable innovations in education. This report presents the results of the in-depth study into success factors of innovative ICT projects in education. In the first phase of the project literature research was used to determine the nature and influence of different factors with respect to the initiation, implementation, and institutionalisation of educational innovation projects on the success of the educational innovation (Chapter 1). A classification of the identified success and failure factors into clusters served as a basis for the two subsequent phases, which were aimed at validating the factors that were found in the literature study and distilling new success factors. In the second phase, a so-called ‘group concept mapping’ procedure was used to determine the factors that experts in the field of educational or organisational innovation and innovation management consider important for the success of largescale educational innovation projects (Chapter 2). In the third phase, interviews with the project leaders of more and less successful SURF Education projects (tenders 1999, 2000, and 2001) were held to find out how they think about and have taken into account the success factors that were found in the literature study (Chapter 3). In addition, an analysis of the documentation of these projects was conducted to find out how the more and less successful projects have dealt with these factors (Chapter 3). A comparison of the results across the different studies was used to design checklists and formulate recommendations for SURF and project applicants/leaders to increase the chances on success, particularly regarding the sustainability of the innovation (Chapter 4). Dissemination of the results of the different phases of this project has played an important role in each of the project phases (Chapter 5). Literature study The first element, the literature study, presents a three stage integrative model (I3 model – Initiation, Implementation, Institutionalisation) as guide for establishing policies and procedures, formulating new guidelines for project proposals, the assessment of SURF Education innovation project proposals, conducting a project’s health check to identify warning signs for failure, and promoting the diffusion and sustainability of the projects’ innovations. The model presents 42 factors that are imperative to accomplishing a successful project, categorized by project stage. The literature study- discussed in depth

VII


in Chapter 1 – delivered a number of lessons that were and should be learned / eyeopeners, namely: 1. Project success is not innovation success There are two types of ‘success’, namely the success of the innovation project itself and the success of the innovation for education (its ultimate implementation). These two ‘successes’ are completely different and neither is a requirement or a guarantee for the other. Also, success of an innovation at the project level does not automatically generalize to success at the institutional level, but needs careful considerations of scalability, generalisability, temporal flexibility, and financial sustainability. 2. An organisational manager is not a project manager Related to the prior eye-opener, there is also a major difference between being a competent project manager and a competent organisational manager. The goal of organisational management is to plan, guide and help the organisation work well and survive; in other words achieve continuity. The goal of project management, on the other hand, is to actively plan the end of a project; in other words to break with continuity and achieve change. A project must end in order to be successful whereas an organisation must survive in order to be successful. Also, in contrast to organisational managers, project managers have to start their task from scratch, initiating and bringing an organisation to life. To start something in order to end it within a fixed period of time, according to a budget and a number of restrictions distinguishes the nature of project managers from the nature of organisational managers. Neglecting this difference can lead to incompetent project management, both in the industrial organisations and in the educational organisation. According to the Standish Group2, 97% of the successful projects investigated in their research had an experienced project manager at the helm. 3. Sustainability should be a planned strategy for maintaining change and not some kind of afterthought. Ways to promote the sustainability of an innovation are: • Requiring applicants to present a detailed plan for sustaining the innovation. • Challenging applicants to realize this plan after the project has ended; that is once the funding period of the innovation grant is over. • Making the applicants/project managers aware of the success factors imperative to sustainable innovations. • Familiarizing project managers and team members with methodologies that can be used in the institutionalisation phase to attain sustainable innovations. • Using (financial) incentives to challenge the project team and the institution to maintain the innovation and diffuse it to other institutes. • Making part of the funding conditional on the realization of the plan for sustaining the innovation. 4. Avoid the six sure-fire failure causers • Lack of balance between investments and output: high investment with low output. 2

The Standish Group, formed in 1985, collects and analyses case information on real-life IT environments. http://www.standishgroup.com

VIII


• • • • •

Information politics: power is abused and information is not transmitted. Lack of responsibility: uncertainty about the responsibility of people inside and outside the project. Culture gap: the gap between ICTers and the rest of the organization and between those planning education (managers) and those administering education (teachers). Over-commitment: not knowing when to cut your losses and stop a project. All-in-one solutions: trying to do everything at once instead of using multiple projects, steps, and phases.

5. Embrace the four critical ‘win drivers’: • Define and anchor the project objectives early to involve the stakeholders and anchor a common understanding and commitment. • Determine the roles and responsibilities of all actors (clients, participants, stakeholders, owners). • Innovative projects involve changes and adequate change management is required. • Systematically and thoroughly monitor and measure control and performance. 6. Small changes often are larger than we think The greater the number of changes, the more factors that influence the chances of success with a resulting increase in the risks of the project. In other words, an increase in interaction leads to a decrease in control, which leads to a decrease of the chance of success. 7. Innovation is like a nuclear reaction: Small is beautiful Mass Diffusion theory argues that for most members of a social system, the innovation-decision depends heavily on the innovation-decisions of other members of the system. When about 10-25% of the system members adopts the innovation, relatively fast adoption by remaining members will follow. Initial full-scale implementation is detrimental. 8. Opinion leaders - Know your enemies and make them allies: Mass Diffusion theory also argues that certain group members (opinion-leaders in which the other group members have great trust) directly affect the innovationdecision of other members. Opinion-leaders need not necessarily be the hierarchical leaders. Getting an acknowledged sceptic on board could be more important than getting a boss on board. Group concept mapping To validate the factors found in earlier literature research as well as to further expand them experts in the field of (educational) innovation and innovation management took part in two research sessions making use of a Group Concept Mapping Procedure. Experts generated and recorded 220 potential failure and success factors. These factors were then clustered by them using an affinity technique and analysed resulting in 13 factor clusters. Finally, the experts were asked to rate each of these factors on their importance for educational ICT innovation projects, thereby creating a hierarchy of potential failure and success factors. The 13 clusters with their mean importance rating are:

IX


Cluster Participation, Information and Communication Added value Stakeholder involvement Quality of institutional management Project and process management Focus and complexity Competency of the project team and its members Relationship between project and real problem / core business Competency of the project manager Means to support the project Organisational culture Position of innovation within the organisation Project optimisation: Start small and then go for it!

Rating 3.75 3.73 3.68 3.65 3.61 3.56 3.53 3.51 3.44 3.42 3.37 3.29 3.25

The 10 most important (highest rating by experts) success and failure factors are: 1 Make the added-value visible 2 Make the benefit of a product clear (new is not always better) 3 Choose a competent project chair 4 Celebrate your successes 5 Management must be involved and competent 6 The project chair must be completely dedicated 7 Ambition counts! 8 Form an expert and professional team 9 Involve all stakeholders (teachers, students, administrators, directors) 10 Keep the culture open The top priorities for innovation projects, thus, lie in the areas of Participation, information and communication, and Stakeholder involvement on the one hand and Added-value on the other. The former are people clusters with as direct practical implication the requirement for involving users (clients) right from the beginning of the project in the analysis, design, development, evaluation, and implementation of the innovative outcomes, practical arrangements known as participatory design. The latter can probably regarded as the most important single cluster with factors such as Make the added-value visible and Make the benefit of a product clear. A conclusion that can be drawn from these results is that to prevent failure there should be a clear reason underlying the educational innovation project that justifies educational change. Interviews/Ratings To determine which factors played a role in innovation projects funded by SURF Education interviews were held with the project chairs of eight completed SURF Education projects selected from the tenders of 1999, 2000, and 2001 and classified as either highly successful or less successful. It began with the identification of meaningful combinations of clusters based on the 42 success and failure factors found in the literature study. After this, experts from SURF Education rated the level of success of the projects resulting in the selection of eight projects (four ‘highly successful’ and four ‘less than successful’). The project chairs of the selected projects were then interviewed to determine how they dealt with the identified success and failure factors, and were asked to rate these factors with respect to their degree of importance, the degree to which they were taken into account during the project, and

X


the degree to which they were influential for success of the project. The interview/rating results reveal that: • A number of highly important clusters were not really taken into account by the project chairs: Project staff qualities, Organisational management, Added value/use/necessity, Expectations involved parties, and Scale/complexity. • Two of these clusters (Project staff qualities and Scale/complexity), though not taken into account, were considered highly influential for the success of the project. Interesting was whether ‘highly successful’ and ‘less than successful’ projects rated or implemented the factors differently. In other words: Are there factors that discriminate between highly successful and less than successful projects? The results from this analysis show that: • Four clusters (Delineation of project content, Communication, Added value, and Scale/complexity) were always rated higher by the project chairs of highly successful projects with respect to all three measures (importance, taken into account, and influence). • Five clusters (Communication, Commitment/engagement, Expectations of parties involved, Project culture, and Scale/complexity) were significantly more taken into account by the project chairs of highly successful projects. • Three clusters (Commitment/engagement, Expectations of parties involved, and Scale/complexity), though considered to be of paramount importance were nonetheless significantly less taken into account by the project chairs of less than successful projects. These three aspects of innovation and innovation projects might be considered watershed aspects (discriminating clusters of factors). The interviews with the project chairs were generally very positive with respect to a number of (successful) procedures of SURF, namely the good support and counselling by SURF regarding process and content problems, SURF and SURF Education management’s flexibility with respect to overcoming problems, and the very apparent commitment of SURF and SURF Education management to the projects and the project staff. Be that as it may, the chairs also listed a number of points for improvement for SURF, namely with respect to intake at the beginning of the project, the review of the content by experts, and a need for a more compact style of reporting Project documentation Finally, a detailed study of the documentation of the eight selected projects was carried out. The project documentation delivered by the project chairs was analysed based on the questions whether and to what extent they paid explicit attention to the 14 clusters used in the interviews and rating questionnaires. Furthermore the documentation was scanned for the lessons learned and success and failure factors mentioned by the project chairs and whether the problems mentioned during the interviews could also be found in this documentation. Also, the reviews and review procedures were analysed on differences between procedures that could indicate whether the project would turn out to be either highly successful or less successful. With respect to the analysis of project documentation delivered by the project chairs it is clear that though there is much documentation, there is also much overlap between the documents giving the impression of reporting for the sake of reporting. This is strengthened by the fact that although the documents were complete and the reviews were thorough, they were/are not sufficient to signal approaching success or failure! Also, no differences or determinants for failure or success can be found in the

XI


procedures, reviews, support, or counselling by SURF. In other words, the documents and their evaluations do not discriminate between highly successful and less successful projects. Finally, the lessons learned section of project document is highly valued by project chairs. Among the lesson’s learned were that: • a committed (financial, human resource capacity, time and resources) management is critical, • management not only needs to be committed, but also capable of delivering the needed resources, • management should be a change-management with a competent change agent. This echoes the finding in the literature study that there is a world of difference between project management and organisational management and that this must be taken into account when staffing a project. • stability of the staff is important; it is especially preferred not to change the project chairs halfway a project. Closing remark Chapter 4 contains an extensive summary of the results as well as recommendations for changes in the tendering procedures, documentation, and funding guidelines for future SURF innovation projects.

XII


Chapter 1: Literature Study

1


Abstract This Chapter presents the first phase of the project in which literature research is used to determine the nature and influence of different factors with respect to the initiation, implementation, and dissemination of educational innovation projects on the success of the educational innovation. After a general introduction, the different opinions regarding the phasing of innovation projects are discussed. Then, the results of the literature research into success and failure factors of innovation projects are presented. Because there is a longer tradition of innovation projects in business than in education, the literature study will focus on the business and educational literature. The synthesis presents an overview of the most important success factors for each of the project phases initiation, implementation, and institutionalisation. To increase the chance on success and sustainable innovations these factors can be used by SURF to formulate guidelines for the assessment of project proposals, and the evaluation of projects. In addition, applicants can use these factors as a tool to support the writing of proposals and by project managers to support the running of the project. Finally, some recommendations for additional actions to stimulate the sustainability of innovations are presented.

Introduction As is the case in most Western countries, the Dutch Government has invested heavily in stimulating better and more creative use of information and communication technologies (ICT) in all forms of education, including higher education. The ultimate goal of this investment is to ensure that students and teachers are equipped with the skills and knowledge required for success in the new knowledge-based economy. All stakeholders (i.e., government, industry, educational institutions, society in general) have placed high priority on achieving this goal. However, these highly funded projects have often resulted in either short-lived or local successes or outright failures (see, De Bie, 2003; Teasly, 1996). As a result, the role of ICT is developing less quickly in higher education institutions than was previously expected. In order to steer these developments in the right direction, SURF3, a governmentfunded national organisation in which all higher education institutes in the Netherlands participate to increase the pace of educational innovation, has recently (1999) set up the SURF Educatie (SURF Education) program for educational innovation projects. The goal of this program is the systematic stimulation of the application of ICT to innovation in higher education. SURF’s educational innovation projects are intended to be a source of inspiration for the introduction of ICT-based innovation in education and are aimed at better and sustainable results. SURF’s educational innovation projects may cover one or more of the following key issues: • Competence/Portfolio • Collaborative learning • Interactive teaching materials • Learning content management systems (LCMS)/Communities • New media An overview of all Surf Education projects (1999-2001) can be found in Appendix I 3

SURF is the Dutch higher education and research partnership organisation for network services and information and communications technology (ICT), which was founded in the mid 1980s. The SURF activities are funded both by the participating institutes al well as by the Dutch government. The participating institutes pay the costs for the operation of the SURF office. In addition to this, SURF receives grants from the Ministry of Education, Culture and Science and from the Ministry of Economic Affairs. These grants may only be used for strategic innovations.

3


Three important questions arise with respect to educational innovations. First, why are some innovations more successful than others are? Second, why do some innovations fail, while others succeed? A third important question that relates to the recent flow of funding of educational innovations is how can the innovations be sustained once the grant funding has ended? The present project, which was funded by SURF Education, tried to answer these questions by identifying the determinants for success of largescale educational innovation projects in Dutch higher education, in particular of those funded by SURF Education. The identification of success factors for educational innovation projects and measuring performance of projects based upon these factors are considered important to SURF Education because they can aid the development and implementation of innovation projects by explicating and making visible those success and failure factors relating to educational innovation projects in higher education. This is necessary to make these factors manageable and to enable future projects to achieve better and more sustainable results. The research project consists of the following phases: 1. A literature study to determine the benchmarks for success of educational innovation projects. 2. A group concept mapping study with experts in the field of innovation and innovation management to validate the success factors found in the literature study and to expand them (Chapter 2). 3. Interviews with the project managers of the SURF Education projects to determine the subjective dimension of success (Chapter 3). 4. Analyses of the project documentation of the most and least successful SURF Education projects that started in 1999, 2000, and 2001 (and are now completed) on the basis of the benchmarks from the literature study and the identification of new benchmarks for success from these projects (Chapter 3). 5. Recommendations for SURF, SURF-tenders (procedures and documentation), and SURF-projects plus checklists for determining the ‘health’ of Surf-sponsored innovation projects (Chapter 4). 6. Dissemination of the results by writing articles for professional and scientific journals and the presentation of the results at national and international conferences and workshops (Chapter 5). This chapter presents the results of the first phase of the research project; the literature review of success factors of educational innovation projects to identify the existing knowledge in this area. A first step in coming to meaningful conclusions is finding an answer to the questions of how ‘success’ can be operationally defined and how the dimension ‘success of educational innovation’ can be tapped. A problem here is that there are two quite different types of ‘success’, namely the success of the project itself and the success of the innovation (its ultimate implementation). These two ‘successes’ are completely different. Neither type is a requirement or a guarantee for the other. In general, success can be considered as the accomplishment of goals and objectives necessary to achieve a particular task. SURF Education projects are considered successful based on the extent to which they stimulated and facilitated new and better use of ICT in education. SURF is particularly interested in the sustainability of the technological innovations, in other words, how they can ensure that the innovations, both technical and pedagogical, make the shift from an externally funded initiative to a

4


sustained ‘standard operating procedure’. The rationale for sustaining successful ICTbased educational innovations is to preserve what has been valued and built for continued use. Inherent to the concept of success factors is a notion that if success factors are implemented in the project, the educational innovation or project will perform better and be successful. Success factors can thus be looked upon as individual independent variables influencing the dependent variable ‘success’. Analogously for failure factors, there is a notion that if failure factors are avoided in the project or implementation, that the innovation will have a better chance of being achieved. Performance measurements for success or failure can be classified as objective and subjective. The objective measures are based on measurement of past performance or output while subjective measures let individuals with an intimate knowledge of educational innovation be the judges of what is successful. It should be recognized that success of an innovation at the project level does not automatically generalize to success at the institutional or national level, but needs careful considerations of scalability, generalisability, temporal flexibility, and financial sustainability. To be able to determine which of the identified success factors for educational innovation projects are most relevant in different educational contexts, a contextual framework to position educational innovation and related success factors needs to be developed. The basic idea is that if we know why some projects fail or have only short-lived successes and we can avoid making these mistakes, it is possible to make sure that new projects will succeed with sustainable results. However, it should be noted that ‘non-failure’ is not considered the same as success. Success/failure should not be considered as a binary classification, but should be viewed along a continuum. Although, SURF sets clear goals for applicants to address sustainability in their project plan, and generally, clear implementation plans are available, clear plans for sustaining the innovation are lacking. Sustainability seems to be more an afterthought rather than a planned strategy for maintaining change. Kenny and Meadowcroft (1999) suggest that forward thinking and visioning are paramount in successfully planning sustainable developments. The observations presented in this literature study are meant to contribute to the understanding of why and how technological innovations in education are adopted and diffused. Together with the outcomes of the other phases of this project, the report is aimed at supporting SURF to steer the innovation projects towards success and sustainable technological innovations and to create mechanisms that empower all stakeholders to sustain innovative developments. For this literature study, we looked for books, accessed ERIC®, PsycLIT®, and used Google® to search the World Wide Web using the following search terms: success, success factors, failure factors, success determinants, innovative projects, innovation, return-on-investment, project, innovative factor, educational innovation, critical success factor, criteria, guidelines, project management, sustainability, success conditions, innovation processes, PT3, standards, Europe, strategy. We located over eighty articles specifically addressing the topic of interest. Reading these and following up crossreferences we established a knowledge base of over thirty books, articles, websites, and papers that addressed different aspects of why educational innovations sometimes fail and sometimes succeed. Also, it has resulted in the following structure for the rest of the review.

5


This chapter has the following structure. Since innovation can be considered a design and implementation exercise composed of a number of distinct phases, the different phases that can be distinguished in innovative projects are first described. Second, the business or corporate view on the success of innovative projects is described. In business there is a long tradition of using (technological) innovation as a major instrument to compete, survive, and grow. In comparison, the field of education has only recently started to worry about competition and surviving. Third, the experiences from educational projects are considered to determine the success and failure factors of innovation projects. Fourth, the factors that determine the sustainability of innovations are discussed. Finally, the different experiences are synthesized in an integrative model that can be used by SURF to assess project proposals to SURF Education regarding their potential to accomplish sustained innovations, to recognize project warning signs, and to increase the chance of success of the diffusion and permanence of the innovations.

Literature review Project phasing Innovating and changing an organisation is becoming more and more complex because organisations and the relationships between organisations are becoming more complex. Factors influencing this complexity are technology, environmental influences, size and structure of the organisation (e.g., tendency towards fusions), interdependence between organisations (e.g., tendency towards IT production), the willingness to change (e.g. human inertia), lack of support from the management, time and money and so on. Innovating (or changing) the structure of an organisation often crashes into a veritable wall of resistance. Not surprisingly, the human factor is often considered the most influential factor on the chance of success. Innovating an organisation or a structure places a heavy burden on the organisation and the employees and therefore it is necessary to have insight in the complex matter of the phases of organisational changes, the way people deal with innovation processes, the methods and strategies for change. Katz and Kahn (1978) argue that planning to initiate an innovation can be done in at least three phases: a diagnostic phase, a goal-setting phase, and an innovative-process phase. In the diagnostic phase, aspects like how the organisation is structured and organised need to be addressed. The initiator (person, group, or organisation) of the project is of great influence in this first phase on the rest of the innovation process. Where the project starts is called the point of entry of the project. Although innovative processes are in real terms a top-down process, the point of entry can also be elsewhere, for example in a government department, or in an organisation like SURF. In this phase, the problem(s) are defined and a first project proposal is written. In the goalsetting phase the goals need to be set and the design of the desired situation needs to be specified. The final phase, the innovative process phase, consists of determining which strategies, roles, methods and interventions, based on the theoretical considerations, can best be used to realise the desired situation. Kor and Wijnen (2001) specify this concept of phasing a bit further and according to them a project can be divided into six phases: initiation, definition, design, preparation, realisation, and maintenance (see Figure 1). In the initiation phase, characterised by searching and negotiation, the material (cause, history, etc) is collected and the desired outcomes or results are defined. In the definition phase, an inventory is

6


made of the demands of the content of the results. In the design phase a solution to the demands is designed and specified. The preparation phase ensures that the design is ready to be developed and implemented. Next, the design is carried out, developed, and implemented in the realisation phase. Finally in the maintenance phase the project results are used and looked after.

Initiation phase

Definition phase

Design phase

Preparation phase

Realisation phase

Maintenance phase

Figure 1. Six phases of Kor and Wijnen (2001)

In another study, Moret Ernst & Young (De Koning & Florijn, 1995), argue that a project can be divided into five phases, namely assessment, re-engineering, design, construction and implementation phase (see Figure 2). These are basically the same phases as described by Kor and Wijnen (2001), except for the preparation phase and the maintenance phase (see Figure 2). The assessment phase is more or less the same as the initiation phase in which an inventory of the goals, structures and condition of the present organisation takes place. The re-engineering phase sharpens the ideas made in the assessment phase and develops a vision for the future. The design phase of both studies is relatively similar. In the construction phase the design is actually build and in the implementation phase the construction is implemented. A major difference might be that Moret Ernst & Young use as a given the fact that there is already an organisation, process or whatever that needs to be re-engineered, thus “starting from scratch” doesn’t exist.

Assessment phase

Re-engineering phase

Design phase

Construction phase

Implementation phase

Figure 2. Five phases of innovation, Moret Ernst and Young (De Koning & Florijn, 1995)

Another more condensed model, in which the maintenance of the innovation is explicitly mentioned, is presented by Alexander and McKenzie (1998) and Fullan (1991). In their view on the phasing of organisational innovation and change, they distinguish between three main types of phases: an initiation phase in which planning and evaluation takes place, an implementation phase in which the development, implementation and evaluation of the project takes place, and finally a maintenance or institutionalisation phase regarding the sustainability of the project (see Figure 3).

7


Initiation phase


Institutuionalisation phase

Figure 3. Condensed model of phasing

Corporate / business point of view Only 28% of the IT projects investigated were successful according to the results of the Standish Group4 2000 study (Johnson, Boucher, Connors, & Robinson, 2001). Although this percentage is better than the percentage of a comparable study of the Standish Group in 1994 (16%), the number of un-successful projects is considerable. Since not all un-successful projects can be rated as failed the Standish Group 2000 study categorises projects into three types: 1. Successful (28%): the project is completed on time and on budget, with all features and functions originally specified; 2. Challenged (49%): the project is completed and operational, but over budget, late, and with fewer features and functions than initially specified; 3. Failed (23%): the project is cancelled before completion, or never implemented. The 2000 study shows that, despite huge technical improvements, experiments and innovations in the last decades, the failure of the IT projects is mostly due not to technical failures but to the way the technology is implemented and used by humans. As previously mentioned, the human factor is often considered as the most influential factor on the chances of failure or success (e.g., Johnson, et al., 2001; Kor & Wijnen, 2001; Schein, 1995; Storm & Jansen, 2004; Turner, 1999). In describing three types of technical improvements and developments in a) information technology, b) in dealing with information technology by organisations, and c) in the relation between developers and the users of information technology, Holmes (2001) reached a number of conclusions regarding the failure of projects. In his view: • The failure of ICT projects cannot be completely due to a deficit in project management. • The culture gap between business-people and IT professionals has deep historical roots and will not be easily washed away. • Organisations have problems with ICT projects approached from a wrong point of view: the technological point of view. • The ICT developer is badly integrated in the organisation. • Many projects do not have a clear division of responsibility of the project. • Projects depend many times on an all-in-one solution. • Investments in ICT are often high, but the outcome is often low and uncertain. Based on these conclusions, Holmes developed a “failure model”, in which he describes six factors that are of influence on the failure or success of projects (see Figure 4). 4

The Standish Group, formed in 1985, collects and analyses case information on real-life IT environments. http://www.standishgroup.com

8


FAILURE MODEL High investments, low output: no balance between the investments and output

Culture gap: gap between ICT and the rest of the organisation

Information politics: abuse of power and information system

Overcommitment: danger of overcommitment for failing projects

Lack of responsibility: uncertainty of responsibility of persons

All-in-one-solutions: instead of using a plan with multiple steps leading to the solution.

Figure 4. Failure model of Holmes (2001)

Moonen (2003) elaborates on the subject of high investment and low output and suggests that corporate institutions make use of return-on-investment (ROI) as a major criterion for decision-making to ensure that investments will be worth the return. In order to determine the ROI of an activity or project, one has to calculate the benefits of the activity and compare them with the costs (by dividing or subtracting) via the formula: ROI (%) = (Net Program Benefits / Program Costs) X 100 With Net Program Benefits = Total Benefits – Costs. Through making this calculation more explicit, thinking about the cost-benefits relation is provoked, which can lead to more valid decisions in project management. As return on investment and expected wins are important criteria when deciding to start a new project, Karlsen, Svendson and Omli (2003) focus on four critical ‘win drivers’ in their research: • Defining and anchoring the project objectives: It is important to define the project’s objectives and goals early in order to involve the stakeholders and anchor a common understanding and commitment • Roles and responsibilities for achieving wins: Important roles like the role of the client, users and project manager need to be well defined and established • Necessary change process: Innovative projects involve changes and adequate change management is often neglected • Follow-up and measure wins: Failure of projects can sometimes be related to lack of control and performance management, this has to be measured systematically and thoroughly.

9


ICT (hardware, software, infrastructure) is continuously changing and at the same time organisations are becoming more and more dependent on it. It is as though a steadystate is an illusion. This “situation” is compounded by the fact that most ICT projects tend to have a great impact on the organisation and the organisation’s need to change. Not only is the technology changing as a result of this, but the internal processes, tasks and structures of an organisation have to change as well. And as the project produces more changes, the influence of the environment on the project will increase too. In order to successfully manage such a project, these reciprocal and/or networked influences need to be dealt with. The more influences of the environment on the project, the greater the chances of failure. So, the challenge is to identify a number of influences of the environment that creates an acceptable level of failure. O’Hara, Watson, and Kavan (1999) investigated the relation between technology and organisational change. In their study, an organisation is seen as a collection of interrelated parts that work together in two main systems: a social system and a technical system. These two systems are dependent on each other and influence each other. Within these two systems, four strongly interrelated variables play an important role: people, tasks, structure, and technology. People carry out tasks, which produce the services and outcomes of the organisation. The structure is the result of communication and organisation. The technology refers to every problem solving intervention. The technical system includes the technology and the tasks to be fulfilled in order to accomplish the goals of the organisation. The social system contains the people and the roles. The attributes of the people (attitudes, skills and values) and the communication, power and the structure of the organisation form the social system. Their study shows that there can be three levels of change: first-order, second-order and third-order change (see Figure 5). The magnitude of change differs on every level. A higher level of change also contains the lower level(s), for example, a second-order change also contains all the changes of first-order change. With first-order change only the interaction between task and technology is influenced. In other words only the tasks people have to carry out are influenced by the introduction of ICT. For example, a computer now does the tasks formerly carried out by people. The task at hand already existed, but with the introduction of ICT, the task will be made and carried out easier and more efficiently. This form of change is the least complex and the effects of the change are relatively easy to deal with and control5. Second-order change influences, besides the interaction between task and technology, involve the interaction between people and tasks and the interaction between people and technology, for example, the role people play in a task changes because the procedures to fulfil that task changes, or the way people deal with technology changes because the technology changes. Finally, third-order change causes a change in all four variables of the organisation (technology, tasks, people and structure). The interaction between structure and people, structure and tasks, and structure and technology is changed besides the changes already made in the second- and first-order change. These three new changes in interaction can imply changes in for example power structures within an organisation, changes in the way the organisation fulfils its tasks or changes in the way new technologies will be selected and implemented. 5

This is of course a simplification. When a machine takes over the work of a person, old jobs are lost and new jobs (e.g., maintenance of the machine) are created and thus a subtle change in the structure will ensue.

10


Figure 5. Three levels of change (O’Hara, Watson, & Kavan, 1999)

To sum up briefly, the greater the number of changes is, the more factors there are which influence the chances of success and therefore there is an increase in the risks of the project. In other words, an increase in interaction leads to a decrease in control, which leads to a decrease of the chance of success. Investigating the magnitude of the change beforehand, and taking the result of this investigation into account when implementing the project, can diminish the chances of failure (O’Hara et al., 1999). A thought experiment might help the reader here. A change from traditional instruction to teambased problem based learning (often using some type of electronic technology) has often been the subject of innovation projects. This seems, at face value, to be a simple change. The instructor only needs to think up a few problems – often from the real-world – throw them on the web, and divide her/his students into teams. But what does this small change really include? The instructor first has to master a PBL-pedagogy, often quite different from what (s)he has done in the past. To ‘throw it on the web’ the instructor also has to have the necessary ICTskills – both teaching skills and technical skills). PBL, either individual or in a team requires a new type of assessment and testing which also necessitates a change in examination rules and regulations (NL: Onderwijs- en Examen Reglement). To work in teams, the infrastructure (physical) must be able to support this; in other words the institute has to have enough rooms to allow the students to work in teams. All of this needs a new form of administration (e.g., the institution needs to administer and record who has done what, who has played what role in which teams, and what the progress of the individuals and the teams are, both individual and team grades need to be recorded, et cetera). This list is no way near being complete, but hopefully demonstrates that ‘small changes’ are almost never really small.

11


PBL pedagogy

ICT (teaching) skills

ICT skills

Other assessment

Other testing

Exam rules & regulations Rooms for teams

Administration

et cetera

Along with acknowledging the possible reasons for the high number of failed projects, a rudimentary recipe for success can be composed based on numerous studies (e.g., Alexander & McKenzie, 1998; the Standish Group; Holmes, 2001, Johnson et al., 2001): 1. Executive support: influences a project’s process and progress. Lack of executive input can jeopardize a project 2. User involvement: even when delivered on time and on budget, a project can fail if it doesn’t meet user needs or expectations 3. Experienced project manager: approximately 97% of successful projects have an experienced project manager 4. Clear business objectives and shared goals 5. Minimised scope: time is a crucial factor for many projects, and since scope affects time or project duration, minimising scope increases a project’s chances of success 6. Standard software infrastructure: by using standard infrastructure, you can concentrate on the project rather than on technology 7. Firm basic requirements: creating minimal, obtainable base requirements and then developing those features will reduce the effect of change 8. Formal methodology: this provides for example a realistic picture of the project and resources committed to it 9. Reliable estimates: systematic project estimating must be approached realistically in the context of the time, budget and expertise available 10. Sufficient funding for the maintenance of the project must be available

12


Educational point of view In addition to the above mentioned studies, which were conducted from a corporate or business point of view, there has been a fair amount of research within educational settings into the determinants of success or failure of innovative projects. Initiation phase and implementation phase Renewal and modernisation of education is necessary to keep up with the changing demands on new employees and graduates. Educational organisations try to cope with the increasing demand for more knowledge, skills, and competencies by changing improving and innovating their education. One way to deal with these increasing demands is the implementation and use of technology. A two-year national study to evaluate the contribution of information and communication technology projects to student learning in higher education was conducted in Australia by Alexander and McKenzie (1998). The study reviewed over 100 projects, which received teaching developments grants and made significant use of a range of information and communication technologies to develop student learning materials. After a detailed literature review, a questionnaire was developed and sent to the project leaders of 173 projects across Australia and finally these questionnaires were analysed. A striking outcome of the questionnaire was the large discrepancy between the intended outcomes of the projects and the actual outcomes reported. While 87% of the projects’ leaders noted “improved quality of learning” to be an intended outcome of the project, only 30% reported this as an actual outcome. Although this discrepancy can be due to failure of the project and/or failure of the project to measure this, it seems justified to state that the majority of the projects have not been successful in achieving their intended outcome. Furthermore, the study has shown that technology itself does not make a difference improving learning outcomes or assures successful educational innovations. Diffusion is the process by which an innovation is communicated through certain channels over time among members of a social system. The diffusion of innovations theory (Rogers, 1995) helps to explain and analyse how an idea or proposal for an innovative project is communicated and accepted by others. Therefore, this theory affects the initiation phase and needs to be taken into account when initiating an innovative project. According to Rogers (1995, p. 6), “diffusion is a kind of social change, defined as the process by which alteration occurs in the structure and function of a social system.” Rogers also defined adoption as being a decision to make full use of an innovation and rejection as a decision not to use an innovation. Five main steps can be distinguished in this innovation-decision process: 1. Knowledge: the individual is first introduced to the innovations and starts to understand how it functions; 2. Persuasion: the individual starts to form an opinion (positive or negative) about the innovation 3. Decision: the individual makes a decision to either adopt or reject the innovation 4. Implementation: the individual starts using the innovation 5. Confirmation: the individual seeks reinforcement of the decision to use the innovation.

13


Rogers also identified five characteristics of innovations that play an important role in the innovation’s rate of adoption, namely: 1. The innovation’s relative advantage as compared with the current practices: the degree to which an innovation is perceived as better than the current practice; 2. Its compatibility with the adopters past experiences, present values and current needs: the degree to which an innovation is perceived as being consistent with the existing values and needs of the potential adopters; 3. Its simplicity / complexity, the perceived difficulty to use and understand the new technology: new ideas that are simpler to understand are adopted more rapidly; 4. The degree to which it may be experimented with on a limited basis: innovations that are testable represent less uncertainty to the potential adopters; 5. Its observability, the degree to which the results of an innovation are easily seen and understood. It can be concluded that innovations that are perceived by potential adopters as having greater relative advantage, compatibility, testability, observability, and less complexity will be adopted more easily than other innovations. The most striking feature of the diffusion theory is that, for most members of a social system, the innovation-decision depends heavily on the innovation-decisions of other members of the system. When about 10-25% of the system members adopt the innovation, relatively fast adoption by the remaining members will follow. In other words, initial full-scale implementation is not necessary for full-scale adoption. As a matter of fact too large implementation could even prove detrimental. Also people will only adopt the innovation if they believe that it will benefit themselves and if they trust and respect the initiator of the change. The diffusion theory further argues that, since leaders and certain group members (the so-called opinion-leaders in which the other group members have great trust) directly affect the innovation-decision of the other members, a powerful way to affect the diffusion of an innovation is to affect the leader’s and opinion-leader’s attitudes (see also Schein, 1995). These opinion-leaders need not necessarily be the hierarchical leaders. Getting an acknowledged sceptic on board could be more important than getting a boss on board. Rutherford (1992) describes a strategy to implement planned change. This strategy has two essential components: (a) the stakeholders who are affected by the change and (b) the decisive factors that effect the change. Focusing on the decisive factors that effect change or innovations, Rutherford (see also Mathias & Rutherford, 1983) comes up with a list of six decisive factors. This list or model is derived from two conceptualisations of innovation processes and is in fact a combination of the models of Lindquist (1978) and Berg and Ostergren (1977, 1979). The six decisive factors are linkage, openness, gain/loss, ownership, leadership, and power. Rutherford (p. 47) summarises these factors as follows: ‘[…] an innovation may be successful if: those likely to be involved can be brought together and presented with new ideas and information in order to develop ‘local’ solutions; this group is aware that a problem exists and is both actively searching for and responsive to new ideas and information; there are clear gains and few losses to be made; there is effective leadership; a strong sense of ownership develops; institutional support can be mobilised to protect and secure the innovation’.

14


Educational innovations, for example implementing a new method such as problembased learning, are sometimes the result of an intuitive and hasty decision to change. And without a proper investigation of, for example the problems the innovation is supposed to solve, the needs of the future users, the willingness of different participants to cooperate and the success and fail factors of this particular project, the success of the innovation is at stake. According to De Bie (2003), in order to be successful, projects need to take the following success factors into account when initiating an innovation: 1. Vision: a vision clarifies the goals of the project, the expected outcome. The way this vision is formed and formulated is important. 2. Strategy of development: cooperation between, for example, management and teachers is very important in an educational innovative project. They have to develop a strategy together and then work according to this strategy. 3. Acknowledgement of problems: besides clarifying the goals, the vision also clarifies the problems of the educational institute, and these problems should be the motive for the innovation. 4. Project plan: vision and perspective will lead to the necessary activities. The project plan describes the problem description, goals, methods, reviews, participants, output, sustainability, and evaluation. 5. Project management: the project management makes a clear division of roles and responsibilities and coaching of the team members for example is an important role for the management (e.g., Alexander & McKenzie, 1998; Johnson et al., 2001). 6. Resources: resources like time, material, computers, etcetera should be available 7. Role of “the outside world”: the outside world plays a role in innovative projects. The outside world comments, evaluates and contributes to the educational institute and therefore to the innovative project. 8. Support from the rest of the institute (Hannan, English, & Silver, 1999). 9. Competent management: the management is able to lead the innovative project (Hannan, English, & Silver, 1999). Apart from these process aspects, educational innovation has another important aspect, namely the content. Both process and content need to be taken into account in order for the project to be successful. The content aspect of the educational innovations not only concerns the actual content of the innovation as seen in the goals of the project plan, but, according to Vinkenburg (2003), also includes the knowledge, skills and attitudes of the people involved in the innovation –as far as this concerns new and to be innovated knowledge, skills and attitudes in order to let the innovation succeed. As shown in the previous section, the process aspects are highly dependent on the competence of the management, a human factor. Numerous projects fail due to incompetent leadership or management (e.g., Alexander & McKenzie, 1998; De Bie, 2003; De Koning & Florijn, 1995; Holmes, 2001; Johnson et al., 2001; Kor & Wijnen, 2001; Mathias & Rutherford, 1983; Rutherford, 1992; Storm & Jansen, 2004), lack of support from the head of department, dean or other person in authority (e.g., Alexander & McKenzie, 1998; Hannan, English, & Silver, 1999; Johnson et al., 2001; Light, 1998) or lack of support from the rest of the organisation and peers (e.g,. Hannan, English & Silver, 1999; Light, 1998; Zhao, Pugh, Sheldon, & Byers, 2002). We must stress that there is a major difference between being a competent project manager and a competent organisation manager. Crucial to an adequate understanding of the nature of project management versus organisational management is the fact that

15


the life-cycle of these two are very different. The main purpose of organisational management is to plan, guide and help the organisation to work well and survive; in other words to achieve continuity. Project management, on the other hand, is to actually actively plan the end of a project; in other words to break with continuity and achieve change. A project must end in order to be successful whereas an organisation must not end in order to be successful (Turner, 1999). It is important to note that the ending of a project does not mean that the innovation must end as well, the innovation must sustain after the project has ended. It must become a part of the organisation and thus facilitate/support continuity. The difference between an organisational and project manager goes even further and starts with the initiation of a project. In contrast to organisational managers, project managers have to start their task from scratch, initiating and bringing an organisation to life. To start something in order to end it within a fixed period of time, according to a budget and according to a number of restrictions distinguishes the nature of project managers from the nature of organisational managers (Turner, 1999). And this difference between an organisational manager and a project manager is often forgotten. Neglecting this difference can lead to incompetent project management, both in the industrial organisations and in the educational organisation. According to the Standish Group (Johnson et al, 2001), 97% of the successful projects investigated in their research had an experienced project manager at the helm. As mentioned above, the concept of Return-on-Investment (ROI) is a well-known concept in the corporate or business world and is used for decision-making within industrial and corporative activities. In educational settings, however, ROI is not so commonly known, but is becoming increasingly recognised as an important decisive factor in the initiation phase of an innovative educational project. A problem in the field of education is that the benefits are often not easy to detect and measure. Even when the benefits are made explicit in, for example, a list of outcomes, it is often difficult to transform them into a number or give them a monetary value. And comparable problems arise when trying to determine the investments (costs). A more usable form of ROI for educational settings is the Simplified Return-on-Investment where one simply deducts the costs from the benefits, which matches better with an intuitive feeling about the impact of an investment (i.e., costs) on the outcomes (i.e., benefits or results) than the normal ROI (Gustafson & Watkins, 1998; Karlsen, Svendsen, & Omli, 2003; Moonen, 2003). To reduce the complexity of the data gathering and difficult ROI calculation, the simplified ROI uses only the positive and negative items that are likely to be significantly different in the new situation compared to the old situation in the calculation. And the results of the calculation only give an indication whether the innovation will be effective (gain) or in-effective (loss). The procedure to calculate Simplified ROI consists of six main activities: finding consensus, organising, measuring, calculating, transforming, and combining. The first activity, finding consensus, leads to finding the relevant items that should be used in the simplified ROI calculation. An example of a simplified ROI calculation can be found in Table 1.

16


Table 1. Simplified ROI with respect to quality

Items Institution Weight Score Competency change: • Improved training for students • Improved market position for students • Improved potential for job rotation & learning to learn in organisation Behavioural items: • Improved attitude towards the training • Improved motivation towards training • Quality of Educational Service (QES): • Development, testing and evaluation of instructional programs • Development and implementation of new procedures • Improved communication possibilities • Improved potential for exchange of content between organisations • Improved potential for exchange of content between users ROI: quality

Actors Instructor Weight Score

Students Weight Score

.81 .6 .6

+1 +1 +3

1.0 .6 .8

+2 +1 +1

.8 1.0 1.0

+1 +22 +1

.8 1.0

+1 +1

1.0 1.0

+1 +1

1.0 1.0

+2 +2

1.0 .8 .8 .6 .6

-23 -1 +1 +1 +1 4.2

1.0 .8 1.0 .8 .8

-2 -1 +24 +2 +1 6.0

.3 .6 1.0 .3 1.0

-2 -1 +2 +1 +5 8.0

Note. Meant to be an illustration, only a limited number of the values in this table receive some comments. 1 For the institution the possibility Blackboard and e-learning offers to improve the training for the students is considered ‘very important’ and therefore gets a weight of .8. 2 Using Blackboard and e-learning during their studies improves the market position of the students. This cell indicates an estimate of this improvement from the student perspective. In comparison with the situation where no e-learning environment was used, the +2 in the table indicates an expected improvement of 20%. Asking current students but in particular previous students how they value this situation can optimise the reliability of this estimate. If there are research data available coping with this situation, such data should of course be used. 3 When using Blackboard and e-learning it can be expected that the development, testing and evaluation of instructional programs will take not only more time but also as a consequ3ence and certainly in the beginning of the change process, the quality of service will get worse than before. The reduction in QES from the perspective of the institution is estimated as being 20%. 4 A strong characteristic of an e-learning environment is of course its potential for communication between its users. Being on-line will certainly create many opportunities for the instructors to consult and communicate with each other. The estimate of the gain is put on 20%.

17


The second activity, organising, is used to bring order to the items. The measuring activity determines the values of the items (if possible) and in the calculating activity the simplified ROI calculation takes place for the separate items. The fifth activity, transforming, is optional and leads to the transformation of different values into the corresponding values if necessary. In the last activity, all the simplified ROI results will be put together in one combined simplified ROI result. The simplified ROI can lead to more explicit thinking about return-on-investment and will help in order to come to valid decisions Zhao, Pugh, Sheldon, and Byers (2002) investigated the complex process of classroom technology innovations. They were specifically interested in understanding the conditions under which technology innovations can take place in classrooms (think back to the earlier thought experiment). They found 11 main factors that significantly impact the degree of success of classroom technology innovations. Each factor can be placed in one of three interactive domains, the teacher, the innovation, and the context (see Figure 6). This study also determined the degree of success for each case by one of three ratings: successful, mixed success and failed.

Figure 6. Conditions for technology innovations (Zhao, Pugh, Sheldon, & Byers, 2002)

Three of the 11 factors can be ascribed to the domain of the teacher: technology proficiency, pedagogical compatibility and social awareness. Technology proficiency is the ability to operate a piece of equipment and the knowledge of the enabling conditions for a technology. In other words, teachers need to know how a technological innovation basically works and what the enabling conditions of that innovation are. Second, successful implementation of technological innovations is more likely when the teacher’s pedagogical beliefs are compatible with the technology. Finally, potential

18


problems are better discovered by teachers who are socially more sophisticated and they can usually better smooth the way for successful classroom technology innovation. The nature of the innovation itself, the second domain, turns out to be a second determinant of whether a project succeeds or not. Innovations can vary along two dimensions, distance and dependence, and success is related to these two dimensions. Distance relates to how much the innovation deviated from the status quo and it is important in three areas: distance from the existing school culture, from the existing practice and from the available technological resources. Dependence refers to the degree to which an innovation relies on other people or resources. Innovations low in distance and dependence are logically the simplest and easiest to implement since they cause less changes and are therefore more likely to be successful than innovative projects that are high in distance and dependence. The third domain, the context, has a strong mediating effect on the success of technological innovations. Three aspects of the context are of importance for the success or failure of an innovation: the human infrastructure, the technological infrastructure and the social support. The human infrastructure refers to the organisational arrangement to support technology integration in the classroom. The technological infrastructure refers to the amount and type of technological infrastructure needed for the innovation. Some innovations for example require frequent access to a computer, which needs to be assured. The last factor in the successful implementations of innovations is the degree to which peers support or discourage the innovators. These three domains contribute significantly to the success of technology integration and innovation, but the contribution of each domain is not equal. Factors associated with the innovator or for example the teacher –the human domain- appeared to play a more significant role than the other two domains. Under the authority of The Platform Vocational Training (in Dutch: Het Platform Beroepsonderwijs) teachers, project leaders, and students involved in nine innovative educational projects were interviewed (Weijers, 2003). The main purpose of these interviews was to gain insight in the factors that influence the successful implementation of educational innovations. Although the projects differ from each other in many ways, the core of the improvements and innovative goals can be indicated with the same set of concepts: competency based learning, problem based learning, life-long learning, cooperation between different organisations and improvement of the learning environment. Based on the interviews four types of success factors can be identified, all of which can be operationalised with respect to the ‘room’ (NL: ruimte) that those in power allow/give and the room that those carrying out the project or are otherwise involved take/demand. The success factors are: 1. Room to deviate: choose for a small-scaled scope at start in which innovations in school will be stimulated and from which it is possible to deviate. 2. Room to make mistakes, develop and learn: let the developments take place on the work floor in order to develop and learn at the same time. 3. Room to do what really matters, to use and develop your talent and qualities: make sure that all the people involved in the project participate and contribute on personal interest, qualities and talents so that they have an intern motivation for this project.

19


4. Room to work together, learn from each other and see each other as companions: focus on teamwork between the members of the project team, schools, or organisations. Do not think of the others as competitors but as companions. Furthermore, the Platform Vocational Training stresses the importance of cooperation and equal distribution of resources between different organisations when working on innovative educational projects. Institutionalisation phase or sustainability… As mentioned previously, an innovation process takes place in three stages: initiation phase, implementation phase and the institutionalisation phase. The determinants, or factors, of success or failure in the first two phases, the initiation phase and implementation phase have been described in the previous sections. Although the last phase is often forgotten or neglected (due to lack of money, time et cetera), it is a very important phase. In order to make the time, money, and effort worthwhile it is important to carefully plan this last phase. The implementation phase may continue for a period of time, but eventually there should be a point at which the new idea becomes institutionalised and regularised as a part of the ongoing operations. It is now no longer an innovation process, but rather a normal process. And whether or not the innovation becomes a durable part of the organisation depends on the commitment and action of the participants as well as on other factors. A number of causes for the lack of durability is given by Geerling, Mittendorff and Nieuwenhuis (2004): • Fuzzy use of language which causes miscommunications and misconceptions about the innovation; • Size, depth and the amount of work of the innovation is highly underestimated, this could decrease the chances of thorough embedding; • Missing of critical foundation: educational innovations are often presented as a concept of education or as a form of learning process but it often does not concern the renewal of quality management and maintenance and financing. Another aspect is human inertia: why people are sometimes so reluctant to change, to innovate. In fact, this is the point where a successful project can become a failed innovation. Schein (1995) found that human change, whether at an individual or group level, is a psychological process that involves painful unlearning an relearning while individuals attempted to restructure their thoughts, perceptions, feelings and attitudes. People need to un-freeze, change and re-freeze. Un-freezing refers to the removal of the restraining or balancing loops that are often associated with group norms and leads to cognitive dissonance or conflict that can be very disorienting to group members as they begin to change. When dealing with such a disorientation or disequilibrium, the group members need to change or reframe their thought process, rethink their ideas and representations of what is “normal”, and interpret new concepts more broadly than before. This is called re-freezing. The key to effective change (or innovation) is to carefully dose the amount of change, and therefore of the perceived threat, produced by the disconfirming information to allow the group members to feel safe to un-freeze, change and re-freeze their ideas and concepts. But just to receive disconfirming information is not enough to change. To become motivated to change, you must accept the information and connect it to something you care about. The information must be valid and relevant. Furthermore, as stated earlier by O’Hara, Watson, and Kavan (1999)

20


the more an innovation plans to change, the greater the influence from the environmental will become, the greater the risks will be and the chances of failure will increase. Light (1998) noticed that many innovations are deserted not at the initiation or implementation phase but in the institutionalisation phase. He further argued that four main factors (with each five sub-factors) influence the degree to which innovations are sustained through this institutionalisation phase: 1. External environment in which an organisation exists: innovations are more likely to be successful when the organisation is fitted in a stable environment which is supporting and collaborating. The success of the innovation can be facilitated of inhibited by: a. Turbulence in the environment: the level of uncertainty in the surroundings can inhibit the innovations chances; b. Level of shock in the system: an extreme event such as a budget crisis can endanger the chances of success; c. Support or encouragement for the innovation: the capacity of the organisation to meet a new idea with enthusiasm can facilitate the innovation; d. Collaboration: balancing collaboration between various subsystems can be crucial for the success; e. Availability of external support for the innovation: support such as external funding can facilitate the innovation. 2. Internal structure: innovations are more likely to be successful when the organisation has a relatively loose, centralized structure with good vertical communication channels. The chances of success of the innovation can be influenced by: a. The shape or hierarchical structure of the organisation: innovations started in relatively loose and centralised organisational structures with good communication channels have more chance to be successful; b. The demographics or level of diversity of its members: innovative teams containing of near peers tend to be more successful; c. Internal turbulence: such as high staff turnover or frequent re-organisations can disrupt the sustainability of an innovation; d. Internal boundaries: creating barriers to widespread diffusion of ideas and innovations within a system can inhibited the innovation; e. Internal resources: such as staff time, funds, office space etcetera facilitate the innovation. 3. Leadership: leaders play a central role at almost every stage of the innovation process. The competence of a leader can be rated through and influenced by: a. The leader’s vision: the leaders ideal and unique image of the future; b. Temperament of the leader: whether the leader is flexible, intuitive or whatever; c. Communication: the degree and sort of communication between the leader and the organisations; d. Durability or endurance: a leader must prepare for stress and anxiety that will result of the learning process and changes; e. Innovation skills: these skills are associated with management skills, a leader must be able to create space to experiment, develop and innovate.

21


4. Internal management systems: good managers help to keep the organisation on track. Six management factors can be identified according to Light (1998) that contribute to an innovation success: a. Mission management: refers to knowing whom or what the organisation serves (strategies, planning, evaluation, contacts etc) facilitates the innovation; b. Pay and personnel: payment and rewarding of the personnel is crucial for the success of an innovation; c. Learning: facilitating organisational learning by the group rather than by one person can contribute to the success; d. Idea generation: providing establishing systems that provoke new ideas can facilitate the innovation; e. Budget: a balance between protecting the organisation’s resources and the investments (Return-on-Investment) increases the chances of success; f. Accountability and governance: monitoring, supporting and coaching are important. Light (1998) suggests with this model that when changing the structures and culture of an organisation, you need to turn the traditional organisation into a “learning organisation”, that is an organisation that adapts to innovations and restructures itself to accommodate change. However, it needs to be noted that this model may not necessarily apply to school restructuring in the same way as it does to business or corporate restructuring.

Synthesis The goal of this study is to review the literature on success and failure factors of innovation projects with a main focus on the sustainability of successful ICT-related educational innovations. SURF’s main interest in this project is to promote that the innovations aimed at in the educational innovation projects they fund will be more successful, particularly, in making the shift from externally-funded to sustained initiatives. To meet that interest the literature study was conducted using both a business and educational perspective to identify the phases and variables that are needed to accomplish a successful innovation project and to prevent project failure. It was recognized that innovation projects take place in different phases, which can be summarized in the three main phases of initiation, implementation, and institutionalisation. In the initiation phase the present situation needs to be assessed in terms of goals, problems, and discrepancies. Then, ideas for the new situation need to be planned taking into consideration the changes envisioned and the route from the present to the new situation. Next, the consequences of the effects of the innovation on other aspects of the organisation need to be evaluated. In the implementation phase the innovation is developed, implemented, and evaluated. In the institutionalisation phase the innovation needs to be scaled and maintained. In this paragraph we synthesize the different perspectives and present an integrative model that might be used by SURF as a guidance in establishing policies and procedures, in formulating new guidelines for project proposals, in the assessment of SURF Education innovation project proposals, in conducting a project’s health check to identify warning signs for failure, and in promoting the diffusion and sustainability of the projects’ innovations. The model presented below shows the factors that are considered imperative to accomplishing a successful project, categorized by project stage. These factors are formulated in such a way that they can easily be converted into

22


a checklist format that can be used by SURF in the assessment of the project (proposals and reports) and by applicants in the writing of project proposals (see Chapter 4). It should be noted that although these variables are identified as major contributors to project success, they will never guarantee success alone. Initiation: In the initiation phase the present situation needs to be assessed in terms of goals, problems, and discrepancies. Then, ideas for the new situation need to be planned taking into consideration the changes envisioned and the route from the present to the new situation. Next, the consequences of the effects of the innovation on other aspects of the organisation need to be evaluated. The following factors are considered important in this phase: • There should be clear project objectives • The general mission of the project should be clearly defined • The project’s scope should be adjusted in such a way that the level of changes needed on the route from the present to the new situation can be clearly envisioned • (Simplified) Return-On-Investment should be taken into account • Stakeholders should be identified • Formal feedback channels must be created • There should be an experienced project manager • The project manager should not be the organisational manager • The project manager should be given responsibility and authority • The project team members should be competent • Care must be taken of clear responsibility and accountability of team members • The project manager and team members should be able to explain their efforts and results in ways that the larger organisation can understand • Project manager (team) should be prepared to re-plan • The project manager and team members should listen to resistors of innovation because they are often aware of unintended consequences of the innovation • There should be commitment from executive management and peers • There should be support from executive management and peers • Realistic expectations should be created Implementation: In the implementation phase the innovation is developed, implemented, and evaluated. The following factors are considered important in this phase: • User involvement should be ensured • User expectations about the innovation should be managed • Stakeholders should be engaged • Initiator of the change should be trusted and respected by the prospected users • Adequate communication channels should be created • Focus should be on adoption rate of approximately 25% of the system members • Focus should be on affecting opinion leaders’ attitudes (the more opinion leaders adopt the system, the lower the critical adoption rate for other system members will be) • Project manager and team should be open to external criticism • Project manager and team should continually question own assumptions • Project manager should continue to modify plan based on realities

23


Institutionalisation: Institutionalisation means continuing the newly implemented change or stabilizing the use of an innovation (Sherry, 2003). In the institutionalisation phase the innovation needs to be scaled and maintained. The following factors are considered important in this phase: • The organisation should be fitted in a stable environment which is supporting and collaborating • The organisation should have a relatively loose, centralized structure with good vertical communication channels • There should be competent leadership and management • The amount of change should be carefully dosed • User expectations should be managed: Innovation projects will fail if the users of a system are dissatisfied with it because it does not meet their expectations. Therefore, project managers should not only manage the development of the system, but also the perception of the system • The innovation’s relative advantage as compared with the current practices should be communicated • Structures should be created that promote learning of new practices and observable incentive systems that support them • Effective communication among all parts of the system should be created • A high degree of observability, that is, a degree to which other persons than the innovator see its results as beneficial, should be created This has led the researchers to a rethinking of the phasing and the preparation (the initiation phase) of the project structure which are presented in Figures 7 and 8.

Initiation phase

assess present situation disseminate plan


design disseminate develop

plan disseminate scale

disseminate

disseminate

disseminate

evaluate consequences

implement

maintain

disseminate

evaluate

Figure 7. Phasing of a project with sub-processes per phase.

24

Institutionalisation phase


Plan the innovation

Backcast to today

Where in 10 years?

Thought experiment

Commitment at the top

Vision at the top

Begin

Figure 8. Keeping implementation in mind during the initiation phase.

Since the educational innovations that are sponsored in the SURF Education program take place within the time and funding limitations of SURF’s grant, and the sustainability of the innovations is considered a major goal by SURF, it seems necessary to require applicants to present a detailed plan for sustaining the innovation and to challenge them to realize this plan after the project has ended, i.e. once the funding period of the innovation grant is over. To accomplish this, the success factors that are imperative for sustainable innovations to materialize, must be clearly communicated to the applicants/project managers. In addition, project managers and team members need to be familiarized with methodologies that can be used in the institutionalisation phase to attain sustainable innovations. In this respect, the CATWOE and Soft Systems Methodology of Checkland and Coles (1990) seem promising (see Appendix II). Furthermore, we believe that a financial incentive could be used to challenge the project team and the responsible educational institutions to maintain the innovation and diffuse it to other institutes. Currently, there is no financial incentive to sustain innovations. Lack of funding in this phase is certainly a disincentive, especially when adopting an innovation means that individuals must go through a learning curve and take on new responsibilities as a result of developing expertise (Sherry, 2003). Alternatively, it would be possible to make part of the funding conditional to the realization of the plan for sustaining the innovation. Regardless of which solution is chosen to promote the sustainability of an innovation, there must be a means to protect the organisation if the innovation proves too costly to sustain. The identification of success and failure factors of educational innovation projects by means of literature study was the main goal of the first stage of this research project. In the second phase, a so-called ‘group concept mapping’ procedure will be used to determine the factors that project management experts consider important for the success of large-scale educational innovation projects (Chapter 2). In the third phase, interviews with the project leaders of more and less successful SURF Education projects (tenders 1999, 2000, and 2001; see Appendix I) will be held to find out how

25


they think about and have taken into account the success factors that were found in the literature study (Chapter 3). In addition, an analysis of the documentation of these projects will be conducted to find out how the more and less successful projects have dealt with these factors (Chapter 3). It is expected that the overall results that emerge from these three stages can be used by SURF in their pursuit of successful projects with sustainable educational innovations. Although, this literature review has shown that the predominantly retrospective analyses of innovation projects can reveal interesting models, comprising numerous failure and success factors of innovation projects, there seems to be no firm empirical basis for these models. Therefore, it seems necessary to conduct more systematic research into the mechanisms that cause project success or failure.

Lessons learned – Eye openers 1. Project success is not innovation success: There are two types of ‘success’, namely the success of the project itself and the success of the innovation (its ultimate implementation). These two ‘successes’ are completely different. Neither type is a requirement or a guarantee for the other. Also, success of an innovation at the project level does not automatically generalize to success at the institutional level, but needs careful considerations of scalability, generalisability, temporal flexibility, and financial sustainability. 2. Sustainability seems to be more an afterthought rather than a planned strategy for maintaining change. 3. A project needs a project manager and not an organisational manager: There is a major difference between being a competent project manager and a competent organisational manager. The main purpose of organisational management is to plan, guide and help the organisation to work well and survive; in other words to achieve continuity. Project management, on the other hand, is to actually actively plan the end of a project; in other words to break with continuity and achieve change. A project must end in order to be successful whereas an organisation must not end in order to be successful. Also, in contrast to organisational managers, project managers have to start their task from scratch, initiating and bringing an organisation to life. To start something in order to end it within a fixed period of time, according to a budget and according to a number of restrictions distinguishes the nature of project managers from the nature of organisational managers. Neglecting this difference can lead to incompetent project management, both in the industrial organisations and in the educational organisation. According to the Standish Group, 97% of the successful projects investigated in their research had an experienced project manager at the helm. 4. Sure fire failure causers: High investment with low output: There is no balance between investments and output. Information politics: Power is abused and information is not transmitted. Lack of responsibility: There is uncertainty about the responsibility of people within outside of the project. Culture gap: This primarily has to do with the gap between ICTers and the rest of

26


the organisation, but could also represent the gap between those planning education (managers) and those administering education (teachers). Over-commitment: Not knowing when to cut your losses and stop a project. All-in-one solutions: trying to do everything at once instead of using multiple steps and phasing. 5. Critical ‘win drivers’: Defining and anchoring the project objectives: Defining the project’s objectives and goals early in order to involve the stakeholders and anchor a common understanding and commitment Roles and responsibilities for achieving wins: Important roles like the role of the client, users and project manager need to be well defined and established Necessary change process: Innovative projects involve changes and adequate change management is often neglected Follow-up and measure wins: Failure of projects can sometimes be related to lack of control and performance management, this has to be measured systematically and thoroughly. 6. Small changes often are larger than we think The greater the number of changes is, the more factors there are which influence the chances of success and therefore there is an increase in the risks of the project. In other words, an increase in interaction leads to a decrease in control, which leads to a decrease of the chance of success. Investigating the magnitude of the change beforehand, and taking the result of this investigation into account when implementing the project, can diminish the chances of failure 7. Innovation is like a nuclear reaction: Small is beautiful Mass Diffusion theory states for most members of a social system, the innovationdecision depends heavily on the innovation-decisions of other members of the system. When about 10-25% of the system members adopts the innovation, relatively fast adoption by remaining members will follow. Initial full-scale implementation is not necessary for full-scale adoption. As a matter of fact too large implementation could even prove detrimental. 8. Opinion leaders - Know your enemies and make them allies: Since leaders and certain group members (opinion-leaders in which the other group members have great trust) directly affect the innovation-decision of the other members, a powerful way to affect the diffusion of an innovation is to affect the leader’s and opinion-leader’s attitudes. Opinion-leaders need not necessarily be the hierarchical leasers. Getting an acknowledged sceptic on board could be more important than getting a boss on board.

27


Chapter 2: Expert Concept Mapping Study6

6

This chapter is a preprint of an article submitted for a special issue of the British Journal of Educational Technology on “Innovation in e-Learning - Lessons to be learned” Wopereis, I. G. J. H., Kirschner, P. A., Paas, F., Stoyanov, S., & Hendriks, M. (2005). Determining Failure and Success Factors of Educational ICT Projects Using a Group Concept Mapping Approach. Manuscript submitted for publication.

29


Abstract Robert Burns wrote: “The best laid schemes of Mice and Men oft go awry”. This could also be the motto of most educational innovation, especially innovation involving the use of information and communication technologies. The question is not only why some innovations succeed (although this is very important question), but also why many or even most others fail? This study investigated the failure and success factors of large-scale educational innovation projects in Dutch higher education. To validate the factors found in earlier literature research as well as to further expand them experts in the field of innovation and innovation management took part in two research sessions making use of a Group Concept Mapping Procedure. Experts were required to record all potential failure and success factors (in total 220) after which they were asked to cluster them using an affinity technique. Finally, the experts were asked to rate each of these factors on their importance for educational ICT innovation projects, thereby creating a hierarchy of potential failure and success factors. The practical implications of this hierarchy are discussed.

Introduction As is the case in most Western countries, the Dutch Government invests heavily in stimulating better and more creative use of information and communication technologies (ICT) in all forms of education, including higher education. Examples of programs implemented by the Dutch government since the mid-1980s for this purpose are the Informatics Stimulation Plan (INSP), the Software Program Development for Computers in Education (POCO), the Operational Plan for Stimulating Activities and Procedures (OPSTAP), Project Implementation of New Technologies (PRINT), Project Management for Effective Stimulation of Technology in Education (PRESTO), and Committee on Multimedia in Teacher Training (COMMITT). The ultimate goal of all of this investment is to ensure that students and teachers are equipped with the knowledge and skills required for success in the new knowledge-based economy. Stakeholders (e.g., government, industry, educational institutions, society in general) have placed high priority on achieving this goal. However, these highly funded projects have often resulted in either short-lived or local successes or outright failures (see, Alexander & McKenzie, 1998; De Bie, 2003; Teasly, 1996) questioning the value of such programmes. A two-year national study to evaluate the contribution of ICT projects to student learning in higher education, for example, was conducted in Australia by Alexander and McKenzie (1998). The study reviewed over 100 projects that had received teaching development grants and made significant use of a range of ICT to develop student learning materials. After a detailed literature review, a questionnaire was developed and sent to the project leaders of 173 projects across Australia and finally these questionnaires were analysed. A striking outcome of the questionnaire was the large discrepancy between the intended outcomes of the projects and the actual outcomes reported. While 87% of the projects’ leaders noted “improved quality of learning” to be an intended outcome of the project, only 30% of them reported this as an actual outcome. Although this discrepancy can be due to failure of the project and/or failure of the project to measure this, it seems justified to state that the majority of the projects have not been successful in achieving their intended outcome. Furthermore, the study has shown that technology, in itself, does not make a difference improving learning outcomes or assures successful educational innovations.

31


This is not strange if we look at the track record from a related field. The Standish Group 2000 study (Johnson, Boucher, Connors, & Robinson, 2001) determined, for example, that only 28% of the IT projects investigated were successful. They defined success as a project that is completed on time and on budget, with all features and functions originally specified. But 28% success means more than 70% that can euphemistically called ‘less than successful’. They determined, thus, that 49% of all projects were challenged, that is the project was completed and operational, but came in over budget, late, and with fewer features and functions than initially specified. Finally, 23% are outright failures whereby the project was cancelled before completion, or was never implemented. Although this percentage is better than the percentage of a comparable study of the Standish Group in 1994 (16%), the number of un-successful projects is considerable. As a result of such results, the role of ICT is developing less quickly in higher education institutions than was previously expected. In order to steer these developments in the right direction, the SURF Platform ICT and Education (ICT-E), a government-funded national organisation in which all higher education institutes in the Netherlands participate to increase the pace of educational innovation, has set up a program for educational innovation projects. The goal of this program is the systematic stimulation of the application of ICT to innovation in higher education. SURF’s educational innovation projects are intended to be a source of inspiration for the introduction of ICT-based innovation in education and are aimed at better and sustainable results. SURF’s educational innovation projects may cover one or more of the following key issues: competence/portfolio, collaborative learning, interactive teaching materials, learning content management systems, communities and new media. Two important questions arise with respect to educational innovations. First, why are some innovations less successful than others? Second, why do so many innovations fail, while just a few others succeed? In order to answer these questions a research project was set up which tried to identify the determinants for failure and success of large-scale educational innovation projects in Dutch higher education, in particular of those funded by the SURF Platform ICT-E. The identification of such factors for educational innovation projects and measuring performance of projects based upon them are considered important as they can aid the development and implementation of innovation projects by explicating and making visible those failure and success factors relating to educational innovation projects in higher education. This is necessary to make these factors manageable and to enable future projects to achieve better and more sustainable results. The research project included four phases. First, a literature study was conducted to identify existing knowledge about success and failure of educational innovation projects (see Chapter 1). Second, experts in the field of educational and organisational management and innovation management participated in a group concept mapping procedure (GCMP). Third, project documentation and evaluations of educational innovation projects funded by the SURF Platform ICT-E were analysed to validate the failure and success factors found in the literature and to derive new factors. Finally, the project managers of these educational innovation projects were interviewed to determine how they have dealt with the identified failure and success factors. The group mapping procedure with experts is at centre of this article and will be discussed in depth.

32


Group Concept Mapping Procedure In the planning stage of this research project, the team determined that a more empirical technique was needed to complement the desk research (literature study and document analysis) and qualitative interview data from the other phases of this project for two reasons. First, by the project’s inception, the research team had the impression that research on success and failure factors for ICT innovation projects within the domain of education would be relatively scarce. The literature study (see Chapter 1) confirmed this presentiment and justified an additional search for factors. Table 2 shows the success and failure factors resulting from this literature study and their subsequent classification in 13 clusters, which was conducted by three experienced educational technologists of the Open University of The Netherlands. The second reason was that the team considered it to be of the utmost importance to validate the factors resulting from the literature review, since many of them would be / were distilled from research in other domains such as information technology and software development (Holmes, 2001; Johnson, Boucher, Connors, & Robinson, 2001; Karlsen, Svendson, & Omli, 2003; O’Hara, Watson, & Kavan, 1999) or general literature on project management and innovation (e.g., Storm & Janssen, 2004). Table 2. Success and Failure Factors Based on the Literature and Subsequent Classification in Clusters

Cluster 1 Delineation of project content

2 Delineation of project process 3 Project chair qualities

4 Project staff qualities 5 Organisational management 6 Organisational typology

7 Communication

Factors The problem is clearly defined The project goal is clearly defined The project objectives are clearly defined The project’s scope is clearly defined The project’s scope is realistic The project planning is specified The project’s prior / limiting conditions are clearly defined Stakeholders are identified Return-On-Investment is taken into account There is a competent / experienced project manager The project manager is not the organisational manager The project manager has the necessary responsibilities The project manager has the necessary authority Clear responsibilities of team members are defined Clear accountabilities of team members are defined The project team members are competent Team members (including project manager) can clearly explain the project Initiator of the change is trusted and respected by the prospective users There is competent organisational management There is competent implementation leadership The organisation exists in a stable environment, is supportive and collaborative The organisation has a relatively loose, centralized structure with good vertical communication channels Turn the traditional organisation into a “learning organisation” Formal feedback channels are created Adequate information-communication channels are created Effective communication among all parts of the system is created

33


8 Commitment / engagement 9 Project management 10 Expectations 11 Added value 12 Project culture 13 Scale / complexity

There is commitment from executive management There is support from executive management User involvement is ensured Stakeholders are engaged Observable incentive systems to support implementation are planned Structures and opportunities are planned that promote learning of new practices There is opportunity in the planning to modify the plan based on realities Realistic expectations are created User expectations about the innovation are managed User expectations are managed. The innovation’s relative advantage as compared with the current practices is communicated The is a focus on affecting opinion leaders’ attitudes There is opportunity planned for external criticism There is room for the team continually question its own assumptions There is a focus on adoption rate of maximally 25% of the institution The amount of change is carefully dosed

GCMP was chosen as empirical technique for generating, sorting, and rating (new) factors in de educational domain since it combines the advantages of qualitative methods for rich information collection with quantitative methods for precise data analysis (see for example, Hays & Scholla, 2003; Jackson & Trochim, 2002; Stoyanov & Kirschner, 2004; Trochim, 1989a, b). It is compatible with some classical methods for collecting information such as interviews and questionnaires, but adds value by imposing an overarching seamless strategy. Further, GCMP applies a sort of an expert focus-group discussion where experts share and describe their ideas in a structured way while working as a real team, not as a nominal group. It provides a user-friendly platform for structuring the discussion of a group of informed people, while triangulating multiple perspectives on a topic of common interest in order to describe their ideas and consolidate mutual understanding about a specific issue. The approach uses a simple and intuitive facilitated procedure, in which specific steps are planned by a facilitator to support participants in articulating their ideas while performing common and familiar activities such as brainstorming, sorting, and rating. It is believed that this procedure not only stimulates people to talk about successful projects, but also about projects that have not met their expectations, thereby uncovering both success and failure factors. As a part of cognitive mapping paradigms, GCMP shares some of the strengths of the other mapping approaches such as classical concept mapping (Novak, 1998), cognitive mapping (Eden, Ackerman, & Cropper, 1997; Eden & Ackermann, 2002), and mind mapping (Buzan & Buzan, 1996). It is an adequate, flexible, and intuitive way of making explicit the individuals’ cognitive realities about a specific issue. GCMP is a concise, compact, and parsimonious technique, which is at the same time rich of information because of the integration of verbal and visual coding. The technique capitalizes on the advantages of the graphical representations, without losing the flexibility and the richness of a natural language system (Stoyanov, 2001). However, in the classical cognitive mapping paradigms the relational structure in these graphical representations (i.e., relational isomorphic maps) is constructed by the labelled links between concepts. There is not an Euclidian framework for determining

34


the relationships between ideas. In addition, these mapping approaches do not provide a meaningful algorithm for combining the efforts of people participating in the discussion. GCMP can aggregate the individual contributions of group members into a final product by applying a specific algorithm. It develops a meaningful Euclidean framework to depict the relationships between ideas where the distance between symbols is interpreted as an empirical estimate of the semantic distance between ideas. The method applies powerful statistical procedures such as multidimensional scaling and hierarchical cluster analysis. Since the GCMP aimed at generating, sorting, and rating success and failure factors, it is important to know how ‘success’ can be operationally defined and how the dimension ‘success of educational innovation’ can be tapped. A problem here is that there are two different types of ‘success’ with respect to innovation, namely success of the innovation project itself (i.e., that the project delivered the promised results within the agreed upon temporal and financial constraints) and success of the innovation (i.e., its implementation in the organisation and its sustainability after initial, often external funding). These two ‘successes’ are completely different. Neither of the successes is a requirement nor a guarantee for the other. In general, success can be considered as the accomplishment of goals and objectives necessary to achieve a particular task. For instance, educational innovation projects funded by the SURF Platform ICT-E are considered successful based on the extent to which they stimulated and facilitated new and better use of ICT in education. The Platform is particularly interested in the sustainability of the technological innovations, in other words, how they can ensure that the innovations, both technical and pedagogical, make the shift from an externally funded initiative to a sustained ‘standard operating procedure’. The rationale for sustaining successful ICT-based educational innovations is to preserve what has been valued and built for continued use. Inherent to the concept of success factors is a notion that if success factors are implemented in the project, the educational innovation or project will perform better and be more successful. Success factors can thus be looked upon as individual, independent variables influencing the dependent variable ‘success’. Analogously for failure factors, there is a notion that if failure factors are avoided in the project or implementation, that the innovation will have a better chance of being achieved. Performance measurements for success or failure can be classified as objective and subjective. The objective measures are based on measurement of past performance or output while subjective measures let individuals with an intimate knowledge of educational innovation be the judges of what is successful. It should be recognized that success of an innovation at the project level does not automatically generalize to success at the institutional or national level, but needs careful considerations of scalability, generalisability, temporal flexibility, and financial sustainability. To be able to determine which of the identified success factors for educational innovation projects are most relevant in different educational contexts, a contextual framework to position educational innovation and related success factors needs to be developed. The basic idea is that if we know why some projects fail or have only short-lived successes and we can avoid making these mistakes, it is possible to make sure that new projects will succeed with sustainable results. However, it should be noted that ‘non-failure’ is not considered the same as success. Success/failure should not be considered as a binary classification, but should be viewed along a continuum.

35


Method Participants Experts (n = 13; all male) in the field of educational or organisational innovation and innovation management from six different universities, two funding organisations for educational innovation projects in higher education, an expertise centre for innovation in education and training, a corporate training institute and an educational consultancy firm took part in the concept mapping procedure. Of the university experts, six had a position as full professor, three were associate professor, and two were assistant professor. The non-university experts were all seniors within their organisations. All experts were eminent project managers with at least 10 years of experience in managing large-scale projects. Procedure The group concept-mapping procedure for identifying success and failure factors was based on the concept mapping procedures of Trochim (1989a, b) and Stoyanov and Kirschner (2004) and consisted of two main activities or phases, namely data collection and data analysis. Data collection. During the data collection phase, success and failure factors of educational ICT innovation projects were successively generated, sorted, and rated by the experts. First, expert meetings were organised for factor generation. To enhance factor generation, a combination of classical brainstorming techniques such as the brainwriting pool and gallery method (VanGundy, 1997) and Post-it® Notes problemsolving (Straker, 1997) were used. Due to pragmatic (i.e., geographical) and organisational (i.e., scheduling a face-to-face meeting) reasons, the expert group was divided into two subgroups and as a consequence two expert meetings were held (the ‘North-group’ (n = 7) and the ‘South-group’ (n = 6)). The variation of expertise in both expert meetings was similar and the procedure was equivalent. First, the experts were introduced to the project, the group concept-mapping procedure in general, and the expert-meeting in particular. After this brief introduction, the expert-meeting continued with the first factor generation session. The experts were asked to individually generate as many failure and success factors as possible and write them down on Post-it® Notes. After this initial 10-minute period of individual brainstorming, the participants were invited to present their generated factors to the group in a round-robin fashion. One after the other the experts picked one of their notes and put it on a large white wall while reading the content of the factor out loud. In this phase, the experts either contributed factors or listened to the contributions of the others with the aim of generating new factors. As is often the case in brainstorm sessions, criticism was not allowed. New (individually) generated factors were also written down on the Post-it® Notes and introduced to the group as just described. The whole procedure proceeded until all the (new) factors were presented and posted to the factor collection on the white wall. After the round-robin procedure, a second, 10 minute, factor-generation session was held. Here, the experts were asked to study the group collection of Post-it® Notes on the wall in order to generate new factors. Again, a 10-minute round-robin procedure was held to present the factors. The expert meeting concluded with a 30minute discussion to clarify, criticize, and defend the generated factors and – again allow experts to generate new factors.

36


After the two expert meetings, the factors on the Post-it® Notes were digitalized in MS-Word® and combined into one list. Double factors were removed from the list. This yielded a total of 220 failure and/or success factors for carrying out ICT-innovation projects and implementing their results in the organisation (see Appendix III). An authentic card sort task was chosen for sorting and rating the factors. Each of the experts received a package of cards with factor description, factor ID, and space for a factor rating. We chose to use a card-sort with real cards instead of an electronic cardsort so as to allow the experts to maintain an overview during the process of sorting such a large number factors. The experts sorted and rated the factors individually at home. Along with the factor cards, the experts received envelopes to file card clusters and an instruction booklet to guide them in the mapping procedure. Thirteen of the envelopes provided were prepared with cluster titles derived from the earlier described literature review of success and failure factors (OTEC, 2004). These titles were given so as to simplify/support the factor classification (of the 220 factors) and to help crossvalidate the results of the literature review. The experts also received unlabeled envelops for filing self-determined clusters. The sorting process consisted of three steps. The first step required the participants to classify as many of the factors as they could into the 13 clusters from the literature review. The clusters were: project content, project processes, project manager, project management, project membership, institutional management, organisation where the innovation occurs, involvement, communication, expectations, surplus value, project culture, and innovation scale/complexity. Factors that did not fit this classification were put aside into a ‘rest category’. The second step required the participants to group the rest category of factors into new clusters. Guidelines were formulated (see also Stoyanov & Kirschner, 2004; Trochim, 1989a) for both of these steps, namely: (a) classify or group the factors for similarity of meaning, (b) arrange the factors in a way that feels best (there is no right or wrong grouping), (c) place each factor into one group only, (d) place each factor somewhere and (e) place a single factor in its own group if it is unrelated (in the participant’s opinion) to the other factors. If new clusters were determined in the second step, the participants were asked to write down a short phrase or title describing the content of the cluster (the third step). For the rating procedure, the experts were asked to rate each of the 220 factors on a 5-point scale of importance. A rating of 1 meant that the factor was unimportant in educational ICT innovation projects and a rating of 5 meant that the factor was very important. The results of the data collection phase were stored in different formats. The success and failure factors generated in the expert-meeting were digitalized in MSWord®, the factor clusters and factor ratings were backed up in MS-Excel®, factor ratings were transferred to SPSS® for calculating descriptive statistics, and the factors, clusters and ratings were read into the Concept System® for multidimensional scaling, hierarchical cluster analysis and the computation of (cluster) ratings. Both the procedure and the results (the collection of Post-it® Notes) of the expert meeting were recorded on digital video as back-up. Data analysis. The data analysis phase involved multidimensional scaling of unstructured sort data, hierarchical cluster analysis, computation of average ratings for each factor and cluster of factors, and the computation of ‘bridging values’ (Trochim, 1989a, 1999a, 1999b). Concept System® was used to analyse and visualise the data. With respect to the computation of the average ratings SPSS® was applied since

37


Concept System® only reports the mean and no other measures. Bridging values were computed by Concept System®. The analysis began with the building of an N x N (N = total number of factors) binary symmetric matrix of similarities (SNxN) for each participant from the sorting data. For any two statements i and j, a 1 was entered in Sij if the two statements were placed in the same pile by the participant, otherwise a 0 was entered in the cell. The total TNxN similarity matrix was built by summing across the individual SNxN matrices. Any cell of this matrix can take integer values between 0 and M, where M is the total number of participants who sorted the statements. The total similarity matrix, TNxN is the raw structure data for the multidimensional scaling analysis. Multidimensional scaling produces a two-dimensional XNx2 configuration (i.e., point map) of the set of N statements based on the criterion that statements piled together most often are more proximately located in a two-dimensional space while those grouped together less often are more separate from each other. Based on traditional estimates of reliability, Trochim (1993) constructed specific reliability estimates for concept-mapping data. He reported strong positive split-half total matrix and map reliabilities and average individual-tototal, individual-to-map, individual-to-individual, and rating-to-rating reliabilities. Citing Kruskal and Wish (1978), Trochim (1999b) argues that two-dimensional solutions of multidimensional scaling are easier to apply than three or more dimensional solutions and more desirable when the results of the clustering need to be displayed. Multidimensional scaling generates a rather detailed point map, which was visualized by Concept System (see Figure 9). The results of the multidimensional scaling were used as input data for a hierarchical cluster analysis, providing a more global picture transforming the point map into non-overlapping clusters (see Figure 10). The number of clusters was identified by taking vertical slices at different heights of the cluster tree. There is no simple mathematical criterion for selecting the number of clusters. Trochim’s approach favours first examining the maximum number of possible options before going down the tree to reduce the number of cluster solutions. This process always involves judgment and interpretation. Three activities were carried out to facilitate judgement and interpretation. First, the concept maps provided by Concept System were examined thoroughly. Overlapping clusters had to be avoided. Second, the bridging values of the factors and factor clusters were examined. According to Jackson and Trochim (2002) the bridging value denotes how often a factor is sorted with others that are close to it on the map or whether it is sorted with items that are farther away on the map. Concept System computes bridging values for single factors and factor clusters. The bridging value ranges from 0 to 1. Statements with lower bridging values are less often sorted with statements from outside a cluster; therefore, they are assumed to be more central to the interpretation of their own cluster. In other words, lower bridging values for a single factor indicate a ‘tighter’ relationship with other factors in the cluster. Clusters with low bridging values are usually more cohesive and easier to interpret. The third activity with regard to choosing a final cluster solution was a semantic analysis of the factors. Rating data provides a third dimension of the data. Concept System was also used for visualizing the results of this analysis (see Figure 11).

38


Results The participants generated 231 factors. Identical (double) factors were removed leading to a final list of 220 factors. Figure 7 depicts the detailed point map of these factors, which was generated by a multidimensional scaling technique. Each point on the map is accompanied by a factor identification number linked to the definition of the factor. 182 35214 52 39 185 170 107 43 83 132 101 109 217 11

180 18 141 211 16 50

187 196

93 24 33 115 156

63 116

137 49 218 103 220 70 55 34 111 44 42 41 74 10

195 36

60 72

153

46

151

206

27

78

152

40

136 193

64 199

106

19

219 181 91

98 203 201

150

82 119 213

12

198 54167 23138 133

166 128

32 207

90

146 99 139

21 25 81 189 191 13205 76 104 100 154 51 161 80 135 95 62 200 134 130 9 68 5 102

56

77

197

216

125 2

79

15 31 194

22

159 47

121

171

26

97

176 6 58 145 164 114 169 92 192 45 129 28 87 210 71

209

29 147

143

148

57

30

163 149

172

126

142 202

4

160 158 105

69 168

173

3

179

122 188

48 110 96 37

123 86 38178

183

85 67

162

174

208

7 131

17 124

112

94

108 89 184 127 165 117 113

120 177

14

190

155

212

65

175 186

66 88

1 215

53

118

84 20

73 157 61 204

59 75 140 8

144

Figure 9. Multidimensional point map of the statements.

Next, a hierarchical cluster analysis of the raw data was carried out to identify how experts classified factors into groups. The resulting 13 clusters are presented in Figure 10.

39


8 9 6

5

7

3

10 1 4 11

13 2 12

1. 2. 3. 4. 5. 6. 7.

Competency of the project team and its members Competency of the project manager Focus and complexity Project optimisation: Start small and then go for it! Participation, Information and Communication Stakeholder involvement Position of innovation within the organisation

8. 9. 10. 11. 12. 13.

Added value Relationship between project and real problem / core business Organisational culture Quality of institutional management Project and process management Means to support the project

Figure 10. Clusters resulting from the hierarchical cluster analysis, indicating how the experts classified factors into groups.

A semantic analysis of the factors within the clusters led to the following descriptions: 1. Competency of the project team and its members. The factors within this cluster deal with the expertise and experience of those staffing the project with regard to content and creativity. It is a cluster aimed at the WHO question. 2. Competency of the project manager. Also a WHO-question. Apart from the fact that a project manager should be dedicated and have a strong backbone, (s)he should be an expert in project management matters (‘knowledge of operational processes’; making a team of the group), and finally should also be a subject matter expert. 3. Focus and complexity. The WHAT-question. This cluster is about the sharpness of the problem. It deals with matters such as where a team is at the beginning of the project and where the team is going. It has also to do with how diffuse and complex the project is. 4. Project optimisation: Start small and then go for it! This cluster deals with the HOW-question. Know the difference between a pilot and implementation, don’t try too much (zone of proximal development), et cetera. 5. Participation, Information and Communication. This cluster also deals with the HOW-question. Participation of users is central within this cluster and in order to effectively and efficiently participate, good information and communication is needed. An interesting consideration here is participative design.

40


6. Stakeholder involvement. The WHO-question. This cluster is related to cluster 5 and has to do with the getting all of the parties ‘on-board’: teachers, students, and administrators. 7. Position of innovation within the organisation. The WHERE-question. This cluster is about the positioning of the project within the organisation and the relation with its mission, goals and ultimate implementation. This cluster includes also implementation, ‘follow up’ innovation, sustainability, and establishing the infrastructure for the innovation. 8. Added value. The WHY-question. This cluster has to do with the importance of the innovation for the stakeholders and the organisation. All the parties should benefit from the innovation. 9. Relationship between project and real problem / core business. This cluster has also to do with the WHY-question. The project should solve problems that really exist and which are related to the core business in education in general and specifically the institution. This means improvement in the quality and quantity of learning activities and not innovation for innovation’s sake. 10. Organisational culture. The focus within this cluster is on the WHERE-question. It has to do with the creation of an organisational culture that supports the initiation, implementation, and institutionalisation of innovative projects; an organisation with a change culture / which is not afraid to change. The learning organisation is an innovative organisation. 11. Quality of institutional management. This is a HOW-type of cluster. It is related to the previous one and describes - in operational terms - how the organisational culture can be transformed into an innovative culture.. 12. Project and process management. This cluster is about the HOW-question. It has to do with good project organisation and structure: planning, management performance, interaction, processes, and group dynamics. The cluster also emphasises characteristics of the project leader important for effective and efficient project management. 13. Means to support the project. The cluster is aimed at the HOW-question. It is about developing strategies and providing means for executing an innovative project. Table 3 presents the clusters. Per cluster, five factors that can be regarded as the best indicators are also given.

41


Table 3. Clusters with 5 Success Factors With the Highest Bridging Values and Their Average Rating

Cluster number and name 1 Expertise and creativity of the team

2 Competency of the project manager

3 Focus and complexity

4 Start small and then go for it!

5 Participation, Information, Communication

42

Factor Competent project members –content and organisation Form an expert and professional team Creativity of participants Creative thinking Teambuilding with participants Average for Cluster 1 (Factors N = 17) Project manager should be competent Project manager should understand the content Completely dedicated and competent project manager Knowledge of operational processes Making a team of a project team Average for Cluster 2 (Factors N = 15) (In)sufficient focus Latitude, elbow room Multidisciplinary view of innovation Limited number of parties involved in the project Consciousness of the need to innovate and change and/or for a paradigm shift Average for Cluster 3 (Factors N = 12) Distinguish between pilots and implementations with respect to the consequences for staffing, control, and tempo Can the innovation, during the process, be adapted? Don’t change the means into an end Innovation that takes small steps has a greater chance of success Stay in the zone of proximal development Average for Cluster 4 (Factors N = 12) Real involvement of teachers from the start Delegating initiative and allowing others to take it Participation of those concerned during the trajectory Inform Clear and frequent communication before, during and after the project Average for Cluster 5 (Factors N = 19)

Bridging value .27

Average rating 4.23

.29 .31 .34 .36 .49 .17 .17 .17

4.33 3.46 3.46 4.15 3.53 4.46 3.77 4.38

.31 .32 .39 .30 .36 .37 .37 .38

3.00 3.85 3.44 3.23 3.77 3.62 2.92 4.15

.39 .19

3.56 3.62

.24 .25 .25

3.92 3.77 3.23

.26 .30 .17 .22 .25 .25 .25

3.00 3.25 3.92 3.77 4.15 4.23 4.15

.32

3.75


Cluster number and name

Factor

Involve students and “clients” in the innovation Involve the consumers (teachers, students, administrators) 6 Attend to “collective concept formation”; give Stakeholder meaning involvement Support of the users Attend to perceptions of those involved: appreciation and use Average for Cluster 6 (Factors N = 9) Position the project in the total “educational value chain” “Follow up” innovation, make it sustainable and less 7 dependent upon “the project” Position of Subject / goal fit the hype of the moment (in the year innovation 2000: E-portfolio; in the year 2004: wireless) within the Formulate a clear idea / goal organisation Don’t get caught up in the innovation-mode (instead of embedding and evaluating, starting on a new adventure) Average for Cluster 7 (Factors N = 19) Quality; added value above existing solutions Student motivation to use innovation: fits own experiences, added value To what extent does the innovation solve a problem? 8 (innovation is not only new but also better) Added value What does the innovation contribute: clients, businesses, educational partners? Their success increases ours. Clear contribution to performance (increase) of the learner Average for Cluster 8 (Factors N = 24) Is there a problem to be solved? Solving a problem that doesn’t exist is a sure way to 9 fail Relationship Relationship to (core) business between project Demonstrable relation innovation and improvement in and real problem the quality and quantity of student learning activities / core business Innovation that increases the flexibility of a system has a better chance of succeeding Average for Cluster 9 (Factors N = 11)

Bridging value .23 .25

Average rating 4.00 4.31

.26

3.85

.32 .35

3.92 3.77

.37 .32

3.68 3.46

.34

3.77

.34

2.38

.36 .36

4.00 3.38

.39 .00 .01

3.29 4.08 3.85

.05

4.46

.06

3.69

.07

4.15

.20 .10 .15

3.73 3.92 3.46

.19 .20

3.27 4.23

.23

3.15

.24

3.51

43


Cluster number and name

Factor

Connection to other projects and developments Organisational change 10 Organisational (re)design and development go along Organisational with innovation of the primary process culture Does it fit in the organisational culture? Room to experiment Average for Cluster 10 (Factors N = 18) Institutional management should minimally understand what it’s all about Institutional management should not concern itself 11 with operational questions and matters Quality of Continual management attention and monitoring institutional Flexibility in the organisation to make time and management funding available Lack of interest by the institutional management is a killer Average for Cluster 11 (Factors N = 17) Effective project management system (no ‘paper 12 tiger’) Project- and Poor / strong project management performance process Clear tasks and responsibilities management Process management A well thought-out project plan Average for Cluster 12 (Factors N = 28) Explicit change strategy and approach Combine design and development approaches 13 Meta-factor: innovation is not implementation Means to support Separate innovation from the status quo: innovation the project upsets! Accent on control is a sure loser! Average for Cluster 13 (Factors N = 19) Note:

Bridging value .27 .28 .29

Average rating 2.69 3.23 3.62

.32 .32 .34 .15

3.77 3.62 3.37 3.85

.15

3.23

.19 .21

4.15 4.00

.22

3.77

.25 .14

3.65 3.38

.16 .16 .17 .17 .28 .19 .20 .23 .23

3.77 3.77 3.69 3.08 3.61 3.85 3.69 3.00 3.31

.25 .28

2.85 3.42

The average values presented at the end of each of the clusters are not the averages of the five factors above them, but of all of the factors in the cluster. Cluster 1, for example, is composed of 17 factors and the averages presented are of those 17.

A complete overview of the 13 clusters and the 220 factors they comprise is presented in Appendix IV. The expert ratings of the 220 factors regarding their importance for educational innovation projects are presented. Table 4 shows the ratings for the top-10 percent of the factors (N = 22). Table 5 shows the 5 percent of the factors (N = 11) that received the lowest ratings. In Appendices III all 220 success and failure factors and the accompanying ratings are listed.

44


Table 4. The 22 Most Important Success and Failure Factors

Rank 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22

Factor Make the added-value visible Make the benefit of a product clear (new is not always better) Choose a competent project chair Celebrate your successes Management must be involved and competent The project chair must be completely dedicated Ambition counts! Form an expert and professional team Involve all stakeholders (teachers, students, administrators, directors) Keep the culture open Demonstrable relation innovation and improvement in the quality and quantity of student learning activities Competent project members –content and organisation Inform Strong key people in key places Clear and frequent communication before, during,after project Shared vision about what and how Added value must be clear for everyone from the beginning Continual management attention and monitoring Successful partial products / visible results Involvement by the organisation Participation of those concerned during the trajectory (Omnipresent) realization of the need to change and continual attention to conceptual change / paradigm shift

M 4.54 4.46 4.46 4.46 4.38 4.38 4.38 4.33 4.31

SD .66 .52 .66 .78 .65 .87 .96 .78 .75

Cluster 8 8 2 5 11 2 5 1 6

4.31 4.23

.95 .60

1 9

4.23 4.23 4.17 4.15 4.15 4.15 4.15 4.15 4.15 4.15 4.15

.73 1.01 1.03 .69 .69 .69 .69 .80 .90 .90 .90

1 5 13 5 8 8 11 12 3 5 3

M 2.17 2.23 2.25 2.25 2.31 2.31 2.38

SD Cluster 1.34 7 .83 2 1.14 1 1.14 6 1.25 10 1.11 3 1.26 7

2.38

1.19

4

2.46

1.33

13

2.50

1.45

8

2.54

1.56

11

Table 5. The 11 Least Important Success and Failure Factors

Rank 220 219 218 217 216 215 214 213 212 211 210

Factor Laboratories for innovation, e.g. Honours Programmes Group training / reflective project leaders Present and ‘possible’ tools Teaching versus knowledge development by teachers Mobility of personnel, rejuvenation of personnel Stop talking about support and resistance Subject / goal fit the hype of the moment (in the year 2000: Eportfolio; in the year 2004: wireless?) Don’t get caught up in the innovation-mode (instead of embedding and evaluating, starting on a new adventure) Don’t let a project organisation turn into an exploitation organisation; it has short term benefits but long term drawbacks Discrepancy between now and the future and its relation to division of labour, culture, rules, and procedures Eliminate middle management

Both the sorting and the rating of the success and failure factors can be depicted in one graphical representation. Figure 11 shows a combined cluster rating and point-rating map with average cluster ratings and individual factor ratings superimposed. Besides a visualisation of the clustering and the ratings of the individual factors, the average

45


rating of the factors within each cluster is also given. In Figure 11 the cluster rating of each cluster is visualized by layers: the more layers a cluster has on the map, the higher the average factor rating for the cluster is. Further Figure 11 visualizes the ratings of the individual factors. Piles of little squares represent the size of the individual factor. High piles indicate high factor ratings. One can see that the clusters Participation, information and communication (Cluster ID: 5), Added value (Cluster ID: 8), and Stakeholder involvement (No. 6) have relatively high average factor ratings, while this is not the case for the clusters Project optimisation: Start small and then go for it! (Cluster ID: 4) and Position of innovation within the organisation (Cluster ID: 7). Point Legend Layer

Value

1

2,17 to 2,64

2

2,64 to 3,12

3

3,12 to 3,59

4

3,59 to 4,06

5

4,06 to 4,54

16 53 506 153

187

137 49 175

196 5 103 55 34 111

93 24 33 115

41

156

63 116

195 36

1 215 60 72

42

46

78

208

212

179

94

3,25 to 3,35

2

3,35 to 3,45

3

3,45 to 3,55

4

3,55 to 3,65

5

3,65 to 3,75

193

199

98 203

2 201

150

82 119 213

147

12

166 128 146 99 139

21 189

68 5

161 80

25 12 81 205 76 191 13 104 100 154 135 95 62

200 134 130 102

56

77

31 194

197

216

125 2

79 13

15

22

159 47

121

171

26

97

176 6 58 145 164 114 169 92 11 192 45 129 28 87 210 71

209

32 207

51

198 10 54167 23138 133

29

90

143

148

57

136 4 64

152

163

19

1

40

30

219 181 91

172

168

126

142 202

149

160 158 105

69

188

48 110 96 37

123 86 38178

183

122

173

3

106

Value

206

27

174

155

7 131

17

85 67 7

162

3

4

Cluster Legend

14

124

112

88

108 89 9184 127 165 117 113

120 177

186

151

66

65

190

218 220 70

44

118

84 20

74

10

1

Layer

182 35214 52 39 185 170 107 43 83 8 132 101 109 217 11

180 18 141 211

157 61

73 204

59 75 140 8

144

9

Figure 11. Cluster rating map visualising the average rating of the factors within each cluster.

Three variables should be taken into account in order to determine cluster quality: the cluster ratings, the cluster bridging value and the total of factors within a cluster. Table 6 provides an overview of the variables mentioned. In this table the clusters are ordered by cluster rating size.

46


Table 6. Mean for Average Cluster Rating, Mean and Rank for Average Cluster Bridging, Cluster ID and Number of Factors within Cluster

Cluster Participation, Information and Communication Added value Stakeholder involvement Quality of institutional management Project and process management Focus and complexity Competency of the project team and its members Relationship between project and real problem / core business Competency of the project manager Means to support the project Organisational culture Position of innovation within the organisation Project optimisation: Start small and then go for it!

Rating Mean 3.75 3.73 3.68 3.65 3.61 3.56 3.53 3.51

Bridging Mean Rank .32 7 .20 1 .37 9 .25 3 .28 4 .39 10 .49 13 .24 2

3.44 3.42 3.37 3.29 3.25

.39 .28 .34 .39 .30

10 4 8 10 6

ID 5 8 6 11 12 3 1 9

Factors N 19 24 9 17 28 12 17 11

2 13 10 7 4

15 19 18 19 12

Note: The cluster IDs correspond with the numbers before the factor descriptions in the result section.

Table 6 shows that the cluster Added value (Cluster ID: 8) can be regarded as a cluster of high quality. As stated earlier the rating of this cluster is high. Further the average bridging value is low which indicates that the cluster is cohesive, easy to interpret, and that it reflects the content well in that part of the map (see Figure 9 and 10). Finally the amount of factors classified within the cluster is high. A total of 24 factors represent the cluster, where a number of 17 factors is expected for each single cluster (220 divided by 13 is 16.92). Also interesting is that the two best-rated factors are located within the cluster Added value (see Table 4). Another remarkable cluster is Participation, information and communication (Cluster ID: 5). This cluster contains a large number of factors, has the highest cluster average and a cluster bridging value that is substantial. Finally, a salient cluster is Project and process management (Cluster ID: 12). A total of 28 factors are classified within this cluster. Also the cluster bridging value is relatively low and the cluster rating is relatively high.

Conclusions and Discussion The main goal of this study was to validate the failure and success factors of large-scale educational ICT innovation projects in Dutch higher education found in earlier literature research (see Chapter 1), and to identify additional determinants for success and failure of these projects. The identification of failure and success factors is considered necessary for future projects to make these factors manageable and to enable the achievement of better and more sustainable results. In comparison to the previous literature research in which 42 failure and success factors were identified, the expert concept mapping procedure resulted in 220 factors. By largely reproducing the 42 factors found in the literature study, and substantially extending the number of identified factors, both the validation and the identification goal of this study have been achieved. In addition, these results show that the experts in this study, all having more than 10 years of experience in the management of large-scale educational ICT projects, are

47


particularly well able to convey the ‘lessons learned’ from their own failed or successful projects. Whereas it is difficult for project managers to talk about projects that have not met expectations, the group concept mapping procedure can be considered an effective means to uncover the factors underlying project failure. The clusters generated contain statements that refer to either successful or failure factors and substantially enrich their operationalisation. The resulting factors can be related to the three stages in which innovations in general take place (Alexander & McKenzie, 1998; Fullan, 2001), namely the initiation phase, implementation phase, and the institutionalisation phase. In the initiation phase the present situation needs to be assessed in terms of goals, problems, and discrepancies. Then, ideas for the new situation need to be planned taking into consideration the changes envisioned and the route from the present to the new situation. Next, the consequences of the effects of the innovation on other aspects of the organisation need to be evaluated. In the implementation phase the innovation is developed, implemented, and evaluated. In the institutionalisation phase the innovation needs to be scaled and maintained. Throughout these three phases, the necessity of participation, information, and communication (i.e., dissemination, both formal and informal) is the key. Without it, failure; with it, better chances for success. This whole can be seen in Figure 12.

Initiation phase






disseminate

disseminate

disseminate


implement

maintain

disseminate

evaluate

Figure 12. Phasing of a project with sub-processes per phase.

Although all clusters identified in the group concept mapping study should be taken into account, some of them seem to better represent a particular phase. The applied method has a potential for depicting the relationship and semantic distance between the generated success and failure factors within the clusters, though there is no specific 1:1

48


relationship. It is more a question of where the emphasis lies. Clusters such as Competency of the project team and its members, Competency of the project manager, Project and process management, and Means to support the project fit the initiation phase. Clusters such as Focus and complexity, Project organisation: Start small and then go for it, Participation, information, communication, and Stakeholder involvement describe the implementation phase. The institutionalisation phase is defined by a set of clusters such as Position of innovation within the organisation, Added value, Relationship between project and real problem, Organisational culture, and Quality of institutional management. For example, in the initiation phase one should form a knowledgeable and competent team and choose a competent project manager. The necessary preconditions should be fulfilled and needed means should be provided. And unfortunately, this is where it often goes wrong. Organisational managers are chosen instead of project managers and the team is often filled with people who have the time to work on the project and not those people who are best suited to do the job. There is, namely, a major difference between being a competent project manager and a competent organisation manager since the life-cycle of these two are very different. The main purpose of organisational management is to plan, guide and help the organisation to work well and survive; in other words to achieve continuity. Project management, on the other hand, is to actively plan the end of a project; in other words to break with continuity and achieve change. A project must end in order to be successful whereas an organisation must not end in order to be successful (Turner, 1999). The difference between an organisational and project manager goes even further and starts with the initiation of a project. In contrast to organisational managers, project managers have to start their task from scratch, initiating and bringing an organisation to life. To start something in order to end it within a fixed period of time, according to a budget and according to a number of restrictions distinguishes the nature of project managers from the nature of organisational managers (Turner, 1999). And this difference between an organisational manager and a project manager is often forgotten. Neglecting this difference can lead to incompetent project management, both in the industrial organisations and in the educational organisation. According to the Standish Group (Johnson et al., 2001), 97% of the successful projects investigated in their research had an experienced project manager at the helm. In addition the content and the structure of project-activities should be carefully planned. The implementation phase requires an analysis of the complex project situation; execution of project activities in small steps; and involvement of users in the design and development of the project’s outcomes. The institutionalisation phase is aimed at providing conditions for a sustainable innovation. Some of the activities for accomplishing this goal are determining the position of innovation within the organisation; defining the pragmatic value of innovation for solving real problems in organisation; creating innovative organisational culture; and providing management support to the innovation initiatives on both strategic and operational levels. The visualisation of the clusters in a map format distinguishes three zones. The zone at the left corner (low) of the map (see Figure 10) includes the clusters that are more appropriate for the initiation phase. The zone in the middle contains the clusters that represent the initiation phase. The zone on the right side (high) of the map includes the clusters, which are most important in the institutionalisation phase. Of course this does

49


not account for all the variables but a certain pattern can be recognized. So the project phasing can be placed diagonally across the map. In contrast to the general phasing of innovations (e.g., Fullan, 2001), which describes success and failure factors in an isolated way, the group concept mapping procedure determines not only the time based connection (sequence) between the different phases, but also the conceptual relationships between them. The meaning of each statement is defined by its relationships to other statements in the same cluster. The analysis of the statements in the clusters representative for the initiation and institutionalisation phase send a clear message: Take into account institutionalisation issues of innovation from the very beginning of the project (see Chapter 1). This can, for example be seen in the ‘cascade’ in Figure 13. Plan the innovation

Backcast to today

Thought experiment

Where in 10 years?

Vision at the top


Begin

Figure 13. Keeping implementation in mind during the initiation phase.

Most of the clusters deal with people issues that were identified as the most important factor to cope with from the literature study supporting this project. They are also identified as a trend in the design methodologies of different professional domains such as software design (Arlow & Neustadt, 2001; Carroll, 2000; Constantine, 2001; Constantine & Lockwood, 1999), strategic management (Eden & Ackermann, 2002; Van der Heijden, 1998; Vennix, 1997), and engineering (Cross, 2000). The Forester research agency (2002), one of the biggest internet research agencies reported that corporate America spends more than 275 billion each year on approximately 200 000 application software projects. Most of them failed. Technology was neither the problem, nor the solution. The people factor was the most crucial factor for the failure or success of these projects. They failed because they simply did not address the need of users. The clusters Participation, information and communication and Stakeholder involvement, which are people like clusters, score very high on the cluster ratings, indicating that the factors which represent the clusters are regarded as very important by

50


the experts. Especially the Participation, information and communication cluster is imperative since other cluster quality measures like the number of factors that define the cluster and the average bridging value are also good (see Table 6). A direct practical implication of this empirical evidence is the requirement for involving users (clients) right from the beginning of the project in the analysis, design, development, evaluation, and implementation of the innovative outcomes, practical arrangements known as participatory design (e.g., Kensing & Blomberg, 1998). Kirschner (2002; Kirschner, Martens, & Strijbos, 2004) emphasise the use of interaction design that does exactly this (see Figure 14). Learner / user experience

What do learners actually do? What do learners want to do?

Support / affordances

How can we support what they do? What affordances are needed?

Constraints / conventions

How does the learner perceive the support?

What are the physical, logical and cultural limitations encountered?

Learner / user perceptions

How does the learner actually use the support?

What has the learner / learning group actually achieved?

Learner / user experience

Learning

Figure 14. Interaction design for involving the user.

The cluster Added value can probably regarded as the most important single cluster. The cluster bridging value is very low –indicating a coherent cluster-, a substantial number of factors define the cluster, and the cluster rating is very high (see Table 6). Interesting factors that represent this cluster are Make the added-value visible and Make the benefit of a product clear (new is not necessarily better), the two best rated factors in this study (see Table 4). A conclusion that can be drawn from these results is that to prevent failure there should be a clear reason underlying the educational innovation project that justifies educational change (e.g., De Bie, 2003, Fullan, 2001). As a next step of this research project, the project managers of unsuccessful and successful projects will be interviewed to determine how they have dealt with the failure and success factors that were found in the literature study and this study, and how they rate these factors. Based on the triangulation of the data from the literature study, the concept mapping study, and the interview study a so-called ‘project health checklist’ will be developed that can be used by funding organisations in the assessment of project proposals and reports and by applicants in the writing of project proposals and to monitor and manage the factors. It should be noted that although these variables are identified as major contributors to project success, they will never guarantee success

51


alone. We hope that the translation of the current results into guidelines and checklists will enable funding organisations and applicants to initiate and run more successful projects with more sustainable results. An additional follow up activity of this research project is an analysis of the discussion of the experts, which was one of the steps of the group concept mapping procedure. For this purpose we are going to apply a cognitive mapping procedure (Eden & Ackermann, 2002). The discussion script will be transformed into a set of statements. A cognitive map represents statements and their relationships in the form of a causal or an implication network (map) of argumentation, which is build in a hierarchical manner. The technique determines not only what each statement means (through its content and context), but also why they fit together as they do. The technique applies a particular convention of coding as each statement (construct) is linked to others through use of arrows indicating ‘may lead to’, ‘has implication for’, or ‘supports’ moving up the arrow; or ‘may be explained by’, ‘is implied by’, or ‘is supported by’ when moving down the arrow. Decision Explorer (Banxia Software, 2003), the software that supports cognitive mapping, provides opportunity for doing different types of analysis: cluster analysis, domain analysis, potent analysis, and centrality to list but a few. It can also identify feedback loops and dilemmas. Finally, we are aware of the fact that there is much more information in the present data than we have been able to present here. However, we believe that interested readers are well able to draw their own conclusions from the data.

52


Chapter 3: Interview and Project Documentation Study

53


Abstract This Chapter presents the results from interviews with the project chairs of eight SURF Education projects that were selected from the SURF Education projects (tenders 1999, 2000, and 2001) and classified as either highly successful or less successful. In addition, results are presented from a study of the documentation of these projects. This part of the study started with the identification of meaningful combinations in the form of clusters on the basis of the 42 success and failure factors that were found in the literature study. Second, eight projects were selected by experts from SURF Education and globally described and analysed based on their project documents and reviews by SURF committees. Third, the project chairs of the selected projects were interviewed to find out how they have dealt with the identified success and failure factors, and they were asked to rate these factors. Fourth, the results and analyses from the interview and document studies were compared and discussed. Finally, the different results are synthesized in an overall conclusion.

Introduction This Chapter deals with the factors that have played a role in innovation projects funded by SURF Education. As such it presents the results from interviews with the project chairs of eight completed SURF Education projects selected from the tenders of 1999, 2000, and 2001 and classified as either highly successful or less successful. In addition, results are presented from a detailed study of the documentation of these projects. This part of the study began with the identification of meaningful combinations of clusters based on the 42 success and failure factors found in the literature study. After this, eight projects were selected by experts from SURF Education. These projects were described and analysed based on their project documents and reviews by SURF committees. Third, the project chairs of the selected projects were interviewed to find out how they have dealt with the identified success and failure factors, and were asked to rate these factors. Fourth, the results and analyses from the interview and document studies were compared and discussed. Finally, the different results are synthesized in an overall conclusion. Describing and clustering factors The literature study (Chapter 1) showed that innovation projects are carried out in phases, which can globally be described as initiation, implementation, and institutionalisation. In the initiation phase the present situation needs to be assessed in terms of goals, problems, and discrepancies between them. Then, ideas for the new situation need to be planned taking into consideration the changes envisioned and the route from the present to the new situation. Next, the consequences of the effects of the innovation on other aspects of the organisation need to be evaluated. In the implementation phase the innovation is developed, implemented, and evaluated. In the institutionalisation phase the innovation needs to be scaled and maintained. The literature review resulted in 42 failure and success factors across these phases. In preparation of the present interview and project documentation study and the group concept mapping procedure (see Chapter 2), the 42 factors were classified into meaningful clusters by three experts. The experts - all experienced educational technologists at the Open University of the Netherlands – independently classified the 42 factors into clusters before deciding in consultation on the final clusters and their labels. This resulted in 13 clusters, which are presented in Table 7 (see also Table 2). Because sustainability of innovation is considered a major goal by SURF, it was added

55


to the clusters as the 14th cluster. These 14 clusters, which are presented in Table 7, constituted the basis for the interview scheme and the detailed study of the project documents (i.e., project plans, reviews, evaluations controlling documents and final reports). Table 7. Fourteen Clusters for the Interview and Project Documentation Study

1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14.

Added value, use, necessity of the innovation Commitment / engagement Communication Delineation of project content Delineation of project process Expectations of the involved parties Organisational management Organisational typology Project culture Project chair qualities Project management Project staff qualities Scale / complexity Sustainability7

Project classification After clustering the factors three experts from SURF were asked to classify the 20 projects from the tenders 1999, 2000 and 2001 (see Appendix I) into three categories: highly successful projects, successful projects and least successful projects. The experts received a selection tool via e-mail that they could use to categorise the projects (see Figure 15). This tool allowed them to ‘drag’ the labels with the abbreviated names of the projects into one of the three categories based on the following criteria reported by the experts: achievement of the project goals, dissemination, long-term use of the innovation, transferability of the innovation, lessons learned, and the innovative character of the innovation. The results from the experts were then combined and analysed. The four projects that were most consistently categorised in the ‘highly successful projects’ category, and the four projects that were most consistently categorised in the ‘least successful projects’ category, formed the basis for the interviews and document analysis. This ‘extreme group’ approach was expected to best reflect the clusters that contribute to success or failure.

7

Extra cluster that was added after the clusters had been defined by the experts.

56


Figure 15. Classification tool: point of departure for classification.

Figure 16. Classification tool: a conceivable classification result. Project names are replaced.

Interview study The project chairs of the eight selected projects were interviewed based on the 14 clusters of failure and success factors extracted. The interviews, which took about 45 minutes per project chair, covered the following subjects (the interview protocol can be found in Appendix V): general questions concerning former project management experiences of the project chairs, their view on the success (or failure) of this project, the lessons learned from this project, their reflection on their success according to themselves, according to SURF and according to the end users, the failure and success

57


factors of their project, sustainable implementation and the role that the 14 clusters have played in this project. The interviews were recorded for later transcription and analysis. A written report of the interview (to be checked by the project chairs who were interviewed8) was sent to each of the eight project chairs to control for errors (see Appendix VI for the approved reports of the interviews). Along with this, we also sent the project chairs a questionnaire with the 14 clusters and asked them to rate the clusters based on their experiences within their own project. They were asked to rate the different clusters to the extent to which the factors within the clusters (1) were taken into account in their own project, (2) had influenced their own project with respect to success or failure and (3) were considered important by them for successful innovation projects in general. The remarks of the project chairs in the interviews revealed a number of success and failure factors, successful procedures of SURF, and points for SURF to improve and which endorse the factors found in the literature study. Recurring factors that contribute to the success of their innovation projects mentioned by the project chairs are (in random order): • Necessity of the innovation: Is there a real problem that needs to be solved? • Results of the innovations should concur with the needs of the (end)users: Have the stakeholders really been consulted? • Commitment of the (end)users to the innovation: In what way has the commitment of the (end)users been assured? • Commitment of the management to the innovation: In what way has the management shown/proven its commitment? • Scalability, the project should not be too complex: Do we realise that even apparently small changes in one area often leads to a chain of changes in the rest of the organisation? • Capacity of the organisation: Organisations become innovation weary. Can the organisation stand another revolution? • Practicability of the project: In the end, are we / will we be asking realistic things of all of the stakeholders? • Communication: What structures are there, i.e., are we setting up to communicate with all of those who are / will be involved in the innovation and its later implementation? The project chairs were generally very positive with respect to a number of (successful) procedures of SURF: • Good support and counselling from SURF regarding process and content problems • Flexibility of SURF and SURF Education management • Commitment of SURF and SURF Education management Be that as it may, the chairs also listed a number of points for improvement for SURF: • Intake at the beginning of the project: A paper trail is not enough. The intake procedure should be refined to assure that good projects can be carried out and ‘bad’ ones are rejected.

8

The individual reports on the interviews that were accepted by the chairs can be found in Appendix VI

58


• •

Review of the content by experts: Instead of, or in addition to the strong focus on the procedures in the review process, the projects chairs would ver much appreciate a review of the project content by experts A more compact style of reporting on the processes, this sometimes takes too much time and is required too often

The interviews showed that, in general, the project chairs of the highly successful project were more critical towards themselves, their own performance and management compared to the project chairs of the less successful projects who mostly ascribed the causes of the mistakes, failures, and problems in their projects outside their own range of influence. Furthermore, the project chairs of the less successful projects used the interviews more to vent their spleen and ‘evaluate’ the processes, the project, and the problems. All of the project chairs would certainly do the project over again with their present knowledge if given the chance and the opportunity to change things in advance or in the process of the project. In addition, the project chairs of the highly successful projects have a clear list of aspects to change if they were given the opportunity to do the project over again. Their list of changes is longer, more profound and aimed at ensuring more commitment from all the parties involved through a sort of intake considering the readiness, willingness and capacity to change, they would like more reviews and feedback concerning the content by (external) content experts, and more responsibilities for the project staff and project chair. The list of proposed changes by the project chairs of less successful projects is shorter, simpler and more aimed at single points of improvement such as communication and scalability. This raises the following question: Is this difference between the two change lists of the two different groups of project chairs a result of the experiences in the project or is it a character trait of the project chair which influenced the success of the project? Detailed results from the interviews: The results from the rating questionnaires were analysed. The means (M) and standard deviations (SD) of the clusters rated by the project chairs for innovation projects in general (M Gen), to the extent to which the clusters were taken into account in their own projects (M Account) and to the extent to which they had influenced the projects success or failure (M Influence) are summed up in Table 8.

59


Table 8. Means and Standard Deviations of the Ratings of the Clusters by the Project Chairs. Note that the 3 lowest and highest Results per category are printed in italics and bold, respectively

Delineation of project content Project staff qualities Project chair qualities Project management Communication Organisation management Organisational typology Project culture Added value, use, necessity Expectations involved parties Commitment / engagement Sustainable implementation Scale / complexity Delineation of project process

M Gen 4,63 4,50 4,50 4.38 4.38 4.38 4.38 4.25 4.25 4.25 4.25 3.88 3.87 3.75

SD .52 .54 .93 .74 .52 1.06 1.06 .71 .89 .89 1.39 1.55 1.13 1.28

M Account 4.63 3.00 3.88 4.00 3.50 3.00 3.00 3.50 4.63 3.00 3.00 3.63 2.50 3.75

SD .52 1.20 .84 1.07 1.20 1.07 1.07 1.20 .52 1.31 1.69 1.19 1.41 1.28

M Influence 4.25 4.25 4.00 4.38 4.00 3.38 3.38 4.00 3.88 3.88 3.75 3.25 4.13 3.38

SD .46 .71 .76 .74 .76 1.51 1.51 .76 1.13 .99 1.28 1.39 1.13 1.06

It can be concluded that the project chairs in the rating procedure value the clusters ‘Delineation of project content’, ‘Project staff qualities’, ‘Project chair qualities’, ‘Project management’ and the ‘Added value/use or necessity of the project or innovation’ as important determinants for success or failure. This confirms the results from the literature study, the results from the interviews, and the results from the expert concept mapping procedure. A solid project plan is the basis of a good innovation project and is therefore rated throughout the interviews and ratings as a very important factor. Good thinking, planning, and analysing the project beforehand, together with taking the expectations and outcomes of the innovation project into account result in a well defined project plan and should be a first step in a innovation project according to the project chairs. The clusters considered least influential for the success or failure of innovation projects are the sustainability of the project, the scalability/complexity of the project and innovation, the organisation management, the organisational typology and the delineation of the processes of the project. Scalability, Sustainability, and Organisation present a different picture. The complexity or scalability, although rated low, was mentioned during the interviews as a cluster of influence and one of the things project chairs would like to alter when doing the project again. The low rating of the sustainability of the project is, considering the innovative, and therefore uncertain, nature of the project understandable but it should be an important factor in innovation projects. The organisation and typology of the management was not taken into great account and rated of little influence within the projects but is considered as a factor of influence in innovation projects in general. A tentative conclusion can be drawn here. Each of these three topics (Scalability, Sustainability, and Organisation) can be regarded as topics that the project staff might consider to be outside of the “scope” or “sphere of influence” of the project (in a strict sense). This concurs with one of the conclusions of the literature study (i.e., the thought experiment) that projects and management need to seriously consider implementation – and what an organisation must do to implement the results when the

60


project ends – at the earliest point in the planning process. If this is not done, successful innovation projects can easily become failed innovations. Because the project chairs did not see a difference between the clusters organisation management and organisational typology, in the rest of the analysis of the interview results, they are considered as one and the same cluster of factors. A closer look at Table 8 reveals a number of substantial discrepancies (more than one full point) between the different means (see Table 9). Most striking is that many of the clusters considered to be the least important are those dealing with implementation (scalability, sustainability organisational aspects, typology and management). Table 9. Discrepancies (of More than One Full Point) Between the Different Means

Delineation of project content Project staff qualities Project chair qualities Project management Communication Organisation management Organisational typology Project culture Added value, use, necessity Expectations involved parties Commitment / engagement Scale / complexity Sustainable implementation

M Gen 4,63 4,50 4,50 4.38 4.38 4.38 4.25 4.25 4.25 4.25 3.88 3.87 3.75

SD .52 .54 .93 .74 .52 1.06 .71 .89 .89 1.39 1.55 1.13 1.28

M Account 4.63 3.00 3.88 4.00 3.50 3.00 3.50 4.63 3.00 3.00 3.63 2.50 3.75

SD .52 1.20 .84 1.07 1.20 1.07 1.20 .52 1.31 1.69 1.19 1.41 1.28

M Influence 4.25 4.25 4.00 4.38 4.00 3.38 4.00 3.88 3.88 3.75 3.25 4.13 3.38

SD .46 .71 .76 .74 .76 1.51 .76 1.13 .99 1.28 1.39 1.13 1.06

The project chairs rate the competence of the project staff as an important general factor, which is of high influence on the success or failure of the project, but, which is taken much less into account9. There was also a discrepancy of more than one full point between its rating as general factor and the rating of the extent to which it was taken into account in the project. This could point to the fact that the project chairs feel that a serious choice of project staffing does not always occur and that it is more a result of who has room in her/his schedule at a given moment and (s)he then becomes a part of the team. Another discrepancy of more than one full point was found between the rating of the extent to which the project was influenced by this factor and the extent it was taken into account in the project. The same discrepancies can be found for the cluster complexity or scalability of the project. While this cluster is considered of general importance to a project and of specific influence to a project’s success, the project chairs report that it really wasn’t taken into account (see the earlier tentative conclusion). Three other clusters were also rated more than one point higher on the importance level as a general factor than the extent to which whey were taken into account. This points to a general shortcoming with respect to the projects carried out, namely that while those in charge of running a project feel that a number of factor clusters are important, they have problems taking them into account while carrying out the projects. 9

The value of 1standard deviation (SD) was chosen based upon the grand average of all of the SDs.

61


We will now take a closer look at the differences in ratings between the project chairs of highly successful projects and the project chairs of less successful project with regard to the extent to which the factors were taken into account by the project chairs. The clusters individually Delineation of project content

M Gen 4.63

SD .52

M Account 4.63

SD .52

M Influence 4.25

SD .46

A good delineation of the content of the project was rated by most of the project chairs as having the highest importance for innovation projects in general. Project delineation was also rated the highest (together with the cluster Added value, use, necessity) with regard to the extent to which the cluster was taken into account by the project chairs within their own projects and is considered of great influence on the success of their project. These results are endorsed by the interviews, literature study and results from the concept mapping procedure. A critical remark on this cluster of factors was made by several project chairs: a clear and well defined and delineated project plan is important, but a project plan must be flexible and there must be room for adjustments and changes according to, for example, a mid-term needs assessment of the (end) users or the results from a mid –term review executed by SURF or external experts. This flexibility and changeability should be taken into account when setting up the project plan. Project staff qualities

M Gen 4.50

SD .54

M Account 3.00

SD 1.20

M Influence 4.25

SD .71

The competence of project staff played an important role and was of great influence on the success of the project in question. It was also a cluster of factors that should be of great influence in innovation projects in general. However, according to the project chairs, the competence of the project staff was not a factor they bore greatly in mind in their project although it played an important role in the success or failure of their project. Most of the time, the staffing of the project team was not the result of a long and thorough selection procedure, but more one of sheer luck. However, the project chairs indicated that their project teams were fairly competent and when there was a lack of expertise, this mostly happened with the team of the partnering organisation. Project chair qualities

M Gen 4.50

SD .93

M Account 3.88

SD .84

M Influence 4.00

SD .76

A good and competent project chair was rated as an important cluster for innovation projects in general and of medium influence on the project’s success or failure and was moderately taken into account when starting the project. The role of a good project chair consisted for example of communicating with everybody, motivating, stimulating and taking care of the processes. Apparently, management seems to have taken the quality of the project chair into account (or more seriously) than the quality of the project staff. Furthermore it was noted by one of the project chairs that project chairs should have more ‘power’ and influence to control the process and take action themselves. Project management

M Gen 4.38

SD .74

M Account 4.00

SD 1.07

M Influence 4.38

The project management was taken into great account and was rated as the most influential cluster on the project success by all project chairs. A project should be

62

SD .74


managed by a competent project chair accompanied by a competent group of team members. They should take care of the project in all its facets and make sure the project works out well and follows its own plan but also consider the possible changing needs of the users, changes in the organisation and project. Communication

M Gen 4.38

SD .52

M Account 3.50

SD 1.20

M Influence 4.00

SD .76

Communication was rated over all of the different aspects (projects in general, influence on own project, taken into account in own project) as a fairly important cluster. The interviews show that this aspect, although fairly important, has been neglected in some cases. This factor was one of the aspects the project chairs would like to change when given the opportunity to do the project all over again (see the proposed phasing in Figure 7). On the other hand, some projects had a well-defined communication plan that was followed during the project. Formal and informal communication on different levels (e.g., at organisational level, within the project, between subdivisions of the project, to SURF, to other interested parties, etc) plays an important role in innovation projects, this can help to keep the participants and parties involved up-to-date, informed and involved, and is especially of importance in case of problems or delays. It also allows for feedback to the project team, thus avoiding isolation and possible groupthink. Organisation management

M Gen 4.38

SD 1.06

M Account 3.00

SD 1.07

M Influence 3.38

SD 1.51

Although the interviews showed that the organisation management (or the organisational typology) played an important role and should be committed to the project, the ratings showed that the role of the organisation management was only moderately taken into account within the projects and moreover it did not play an important role. Several project chairs mentioned the importance of a committed and supporting organisation management in the interviews. The organisation management should provide the right basis for an innovation project and change should be made possible, promoted and encouraged. Another role for the organisational management should be the informing of several parties and assuring that there is enough capacity for the project and innovation. Project culture

M Gen 4.25

SD .71

M Account 3.50

SD 1.20

M Influence 4.00

SD .76

The project culture was in most of the projects open, flexible and critical. It was rated as a fairly important cluster, moderately taken into account and of relatively great influence. Added value, use, necessity

M Gen 4.25

SD .89

M Account 4.63

SD .52

M Influence 3.88

SD 1.13

A project or innovation should have a clear and obvious added value. This cluster was rated as the cluster taken into the highest account (together with the cluster Delineation of project content) within the projects and is an important factor for innovation projects in general according to the project chairs. A clear added value contributes to the commitment, support and involvement of the different parties involved. The necessity of the project and innovation should be obvious and closely connected to a problem, needs or expectations of the (end)users.

63


Expectations involved parties

M Gen 4.25

SD .89

M Account 3.00

SD 1.31

M Influence 3.88

SD .99

The expectations of especially the end users were important and should be, according to the interviews and to a lesser degree according to the results of the rating, taken into account. Commitment / engagement

M Gen 4.25

SD 1.39

M Account 3.00

SD 1.69

M Influence 3.75

SD 1.28

Commitment and / or engagement to the project and innovation was in the interviews as well as in the ratings valued as an important cluster. Commitment, engagement and support from different sides like management, organisation and users is considered important for the success of a project or innovation. Sustainable implementation

M Gen 3.88

SD 1.55

M Account 3.63

SD 1.19

M Influence 3.25

SD 1.39

Although the literature rates the sustainability of an innovation project as a very important cluster, the interviews and ratings show mixed results. Sustainability was rated as the least important cluster regarding its influence on the success or failure of the project and was valued as a relatively less important cluster in general. Taking sustainability of an innovation into account from the start of a project was experienced as difficult since the ongoing use of an innovation, thus the sustainable implementation, depends on numerous other factors, such as the success of the implementation of the innovation, the linking up of the innovation to the needs of users and the clear necessity of the innovation. Apparently sustainability is not seen as critical to the innovation project itself, but rather as something to be taken care of “after” the project, or as the Dutch say “Those who are still alive, are those who should take care of it”. Scalability, complexity

M Gen 3.87

SD 1.13

M Account 2.50

SD 1.41

M Influence 4.13

SD 1.13

The complexity or scalability was rated as a cluster of substantial influence, but was rated fairly low as a general cluster and moreover, it was rated the lowest with regard to the extent to which it was taken into account. Surprisingly, or maybe not, it was mentioned as one of the factors that most of the project chairs would like to change and would like to take into greater account when given the chance to do the project all over. Some of the projects were considered by the project chairs as over-ambitious (too complex and not appropriately scaled) given these circumstances. As seen in the literature study (Chapter 1), complexity is influenced by numerous other aspects, such as technology, environmental influences, size and structure of the organisation and interdependence between these different aspects. Although, the rating suggests that it is not considered as a very important factor in innovation projects in general, it was mentioned as a failure or risk factor more than once. Delineation of project process

M Gen 3.75

SD 1.28

M Account 3.75

SD 1.28

M Influence 3.38

SD 1.06

The delineation of the processes of the project was a very important cluster according to the interviews, but was rated as the least important general cluster. The planning of the project should be clear and straightforward with a range for flexibility built-in.

64


Comparisons of highly successful projects versus less successful projects When comparing the ratings of the clusters by project chairs of the highly successful projects to the ratings of the clusters by the project chairs of the less successful projects, some interesting differences become visible. The rating of the clusters as important for innovation projects in general are fairly equal divided over the ratings of the four project chairs per condition (highly successful versus less successful) (see Table 10). The final two columns in the table show whether this cluster was rated higher by the project chairs of the highly successful project (+) or by the project chairs of the less successful projects (-) or that it was rated equally important by both and the differences between them (∆). Table 10. Comparison of the Rated Importance of the Clusters in Innovation Projects in General by the Project Chairs of Highly Successful Projects versus Less Successful Projects

Delineation of project content Delineation of project process Project chair qualities Project staff qualities Organisation management Communication Commitment / engagement Project management Expectations of parties involved Added value, use, necessity Project culture Scale, complexity Sustainable implementation

Highly Successful M 4.75 3.75 4.50 4.50 4.00 4.50 4.00 4.50 4.00 4.50 4.25 4.00 3.75

Less Successful M 4.50 3.75 4.50 4.50 4.75 4.25 4.50 4.25 4.50 4.00 4.25 3.75 4.00

+/0/-

∆

+ 0 0 0 + 0 + 0 + -

0.25

0.75 0.25 0.25 0.50 0.50 0.25 0.25

To sum up, 5 clusters were rated higher on the importance scale for innovation projects in general by the project chairs of highly successful projects, 4 clusters were rated higher by the project chairs of the less successful projects and 4 clusters were rated equally high by the two different groups of project chairs. For none of the clusters there was a difference of more than 1 point. Table 11 shows the results of the comparison of the clusters with regard to the extent to which the clusters were taken into account by both the project chairs of the highly successful projects and the project chairs of the less successful projects. Again, the last column shows whether the cluster was rated higher by the project chairs of the highly successful projects or by the project chairs of the less successful projects.

65


Table 11. Comparison of the Ratings of the Extent to Which the Clusters Were Taken into Account According to the Two Groups of Project Chairs

Delineation of project content Delineation of project process Project chair qualities Project staff qualities Organisation management Communication Commitment, engagement Project management Expectations of parties involved Added value, use, necessity Project culture Scale, complexity Sustainable implementation



+/0/-

∆

+ + + 0 ++ ++ 0 ++ + ++ ++ -

0.75 0.50 0.75 0.50 1.00 1.00 1.00 0.75 1.00 1.00 0.25

Table 11 shows that two clusters were rated higher by the project chairs of the less successful projects and two clusters were rated equally high. The table furthermore shows that nine of the thirteen clusters were rated higher by the project chairs of the highly successful projects regarding the extent to which these clusters were taken into account in their projects than by the project chairs of the less successful projects. Five of these nine clusters were even rated with a discrepancy of more than one full point (indicated with ++ in the last column). Combining these results with the results from Table 9 (Discrepancies between different means) reveals that three clusters have both discrepancies of more than one full point on both of the aspects (between the different means and between the highly successful projects and less successful projects). The combined results from the above-described tables are presented in Table 12. Table 12. Combining the Discrepancies between Means and Highly Successful Projects and Least Successful Projects

M Gen Commitment, 4.25 engagement Expectations involved 4.25 parties Scale, 3.87 complexity

M Account

Highly Successful M

Less Successful M

3.00

M Influence 3.75

3.50

2.50

3.75

2.25

3.00

2.00

3.00

3.88

2.50

4.13

In the first two columns with means, the results of Table 9 are shown: the means rated by the whole group of project chairs together for these three clusters as important clusters in general and to the extent to which these clusters were taken into account by the project chairs. In the following two columns, the degree to which the clusters were taken into account were split to reflect the differences in the ratings of these clusters by the project chairs of the most successful versus the least successful projects (extracted

66


from Table 11). Finally, the last column (of Table 12) shows the means of the whole group of project chairs again to the extent to which the clusters had an influence on the project (extracted from Table 3). The discrepancy between the general importance and whether the factor was taken into account in the less successful projects approaches the two points difference (the differences as respectively 1.75, 2.00 and 1.87 for the clusters commitment / engagement, expectations and scale / complexity). The discrepancy between the ratings by the whole group of project chairs regarding the influence of the cluster scale and complexity on their project (4.13) and the ratings of the project chairs of less successful projects with regard to the extent to which the cluster was taken into account in their project (2.00) was over 2 full points: 2.13. A tentative conclusion that can be drawn here is that these three aspects of innovation and innovation projects might be considered watershed aspects (discriminating clusters of factors). Table 13 sums up the ratings of the influence of the clusters on the projects. Table 13. Comparison of the Rating of the Extent to Which the Clusters Had Influence on the Success or Failure of The Projects According to The Two Groups of Project Chairs

Delineation of project content Delineation of project process Project chair qualities Project staff qualities Organisation management Communication Commitment, engagement Project management Expectations of parties involved Added value, use, necessity Project culture Scale, complexity Sustainable implementation



+/0/-

∆

+ + 0 + + 0 + + + + -

0.50 0.25 0.50 0.25 0.50 0.25 0.25 0.25 0.50 0.25 0.50

Eight clusters were rated higher by the project chairs of the highly successful projects regarding their influence on their project’s success or failure, three clusters were rated higher by the project chairs of the less successful projects and two clusters were rated equally high by both groups of project chairs. None differed by more than 1 point. Combining the results from the last three tables the following conclusions can be drawn (see Table 14): • Four clusters (‘Delineation of project content’, ‘Communication’, Added value, and ‘Scale, complexity’) were always rated higher by the project chairs of highly successful projects. • Six clusters (‘Delineation of project content’, ‘Delineation of project process’, ‘Communication’, ‘Expectations of the involved parties’, ‘Added value, use, necessity’ and the ‘Scalability, complexity’) were rated higher by the project chairs of the highly successful projects with regard to the extent to which they were taken into account in their project and had an influence on the success or failure of their projects.

67


•

Two clusters (‘Organisation management’ and ‘Sustainable implementation’) were always rated higher by the project chairs of less successful projects. • Five clusters (‘Communication’, ‘Commitment, engagement’, ‘Expectations of parties involved’, ‘Project culture’ and ‘Scale, complexity’) were significantly more taken into account by the project chairs of highly successful projects. These clusters of factors can be tentatively described as distinguishing clusters between highly successful and less successful projects. Table 14. Summary of the Comparison of the Ratings

Clusters Delineation of project content Communication Added value, use, necessity Scale, complexity Delineation of project process Expectations of parties involved Project management Project chair qualities Project staff qualities Commitment, engagement Project culture Organisation management Sustainable implementation

General + + + + 0 + 0 0 0 -

Account Influence + + ++ + + + ++ + + + ++ + 0 + + 0 0 + ++ 0 ++ -

Project documentation study The second part of this project study consisted of a detailed study of the documentation of the eight selected projects. The project documentation delivered by the project chairs and the reviews were analysed, compared, and will be discussed in this last part of this chapter. The project documentation delivered by the project chairs was analysed based on the questions whether and to what extent they paid explicit attention to the 14 clusters used in the interviews and rating questionnaires. Furthermore the documentation was scanned for the lessons learned and success and failure factors mentioned by the project chairs and whether the problems mentioned during the interviews could also be found in this documentation. Next, the reviews and review procedures were analysed on differences between procedures that could indicate whether the project would turn out to be either highly successful or less successful. Furthermore, the results from the reviews were analysed, compared and discussed. The project documentation delivered by the project chairs consists of a number of different types of documents. It starts with a project proposal and a project plan, followed by controlling documents and (mid)term reports, a self evaluation, and an endreport. During the project, these documents are monitored by daily and more informal project controller, the Platform Manager (appointed by SURF) and several formal controlling committees10 appointed by SURF. These committees and the Platform 10

Committees such as Committee Project Control (formerly known as the SOS), a formal project controlling committee which assesses the intake, mid term and end reviews (Commissie Project Bewaking), consisting of

68


Manager monitor the project's progress, the documents delivered by the project chairs, they review the project and they advise SURF in granting the funding to the project. Since the method of documentation and the methods of the committees have changed slightly over the years (from tender 1999 – 2001) the documents could not be analysed and compared equally. A closer look at the project documentation delivered by the project chairs shows that although not all projects have a solid communication plan, communication did play a role in most of the projects. Unfortunately the reviews did not always pay enough attention to this deficiency. Furthermore, because of the extensive reporting required by the procedures outlined by SURF, the reports sometimes show substantial overlap. A useful part of the documentation delivered by the project chairs is the ‘Lessons learned’ section. In this section the project chairs evaluate and analyse their projects and extract their lessons learned. The following relevant lessons were mentioned by the project chairs: • a committed (financial, human resource capacity, time and resources) management is critical • management not only needs to be committed, but must also capable of delivering the needed resources • management should be a change-management with a competent change agent. This echoes the finding in the literature study that there is a world of difference between project management and organisational management and that this must be taken into account when staffing a project. • stability of the staff is important; it is especially preferred not to change the project chairs halfway a project Furthermore, the project chairs identified the following success factors in their projects: • devoted (end) users • sufficient support for users • flexibility throughout the whole project concerning project planning, goals, human resources, etc. • simplicity and clarity of the product • embedding of the project and innovation in the organisation and policy of the organisation • stability of the used technology and tool • adequate (technical) infrastructure • commitment among users, organisation and management • use of standard software and technology that can be used for a longer period of time and can be easily transformed into a new format / software / technology The review documents were also analysed for discrepancies between the review procedures and reactions of the controlling committees on the delivered documents of the highly successful projects and the less successful projects. There were no apparent differences between the results of the review procedures of the highly successful and less successful projects in the documentation. The less successful projects as well as the highly successful projects were both equally thoroughly reviewed and criticized. Both 12 expert members and a Review Committee consisting of one SURF member and one member of the Committee Project Control.

69


the highly successful and less successful projects had their setbacks, drop-outs and problems, which were reported in the various documents. According to the document study, no differences or determinants for failure or success can be found in the procedures, reviews, support, or counselling by SURF. In other words, the documents and their evaluations do not discriminate between highly successful and less successful projects. All the projects were finally completed and approved and most of the goals were more or less realised. Even ‘doubtful’ projects throughout the course of events of the project could turn out to be highly successful projects in the end. One particular project that turned out to be highly successful had been rejected in previous tenders as a result of its (too) high ambition. The revised version however was accepted by the Scientific Technical Council (in Dutch Wetenschappelijk Technische Raad). Regarded in retrospect, the project chair of this particular project still thinks the project was too ambitious and complex even when it turned out to be highly successful after all. The lack of influence of the project management on the participation and effort of some of the parties hampered the process and resulted in turning down a mid-term review of one of the selected projects. These problems can be attributed to a number of factors, such as the complexity and scalability of the project, the overambitious goals, and lack of commitment from some participating parties. There was too little management capacity with too little influence and effect on the answering of the agreed upon project deployment according to the project managers remarks during the interview. On the other hand, the intrinsic motivation and huge commitment from other parties in this particular project was substantial and contributed to the success of the project (or at least the successes of parts of the project). The project was in the end closed and approved of by SURF and considered a success. Another highly successful project also suffered from some shortcomings: the educational evaluation of the report of the project was somewhat underexposed and the knowledge dissemination to the various target groups could have received more attention. But these points should have been taken into account by the Platform ICT and Education of SURF and not by the project chair according to the review report. Problems such as financial shortages, changes of a project chair, and setbacks concerning for example the use and availability of the technological source and commitment of the end users and an advice to discontinue the funding of a project, were the problems some less successful project encountered. But these projects too managed to finish the project, with the goals fulfilled to some extent and to the more or less satisfaction of SURF. Apparently, though the documents were complete and the reviews were thorough, they were/are not sufficient to signal approaching success or failure.

To conclude The success and failure factors found in the literature are also encountered in the interviews, the ratings and the documentation study of the eight selected projects from the 1999, 2000 and 2001 SURF tenders and were all rated as important determinants for failure or success. It can be concluded that, generally speaking, a well-defined project that takes the expectation of all of the stakeholders into account and that is aimed at an obvious necessity or problem (i.e., has definite added value), that ensures formal and informal

70


communication and which is not too complex will potentially be a highly successful project. Beyond this general conclusion, we see that five clusters of factors (‘Communication’, ‘Commitment, engagement’, ‘Expectations of parties involved’, ‘Project culture’ and ‘Scale, complexity’) were taken ‘significantly’11 more often into account by the project chairs of the highly successful project and thus, that these clusters can tentatively be described as distinguishing factors between success and failure. Taking these factors into account and making sure they are taken care off and are carefully monitored and used in a flexible way throughout the project, the chances on success can be enlarged.

Eye openers 1. What the project chairs say: The project chairs revealed a number of success and failure factors, successful procedures of SURF, and points for SURF to improve. These are: • Necessity of the innovation • Results of the innovations should concur with the needs of the (end)users • Commitment of the (end)users to the innovation • Commitment of the management to the innovation • Scalability, the project should not be too complex • Capacity of the organisation • Practicability of the project • Communication The project chairs were generally very positive with respect to a number of (successful) procedures of SURF: • Good support and counselling regarding process and content problems • Flexibility of SURF and SURF Education management • Commitment of SURF and SURF Education management Be that as it may, the chairs also listed a number of points for improvement for SURF: • Intake at the beginning of the project • Review of the content by experts • A more compact style of reporting 2. It’s not our business: Scalability, Sustainability, and Organisation appear to be regarded as topics that the project staff might consider to be outside of the “scope” or “sphere of influence” of the project (in a strict sense). 3. Creating a team: A serious choice of project staffing does not always occur and that it is more a result of who has room in her/his schedule at a given moment and (s)he then becomes a part of the team.

11

According to the soft criterion of a difference ≥ 1 SD

71


4. Discrimination is not always bad: The clusters ‘Scale and Complexity’, ‘Commitment/engagement’ and ‘Expectations of involved parties’ of innovation and innovation projects are discriminating clusters of factors). 5. Lessons told by the project chairs: • committed management is critical • management must also capable of delivering the needed resources • management should be a change-management with a competent change agent • stability of the staff is important 6. Added blessings: • devoted (end) users • sufficient support for users • flexibility throughout the project • simplicity and clarity of the product • embedding in the organisation and policy of the organisation • stability of the used technology and tool • adequate (technical) infrastructure • commitment among users, organisation and management • use of standard software and technology 7. Differences between highly successful and less successful projects: • Project chairs of the highly successful projects were more critical towards themselves. PC of the less successful projects ascribed the problems to factors outside their own range of influence (extracted from the interviews). • The change list of project chairs of the highly successful projects is longer, more profound and aimed more at improving ‘more complex points of improvement’ such as ensuring commitment, more expert feedbacks and responsibilities for the team members and project chair. The change list of project chairs of the low successful projects is shorter, simpler, and more aimed at simple points of improvement such as communication and scalability. 8. The significant others: Clusters rated with more than 1 full point discrepancy between the highly successful en low successful project chairs: • Communication • Commitment • Expectations • Project culture • Complexity, sustainability 9. Paper is patient: According to the document study, no differences or determinants for failure or success can be found in the procedures, reviews, support, or counselling by SURF. In other words, the documents and their evaluations do not discriminate between highly successful and less successful projects.

72


Chapter 4: Recommendations and Checklists

73


SURF’s main interest in this project was to promote that the educational innovation projects they fund will be more successful, particularly, in making the shift from externally-funded to sustained initiatives.

SURF itself The project chairs were generally very positive with respect to a number of (successful) SURF procedures namely: • the good support and counselling provided by SURF regarding process and content problems; • the flexibility of SURF and SURF Education management with respect to overcoming problems; and • the commitment of SURF and SURF Education management to the projects and the project staff. Be that as it may, the chairs also listed a number of points for improvement for SURF: 1. The project chairs would like a more refined and thorough intake at the beginning of the project as a more selective tool assuring good projects to be carried out and not so promising projects to be rejected. Such an intake should concern and pinpoint the consequences of the project or innovation for the institution itself before starting the project. 2. There is a definite need for a more compact and effective style of reporting. The project chairs feel that it is too bureaucratic and often consists of rewriting what has been written earlier. 3. Related to this is the review of the content by experts. Relevant here was the conclusion from the documentation study which confirms the interview data that no differences or determinants for failure or success could be found in the procedures, reviews, support, or counselling by SURF. In other words, the documents and their evaluations do not discriminate between highly successful and less successful projects. 4. SURF could/should help assure that there is sufficient knowledge and skills (i.e., technological, pedagogical and administrative) for relevant stakeholders to carry out their task in the form of extra funds for project teams that signal this problem (e.g., grassroots – small subsidies for specific courses and tools). This means that before the project begins there should be a ‘competence’ review with respect to the staffing and the actions required by them as well as an ‘allocation’ review with respect to resources and funding (i.e., time, staff, financial). 5. SURF should require proof of adequate commitment and assurances from the management of the institution with respect to flanking policy and initiatives to insure institutionalisation and sustainability after completion of the project. 6. SURF could/should help assure that there is sufficient dissemination and dissemination skills (i.e., preparing a conference proposal, writing an article, presenting at a conference, et cetera) for team members in a planned and helpful way (see Dissemination later in this chapter)

Projects The major recommendations flowing from a combination of the literature study, concept mapping, interviews, and the document study are:

75


1. Split the project into three phases (initiation, implementation, and institutionalisation), including the recommended sub phases (see following figure).

Initiation phase






disseminate

disseminate

disseminate


implement

maintain

disseminate

evaluate

In the initiation phase the following factors are important: • clear and realistic project objectives and expectations • clearly defined project mission • clear and limited project scope • return-on-investment • identification of stakeholders • creation of formal feedback channels • initiator of the change should be trusted and respected by the prospected users • experienced project manager who is NOT the organisational manager and who has the necessary responsibility and authority • competent project team members who have clear responsibilities and accountability • willingness and ability to re-plan end revise where necessary • communication of plans and goals to the larger organisation • cognisance of resistors of innovation because they are often aware of unintended consequences of the innovation • commitment and support from executive management and peers In the implementation phase the following factors are important: • ensuring user involvement • framing and limiting user expectations about the innovation • engagement of stakeholders during the process • creation of adequate and continuous communication channels • adoption rate of between 10% and 25% of the system members is maximal

76


• • • •

focus on affecting opinion leaders’ attitudes (the more opinion leaders – positive and negative – that adopt the system, the lower the threshold for adoption by other system members) receptiveness to external criticism continual reflection on assumptions continual modification of the plan based on realities

In the institutionalisation phase the following factors are considered important: • stability of the organisational environment which is supporting and collaborating • relatively loose, centralized organisational structure with good vertical communication channels • competent leadership and management • careful dosing of degree/amount of change • careful managing of user expectations: failure is assured if the users are dissatisfied because the ‘product’ does not meet their expectations. Therefore, project managers should not only manage the development of the system, but also the perception of the system. • visibility: communicating the relative advantage as compared with the current practices • creating structures that promote learning of new practices and observable incentive systems that support them • creating effective communication among all parts of the system 2. Require an analysis of where the organisation needs to be (technological infrastructure, administrative infrastructure, knowledge / skills for all relevant stakeholders, continuing schooling for all relevant stakeholders, and maintenance of the innovation). CATWOE is, for example, a good method for doing this (see Appendix II) as is the thought experiment (see the following figure). Require one or both in the initiation phase!! Plan the innovation

Backcast to today

Where in 10 years?

Vision at the top

Thought experiment


Begin

77


3. Require the organisation to agree to have the solutions to the needs determined in point 3 ready at the time of implementation. 4. Implement the project health checklists in each of the phases. 5. Implement schooling and training for team members where necessary, desirable or advisable (also reserve budget within the projects for ‘grassroots’ training where necessary, desirable or advisable). Further points of attention are: 6. Added value is probably the most important single aspect that needs to be kept in mind. If the problem is not real, if the necessity does not exist, and is not visible to ALL of the stakeholders, then failure is ensured 7. Participation, information and communication and Stakeholder involvement – both people aspects - are also critical. 8. Project success is not the same as innovation success: There are two types of ‘success’, namely the success of the project itself and the success of the innovation (its ultimate implementation). These two ‘successes’ are completely different. Neither type is a requirement or a guarantee for the other. 9. Success of an innovation at the project level does not automatically generalize to success at the institutional level, but needs careful considerations of scalability, generalisability, temporal flexibility, and financial sustainability. 10. The greater the number of changes is, the more factors which influence the chances of success and therefore the greater the risks of the project. This increase in interaction leads to a decrease in control, which leads to a decrease of the chances for success. Investigating the magnitude of the change beforehand, and taking the result of this investigation into account when implementing the project, can diminish the chances of failure. 11. The greatest problem for implementing and sustaining innovation is the state of mind characterised by: “It’s not our business”. Scalability, Sustainability, and Organisation appear to be regarded as topics that the project staff considers to be outside of the “scope” or “sphere of influence” of the project (in a strict sense).

Tender procedure Split the tender procedure into four phases (idea, proposal, presentation, and decision) The phases are: • Idea phase: In this phase the initiators present and test their project ideas to SURF Education in a general way. A SURF project advisor gives the initiators advice such that the initiators know whether they have a decent chance of receiving funding and what they need to addressed in the full proposal. In order to do this, the following actions/needs can be defined: • Clear-cut criteria for evaluating the idea need to be defined. • A form (see appendix VII for an example of such a form) for evaluating the idea and communicating both formative feedback and the decision must be developed. • A capable contact person (project advisor) needs to be appointed. • Proposal phase: In this phase the project is presented - on paper - to a referent- / review panel of external examiners under leadership of the responsible staff member of SURF Education. The result is a provisional evaluation of the project

78


•

•

with three possible outcomes: presentable, presentable if amended, or not presentable. In order to do this, the following actions/needs can be defined: • A referent-/review team needs to be formed. • Clear-cut criteria (see Criteria for the final proposal which follows) for evaluating the proposal need to be defined. • A form for evaluating the proposal and communicating both formative feedback and the decision must be developed. Presentation phase: In this phase projects that received the designation ‘presentable’ or ‘presentable if amended’ (subject to proper amendment) are asked / required to present the project orally (in 10 minutes) to the referent- / review panel of external examiners. In order to do this, the following actions/needs can be defined: • A referent-/review team needs to be formed. • Clear-cut criteria for evaluating the presentation as well as the project need to be defined. • The referent- / review panel of external examiners along with the responsible staff of SURF Education may ask questions to clarify the content and/or to test such things as expertise, commitment, et cetera. Decision phase • The referent- / review panel of external examiners along with the responsible staff of SURF Education makes a decision based upon the oral presentation and the amended proposal.

Criteria for the final proposal In the tender procedure, the following guidelines – taken from the literature study – should be controlled for (see the Initiation Checklist in the second part of this chapter): • There should be clear project objectives. • The general mission of the project should be clearly defined. • The project’s scope should be adjusted in such a way that the level of changes needed on the route from the present to the new situation can be clearly envisioned. • (Simplified) Return-On-Investment should be taken into account. • Stakeholders should be identified. • Formal feedback channels must be created. • There should be an experienced project manager. • The project manager should not be the organisational manager. • The project manager should be given responsibility and authority. • The project team members should be competent. • Care must be taken of clear responsibility and accountability of team members. • The project manager and team members should be able to explain their efforts and results in ways that the larger organisation can understand. • Project manager (team) should be prepared to re-plan. • The project manager and team members should listen to resistors of innovation because they are often aware of unintended consequences of the innovation. • There should be commitment from executive management and peers. • There should be support from executive management and peers. • Realistic expectations should be created. On top of this, the referent- / review panel of external examiners should be aware that:

79


• • •

Sustainability should be a forethought; there must be a planned strategy for maintaining change The project should be managed by a project manager and not by an organisational manager The proposal should contain the following critical ‘win drivers’: • The project objectives are defined and anchored. • The roles and responsibilities of the client, users and project manager are welldefined and established. • The project chair is a change manager. • There are adequate tools / procedures for systematic and thorough measurement of control and performance management. A relevant technique for doing this is to make use of interaction design (see below; Kirschner 2002; Kirschner, Martens, & Strijbos, 2004). Learner / user experience

What do learners actually do? What do learners want to do?

Support / affordances

How can we support what they do? What affordances are needed?

Constraints / conventions

How does the learner perceive the support?

What are the physical, logical and cultural limitations encountered?

Learner / user perceptions

How does the learner actually use the support?

What has the learner / learning group actually achieved?

• • •

Learner / user experience

Learning

The magnitude of the change must be small enough. The work space must be both small and large enough (10-25% of the system members). There are acknowledged sceptics and opinion leaders on board.

Project start Before beginning, the following requirements must be attended to: • Management is committed to the project, but ALSO to the institutionalisation. • Management is capable of delivering the needed resources. • Management should be a change-management with a competent change agent. • Staffing the project needs to be based upon expertise and not fortuity or coincidence such as “Who is available?” • Allied to the previous point, the project must assure that there is sufficient knowledge and skills (i.e., technological, pedagogical and administrative) for relevant stakeholders to carry out their task (see SURF itself). • Stability of the staff is important and must be guaranteed.

80


Project process This is essentially the implementation phase where the innovation is developed, implemented, and evaluated. The following factors are important in this phase: • User involvement should be ensured. • User expectations about the innovation should be managed. • Stakeholders should be engaged. • Initiator of the change should be trusted and respected by the prospected users. • Adequate communication channels should be created. • Focus should be on adoption rate of approximately 25% of the system members. • Focus should be on affecting opinion leaders’ attitudes (the more leaders who adopt the system, the lower the critical adoption rate for other system members will be). • Project manager and team should be open to external criticism. • Project manager and team should continually question own assumptions. • Project manager should continue to modify plan based on realities. According to the project chairs of the most successful projects, there are five clusters of factors that were ‘significantly’ more often taken into account and may, thus, be described as distinguishing factors between success and failure. These clusters are: • Communication • Commitment, engagement • Expectations of parties involved • Project culture • Scale, complexity Finally, both SURF and the project teams must do all possible to prevent the failure causers: • High investment with low output: No balance between investments and output. • Information politics: Power is abused and information is not transmitted. • Lack of responsibility: Uncertainty about responsibility inside and outside the project. • Culture gap: Between ICTers and the rest of the organisation and between those planning education (managers) and those administering education (teachers). • Over-commitment: Not knowing when to cut your losses and stop a project. • All-in-one solutions: Trying to do everything at once.

Prepare for institutionalisation The institutionalisation phase involves continuing the newly implemented change or stabilizing the use of an innovation. In the institutionalisation phase the innovation needs to be scaled and maintained. The following factors are important in this phase: • The organisation should be fitted in a stable environment which is supporting and collaborating. • The organisation should have a relatively loose, centralized structure with good vertical communication channels. • There should be competent leadership and management. • The amount of change should be carefully dosed. • User expectations should be managed: Innovation projects will fail if the users of a system are dissatisfied with it because it does not meet their expectations.

81


• • • •

Therefore, project managers should not only manage the development of the system, but also the perception of the system. The innovation’s relative advantage as compared with the current practices should be communicated. Structures should be created that promote learning of new practices and observable incentive systems that support them. Effective communication among all parts of the system should be created. A high degree of observability, that is, the degree to which other persons than the innovator see its results as beneficial should be created.

Added blessings for the institutionalisation of the innovation are the following aspects that need to be achieved in the first two phases: • Devoted (end) users. • Sufficient support for users. • Flexibility throughout the project. • Simplicity and clarity of the product. • Embedding in the organisation and policy of the organisation. • Stability of the used technology and tool. • Adequate (technical) infrastructure. • Commitment among users, organisation, and management. • Use of standard software and technology.

Dissemination Dissemination is a major part of the project planning and implementation and takes pace at three levels, namely the project team level, the institutional level, and the external level (national and international). • Project team level • Implement a team workspace, preferably making use of the ICT-tools intended by the project or that are standard in the institution. • Create a team: celebrate your successes, put people in the limelight, et cetera. • Institutional level • Involve the rest of the institution, both formally (planned in the tender) or informally throughout the project. • Make good use of available institutional media for continual dissemination of information. This includes setbacks as well as successes, planned presentations, preliminary results, et cetera. • Organise dissemination possibilities (organise workshops and invite relevant stakeholders to team functions). • External level (articles in academic and professional publications and presentations at workshops and conferences) • Plan for written dissemination (time and topic) in the project tender. • Plan for oral dissemination at conferences and workshops (time and topic) in the project tender. • Teams should be ‘required’ to make use of SURF facilities for Support and Schooling. These facilities that SURF should provide include: • Implementing a year schema of possible courses that SURF organises to help team members acquire the necessary awareness, knowledge, and skills

82


• •

to find/access journals, conferences, and workshops; prepare conference/workshop proposals, presenting at conferences. Make funding available for following these ‘courses’ at other relevant institutions (if the need to use external facilities has been proven). Implement a referent- / review team to pre-review proposals and require teams to make use of this (with ample time for review).

83


Checklist Nr 1.

Item

2.

The project goal is clearly defined

3.

The project objectives are clearly defined

4.

Analysis where the organisation needs to be at the time of institutionalisation is conducted

5.

The problem is clearly defined

-

The project’s scope is clearly defined -

6.

The project’s scope is realistic

-

84

Specification The problem has been stated simply, realistically and to-the-point It is evident who experiences the problem It is evident who is responsible for the problem; who the owner of the problem is The desired end result of the project has been stated in the goal The goal statement is smart (specific, measurable, assignable, realistic, time-related) Criteria for the deliverables are included All objectives are complete and necessary to achieve the stated goal All objectives are stated simply, realistically and to-the-point All objectives have a definable and measurable end result All objectives are smart (specific, measurable, assignable, realistic, time-related); compared to the project goal, the objectives are described in a more correct manner (i.e. ‘smarter’) Technological infrastructure, administrative infrastructure, knowledge / skills for all relevant stakeholders, continuing schooling for all relevant stakeholders and maintenance of the innovation are analysed and are taken into account. As a method CATWOE or a thought experiment can be used. The boundaries regarding: problem statement, project planning, project objectives and project goal are stated The scope is stated in such a way that the level of changes needed within the different parts of the organisation on the route from the present to the new situation can be clearly envisioned All deliverables are explicitly coupled to project activities and people. In other words all contributors know what their responsibilities are. Minimised/realistic scope: Time and resources (people/money) are crucial factors for many projects, and since scope affects time or project duration and resources required, minimising scope increases a project’s chances of success Realistic change: The greater the number of changes intended, the more factors there are which influence the chances of success and therefore there is an increase in the risks of the project.


7.

The project’s prior / limiting conditions are clearly defined

8.

The project planning is specified

9.

There is a planned strategy for maintaining change Return-On-Investment is taken into account

10.

-

11. 12.

12

The innovation’s relative advantage as compared with the current practices is communicated Added value is taken into account

-

The cost limits the project will not exceed are stated A list of project deliverables is included Specific due dates of the deliverables are specified Specific due dates of the deliverables are realistic The responsibilities and the authorities have been defined Work packages with responsibilities of the managers of those packages have been defined Time schedule Who does what and when The scalability, sustainability, and organisation of the innovation are treated as topics that are within the “scope” or “sphere of influence The costs are deducted from the benefits Not only the visible, but also the transaction costs12 are calculated. Experts (or at the least knowledgeable crafts persons) were used to help estimate and review each activity’s budget and potential benefits All cost categories and potential benefits for each activity have been estimated (labour, equipment, travel, supplies, etc) Estimates for all internal resource costs and potential benefits were estimated before the need or use of external resources was estimated Cost and potential benefits estimates are realistic and conservative Cost and potential benefits estimates for each activity are comprehensive; there are no hidden costs or surprises Return on investment should be taken into account and communicated Transaction costs should be taken into account and communicated Observability: the degree to which other persons than the innovator see its results as beneficial The added value is made explicit OR explicitly stated To what extent does the innovation solve a problem? The innovation is not only new but also better Added value is clear to all the stakeholders from the beginning Clear contribution to performance (increase) of the learner

The costs, other than the money price, that are incurred in trading goods or services. Elementary versions of economic theorizing often make the simplifying assumption that information and other transaction costs are zero. But realism nevertheless demands that we keep in mind the fact that the benefits to the participants in an exchange have to be high enough to cover their transaction costs if the trade is to take place at all. Indeed, many otherwise mutually advantageous trades do not take place because of the very high transaction costs that would be involved. High transaction costs are very often at the root of the problems discussed under the heading of externalities, especially in those situations where the external costs or benefits accrue to very large numbers of third parties and therefore a contractual agreement to internalize the externality is extremely costly to negotiate.

85


13.

Formal feedback channels are created

-

14. 15. 16. 17.

There is opportunity planned for external criticism There is room for the team to continuously question its own assumptions There is opportunity in the planning to modify the plan based on realities There is a competent / experienced project manager

86

-

Meaningful reporting frequencies have been established The reporting requirements and informed the work package managers accordingly have been specified Variances from plan that will require immediate action (critical path analysis) have been defined and communicated to the project team Responsibilities for completing project status reports have been assigned and communicated to the project team and other interested parties Top management has been briefed on the reporting procedures and they have been approved by them There is attention paid, and opportunities are provided for reporting (orally and in writing) to the organisation at all relevant levels Regular meetings between the project team and the organisation / stakeholders have been planned

- Regular meetings between the project work packages have been planned - Feedback loops with the organisation and stakeholders have been planned - There is room in the planning for course corrections - Successful projects have an experienced project manager. This means that (s)he: o is experienced enough in this project area to successfully commit to complete the project on time, within the budget, and according to specifications o is strategically capable to envision and conceptualise the entire project process through all phases o is technically competent to help design and execute a total project plan o is interpersonally able and experienced to inspire, motivate, discipline, resolve conflicts, negotiate competing interests, and sell the project - The competence of a leader can be rated through and influenced by: o the leader’s vision: the leaders ideal and unique image of the future; o temperament of the leader: whether the leader is flexible, intuitive or whatever; o communication: the degree and type of communication between the leader and the organisation; o durability or endurance: prepared for stress and anxiety that will result of the learning process and changes; o innovation skills: associated with management skills, a leader must be able to create space to experiment, develop and innovate.


18.

The project manager is not the organisational manager

-

-

The purpose of organisational management is to plan, guide and help the organisation work well and survive (i.e., achieve continuity. Project management is to actually actively plan the end of a project (i.e., to break with continuity and achieve change) In contrast to organisational managers, project managers have to start their task from scratch, initiating and bringing an organisation to life. To start something in order to end it within a fixed period of time, according to a budget and according to a number of restrictions distinguishes project managers from organisational managers The project manager is responsible for the project’s success

-

The project manager has officially been given authority by executive management

-

The roles and responsibilities of the client, users and project manager are well-defined and established

-

Who has been given the authority to do and decide upon what

-

19. 20. 21. 22.

The project manager has the necessary responsibilities The project manager has the necessary authority Clear responsibilities of team members are defined Clear accountabilities of team members are defined

87


23.

The project team members are competent

-

24.

25. 26.

The project team members have the necessary time to carry out the project

-

There is commitment from executive management There is support from executive management

88

The project team (i.e., the members together) have the needed expertise, experience and skills for the project The project team members: o are committed to planning and executing the project through all phases o are technically experienced in their specific area to help successfully complete the project on time, within budget, and according to specifications o understand and have used project management concepts and tools o are interpersonally capable of sharing information and expertise, solving conflicts jointly, planning and problem solving with others, actively listening, receiving and giving non-judgemental feedback, communication assertively o are able to work with two ore more bosses o are self-motivated and entrepreneurial but not self-oriented o are flexible o are responsible and can own mistakes as well as achievements o can share rewards and failures o are willing to delegate and be delegated changing work assignments o can work well under pressure and changing deadlines Schooling for team members where necessary, desirable or advisable is implemented (also reserve budget within the projects for ‘grassroots’ training where necessary, desirable or advisable) A project team member’s work cannot be carried out in ‘spare time’. There should be ample room in the member’s work package to allow the necessary work. Stability of the staff is important and must be guaranteed There is ample room in the member’s work package to allow the necessary schooling. The plan includes the signature of management proving commitment in terms of all aspects of the project planning and execution Resources like time, material, budget, personnel is guaranteed to be available for the project, including implementation at the end of the project


27.

28.

29.

30.

There is competent organisational management

Good managers help to keep the organisation on track. Factors that contribute to an innovation success: Mission management: refers to knowing whom or what the organisation serves (strategies, planning, evaluation, contacts etc) facilitates the innovation; - Pay and personnel: payment and rewarding of the personnel is crucial for the success of an innovation; - Learning: facilitating organisational learning by the group rather than by one person can contribute to the success; - Idea generation: providing establishing systems that provoke new ideas can facilitate the innovation; - Budget: a balance between protecting the organisation’s resources and the investments (Return-onInvestment) increases the chances of success; - Accountability and governance: monitoring, supporting and coaching are important. The organisation exists in a stable The success of the innovation can be facilitated or inhibited by: environment, is supportive and - turbulence in the environment: the level of uncertainty in the surroundings can inhibit the innovation’s collaborative) chances; - level of shock to the system: an extreme event such as a budget crisis can endanger the chances of success; - support or encouragement for the innovation: the capacity of the organisation to meet a new idea with enthusiasm can facilitate the innovation; - collaboration: balancing collaboration between various subsystems can be crucial for the success; - availability of external support for the innovation: support such as external funding can facilitate the innovation; - dedication to implementation. The organisation has a relatively The chances of success of the innovation can be influenced by: loose, centralized structure with good - the structure of the organisation: innovations started in relatively loose yet centralised organisational vertical communication channels structures with good communication channels have more chance to be successful; - the demographics or level of diversity of its members: innovative teams containing of near peers tend to be more successful; - internal turbulence: such as high staff turnover or frequent re-organisations can disrupt the sustainability of an innovation; - internal boundaries: creating barriers to widespread diffusion of ideas and innovations within a system can inhibited the innovation; - internal resources: such as staff time, funds, office space etcetera facilitate the innovation. Realistic expectations are created - Expectations regarding to scope, planning, goal, objectives, etc. -

89


31.

User involvement is ensured

32.

User expectations about the innovation are managed

33.

Stakeholders are identified

-

34.

Stakeholders are engaged and involvement is taken into account

-

35.

Adequate informationcommunication channels are created

-

36. 37. 38.

13

-

There is continuous reporting to the organisation (end users) There is sufficient support for users Innovation projects will fail if the users of a system are dissatisfied with it because it does not meet their expectations. Therefore, project managers should not only manage the development of the system, but also the perception of the system It is clear who has a stake in what will be done (e.g., problem owner, target group, government, industry, educational institutions, students, teachers, tutors, society in general) The groups that must be interacted with to accomplish project goals have been identified Strategic interests / concerns of each stakeholder has in the project are identified in order to negotiate common interests All stakeholders are involved from the beginning Participative design as methodology is (to be) used Participants are ambitious; ambition counts! Successes are (to be) celebrated A high degree of observability, that is, a degree to which other persons than the innovator see its results as beneficial, should be created Lack of adequate information communication and fuzzy use of language can cause miscommunications and misconceptions about the innovation Effective communication among all parts of the system should be created Clear and frequent communication before, during and after the project Information facilities are sufficient PR is taken into account Formal feedback channels are created Dissemination is taken care of at three levels, namely the project team level, the institutional level, and the external level (national and international) Effective communication among all parts of the system should be created

Dissemination is a major part of the project The project manager and team members are able to clearly explain the project13 There is a focus on affecting opinion - The more opinion leaders that adopt the system, the lower the critical adoption rate for other system leaders’ attitudes members will be. - There are acknowledged sceptics and opinion leaders on board of the project This means that the SURF procedure includes an oral presentation AS WELL AS a written project plan.

90


39. 40. 41.

42.

43. 44.

Initiator of the change is trusted and respected by the prospective users There is a focus on adoption rate of maximally 25% of the institution

-

- The work space must be both small and large enough (10-25% of the system members) - The step (transition) from project to adoption / implementation is not too large. - The magnitude of the change must be small enough The amount of change is carefully - The key to effective change (or innovation) is to carefully dose the amount of change, and therefore of dosed the perceived threat, produced by the disconfirming information to allow the group members to feel safe to un-freeze, change and re-freeze their ideas and concepts. - The more an innovation plans to change, the greater the influence from the environmental will become, the greater the risks will be and the chances of failure will increase - The greater the number of changes is, the more factors there are that influence the chances of success and therefore there is an increase in the risks of the project - The magnitude of the change is investigated beforehand, and the result of this investigation is taken into account during the implementation of the project. Structures and opportunities are - Structures should be created that promote learning of new practices and observable incentive systems planned that promote learning of new that support them. practices - Dissemination should be a major part of the project planning and implementation. This dissemination takes pace at three levels, namely the project team level, the institutional level, and the external level (national and international). Observable incentive systems to - Sustainability should be a forethought; there must be a planned strategy for maintaining change support implementation are planned Turn the traditional organisation into - A learning organisation that adapts to innovations and restructures itself to accommodate change a “learning organisation”

91


Chapter 5: Dissemination

93


Publications Hendriks, M., Kirschner, P. A., Paas, F., & Wopereis, I. G. J. H. (2005a). Determinants for failure and success of innovation projects: The road to sustainable educational innovations. A literature study. Manuscript in preparation. Hendriks, M., Kirschner, P. A., Paas, F., & Wopereis, I. G. J. H. (2005b). Faal- en succesfactoren van innovatieve ICT-onderwijsprojecten [Failure and success factors of innovative ICT projects in education]. Manuscript submitted for publication. Hendriks, M., Kirschner, P. A., Paas, F., & Wopereis, I. G. J. H. (2005c). Failure and success factors of educational ICT projects: An in-depth analysis of projects. Manuscript in preparation. Wopereis, I. G. J. H., Kirschner, P. A., Paas, F., Stoyanov, S., & Hendriks, M. (2005). Determining failure and success factors of educational ICT projects using a group concept mapping approach. Manuscript submitted for publication.

Conferences Kirschner, P. A., Cordewener, B., Paas, F., Wopereis, I., & Hendriks, M. (2005). Determinants for success and failure of educational innovation projects. Paper submitted to the EUNIS conference 2005. Kirschner, P. A., Cordewener, B., Wopereis, I., Paas, F., & Hendriks, M. (2005). Determinants for success and failure of educational innovation projects. Paper submitted to the EDUCAUSE 2005 Conference. Kirschner, P. A., Hendriks, M., Paas, F., Wopereis, I., & Cordewener, B. (2004, October). Determinants for failure and success of innovation projects: the road to sustainable educational innovations. Paper presented at EDUCAUSE 2004 Conference, Denver, Co, USA. Kirschner, P. A., Paas, F., Hendriks, M., Wopereis, I., & Cordewener, B. (2004, October). Determinants for success of educational innovation projects: A study of “SURF Education” projects in the Netherlands. Paper presented at the 2004 AECT International Convention, Chicago, Il, USA. Kirschner, P. A., Paas, F., Hendriks, M., Wopereis, I., & Cordewener, B. (2005, April). Project features as a crucial factor: Determinants for success and failure of educational innovation projects. In W. Jochems (Chair), Factors that make innovation in higher education a success or a failure: Dutch and international experiences. Symposium conducted at the Annual Meeting of the American Educational Research Association (AERA), Montreal, Canada. Paas, F., Wopereis, I., Hendriks, M., & Kirschner, P. (2004, November). Succes- en faalfactoren van SURF onderwijsinnovatieprojecten: De weg naar duurzame onderwijsinnovatie [Success and failure factors of SURF educational innovation projects: The road to sustainable educational innovation projects]. Paper presented at the SURF Onderwijsdagen, November 16-17, Utrecht, The Netherlands. Wopereis, I., Kirschner, P., Paas, F., Hendriks, M., & Cordewener, B. (2005). Success and Failure Factors in ICT Innovations in Higher Education. Paper submitted to the EARLI conference 2005. Wopereis, I., Kirschner, P., Paas, F, Hendriks, M., & Cordewener, B. (2005). Succesen faalfactoren van onderwijsinnovatieprojecten in het hoger onderwijs [Success

95


and failure factors of educational innovation projects in higher education]. Paper submitted to the ORD conference 2005.

Workshops Kirschner, P. A., & Wopereis, I. G. J. H. (2005, March). Succes- en faalfactoren van onderwijsinnovatieprojecten: Uitgangspunten voor advisering [Success and failure factors of educational innovation projects: Principles for advice]. Workshop at the CRWO Conference, March 18, Nijmegen, The Netherlands.

Articles about the project Gielen, A. (2004, November 2). Determinants for succes and failure of educational innovation projects. Edusite. Retrieved February 24, 2005, from http://test3.eccoo.rug.nl/edutrip2004/verslagen_leadership/13814 Schoppert, S. (2004, October 22). Hoogleraar Kirschner op EDUCAUSE 2004 [Professor Kirschner at EDUCAUSE 2004]. Edusite. Retrieved February 24, 2005, from http://test3.eccoo.rug.nl/edutrip2004/verslagen_leadership/13728 Schoppert, S. (2004, October 23). EDUCAUSE 2004 / Hoogleraar Kirschner: "Goede organisatie verkeert in staat van dynamische stabiliteit" [EDUCAUSE 2004 / Professor Kirschner: “A good organisation is in a state of dynamic stability”]. Edusite. Retrieved February 24, 2005, from http://test3.eccoo.rug.nl/edusite/home/13733

96


References Alexander, S., & McKenzie, J. (1998). An evaluation of information technology projects for university learning. Canberra: Australian Government Publishing Services. Arlow, J., & Neustadt, I. (2001). UML and the unified process: Practical objectoriented analysis and design. Boston, MA: Addison Wesley. Berg, B., & Ostergren, B. (1977). Innovations and innovation processes in higher education. Stockholm: National Board of Universities and Colleges. Berg, B., & Ostergren, B. (1979). Innovation processes in higher education. Studies in Higher Education, 4, 261-268. Buzan, T., & Buzan, B. (1996). The mind map book. New York: Plume. Carroll, J. (2000). Making use: Scenario-based design of human-computer interaction. Cambridge: MIT Press. Checkland, P. B., & Scholes, J. (1990). Soft Systems Methodology in Action. Chichester: Wiley. Constantine, L. L. (2001). The peopleware papers: Notes on the human side of software. Upper Saddle River, NJ: Prentice-Hall. Constantine, L. L., & Lockwood, L. A. D. (1999). Software for use: A practical guide to the models and methods of usage-centered design. Boston: Addison Wesley. Cross, N. (2000). Engineering design methods. Strategy for product design (3rd ed.). Chichester, UK: Wiley. Decision Explorer [Computer software]. (2003). Kendal, Cumbria, UK: Banxia Software. De Bie, D. (2003). Morgen doen we het beter: Handboek voor de competente onderwijsvernieuwer [Tomorrow, we will do better: Handbook for the competent educational innovater]. Houten, The Netherlands: Bohn Stafleu Van Loghum. De Koning, P. C. J., & Florijn, R. (1995). Ingrijpend innoveren; business process innovation. De methodiek in praktijk [Radical innovation; business process innovation. The methodology in practice]. Amsterdam: Giarte / Moret Ernst & Young Management Consultants. Eden, C., Ackermann, F., & Cropper, S. (1997). Getting started with cognitive mapping. Glasgow, UK: Banxia Software. Eden, C., & Ackermann, F. (2002). Making strategy. The journey of strategic management. London, UK: Sage Publications. European Commission. (2002). Report on the implementation of the European label for innovative projects in language teaching and learning 1999-2001. (EAC.B.4/PB/pl D). Retrieved April, 29, 2004 from http://europa.eu.int/comm/education/policies/lang/innov/report_1999-2001_en.pdf Forester (2002). Lessons learned from design leaders. Retrieved May 31, 2002 from http://www.forrester.com/ER/Research/ Fullan, M. G. (1991). The new meaning of educational change (2nd ed.). London: Cassell Education Limited. Fullan, M. G. (2001). The new meaning of educational change (3rd ed.). New York, NY: Teachers College. Geerlings, J., Mittendorff, K., & Nieuwenhuis, L. (2004). Succesvol innoveren van beroepsonderwijs [Successful innovation of vocational education]. Retrieved March

97


25, 2004 from http://www.hetplatformberoepsonderwijs.nl/download/Succesvol%20innoveren.doc Gustafson, K., & Watkins, K. (1998). Return on investment (ROI): An idea whose time has come – again? Educational Technology, 38(4), 5-6. Hannan, A., English, S., & Silver, H. (1999). Why innovate? Some preliminary findings form a research project on ‘Innovations in teaching and learning in higher education’. Studies in Higher Education, 24, 279-289. Hays, S. P., & Scholla, K. L. (2003, April). Engaging youth in tobacco policy change: Results from a concept mapping project in Madison County, Illinois. Paper presented at the 2003 Annual Meeting of the Midwest Political Science Association, Chicago, IL. Retrieved February 10, 2005 from http://www.civicyouth.org/PopUps/Scott%20Hays.pdf Hendriks, M., Kirschner, P. A., Paas, F., & Wopereis, I. (2005). Determinants for failure and success of innovation projects: The road to sustainable educational innovations. A literature study. Manuscript submitted for publication. Holmes, A. (2001). Failsafe IS project delivery. Burlington: Ashgate. Jackson, K. M., & Trochim, W. M. K. (2002). Concept mapping as an alternative approach for the analysis of open-ended survey responses. Organisational Research Methods, 5(4), 307-336. Johnson, J., Boucher, K. D., Connors, K., & Robinson, J. (2001). Project management: the criteria for success. Retrieved January 22, 2004 from http://www.softwaremag.com/archive/2001feb/collaborativeMgt.html Karlsen, J. T., Svendsen, E. K., & Omli, L. E. (2003). Project management - return on investment. In J. A. Langlo (Ed.), Project management: Dreams, nightmares, and realities: Proceedings NORDNET 2003, International Project Management Conference, Oslo, Norway, September 24-26, 2003 (pp. 159-170). Trondheim, Norway: Norwegian Centre of Project Management / Oslo, Norway: Norwegian Association of Project Management. Katz, D., & Kahn, R. L. (1978). The social psychology of organisations. New York: Wiley. Kenny, M., & Meadowcroft, J. (Eds.). (1999). Planning sustainability. London: Routledge. Kensing, F., & Blomberg, J. (1998). Participatory design: Issues and concerns. In Computer Supported Cooperative Work (CSCW) (Vol. 7(3/4), pp. 167-185). Dordrecht, The Netherlands: Kluwer Academic Publishers. Kirschner, P. A. (2002). Can we support CSCL? Educational, social and technological affordances for learning. In P. A. Kirschner (Ed.), Three worlds of CSCL: Can we support CSCL. Inaugural address, Open University of the Netherlands. Kirschner, P. A., Martens, R. L., & Strijbos, J. W. (2004). CSCL in higher education? A framework for designing multiple collaborative environments. In P. Dillenbourg (Series Ed.) & J. W. Strijbos, P. A. Kirschner & R. L. Martens (Vol. Eds.), Computer-supported collaborative learning: Vol 3. What we know about CSCL ... and implementing it in higher education (pp. 3-30). Boston, MA: Kluwer Academic Publishers. Kor, R. & Wijnen, G. (2001). 50 Checklisten voor projecten programmamanagement. [50 Checklists for project and programme management]. Deventer, The Netherlands: Kluwer.

98


Kruskal, J. B., & Wish, M. (1978). Multidimensional scaling. Beverly Hills, CA: Sage. Light, P. C. (1998). Sustaining innovation. San Francisco: Jossey-Bass. Lindquist, J. (1978). Strategies for change. Berkeley, CA: Pacific Soundings Press. Mathias, H., & Rutherford, D. (1983). Decisive factors affecting innovation: a case study. Studies in Higher Education, 8, 45-55. Moonen, J. (2003). Simplified Return-on-Investment. Interactive Learning Environments, 11 (2), 147-165. Novak, J. (1998). Learning, creating and using knowledge. Concept maps as facilitative tools in schools and corporation. Mahwah, NJ: LEA. O’Hara, M. T., Watson, R. T., & Kavan, C. B. (1999). Managing the three levels of change. Information Systems Management, 16(3), 63-70. Powers, J. (2000). The use of institutional incentive grants for strategic change in higher education. The Review of Higher Education, 23, 281-298. Rogers, E. M. (1995). Diffusion of innovations (4th ed.) New York: The Free Press. Rutherford, D. (1992). Appraisal in action: a case study of innovation and leadership. Studies in Higher Education, 17, 201-210. Schein, E. H. (1995). Kurt Lewin’s change theory in the field and the classroom: Notes toward a model of managed learning. Retrieved March 28, 2004 from http://learning.mit.edu/res/wp/10006.html Sherry, L. (2003). Sustainability of innovations. Journal of Interactive Learning Research, 13, 209-236. Storm, P. M., & Jansen, R. E. (2004, August). High performance projects. A speculative model for measuring and predicting project success. Paper presented at the IRNOP VI project research conference, Turku, Finland. Stoyanov, S. (2001). Mapping in the educational and training design. Unpublished doctoral dissertation, University of Twente, The Netherlands. Stoyanov, S., & Kirschner, P. A. (2004). Expert concept mapping method for defining the characteristics of adaptive e-learning: ALFANET project case. Educational Technology Research and Development, 52(2), 41-56. Straker, D. (1997). Rapid problem solving with Post-it notes. Tucson, AZ: Fisher. Teasly, U. H. (1996, February). Factors affecting teachers’ decisions about use and non-use of computers: Administrative actions. Paper presented at the Association for Educational Communication and Technology, Indianapolis, IN. Turner, J. R. (1999). The handbook of Project-Based management. London: McGrawHill Trochim, W. M. K. (1989a). An introduction to concept mapping for planning and evaluation. Evaluation and Program Planning, 12, 1-16. Trochim, W. M. K. (1989b). Concept mapping: Soft science or hard art? Evaluation and Program Planning, 12, 87-110. Trochim, W. M. K. (1993, November). Reliability of concept mapping. Paper presented at the Annual conference of the American Evaluation Association. Dallas, Texas. Trochim, W. M. K. (1999a, November). Measuring organisational performance as a result of installing a new information system: Using concept mapping as the basis for performance measurement. Paper presented at the annual conference of the American Evaluation Association. Orlando, Florida. Trochim, W. M. K. (1999b, October). The evaluator as cartographer: Technology for mapping where we’re going and where we’ve been. Paper presented at the

99


conference of the Oregon Program Evaluators Network “Evaluation and Technology: Tools for the 21st Century” Portland, Oregon. Turner, J. R. (1999). The handbook of Project-Based management. London: McGrawHill. Van der Heijden, K. (1996). Scenarios: The art of strategic conversation. Chichester: John Wiley & Sons. VanGundy, A. (1997). Techniques of structured problem solving. New York: Van Nostrand. Vennix, J. A. M.. (1997). Group model building. Facilitating team learning using system dynamics. Chichester: John Wiley & Sons. Vinkenburg, H. (2003). Vijf aspecten van onderwijsvernieuwing. Onderwijsinnovatie, september, 2003. Weijers, F. (2003, May). Ruimte voor verandering: Succesfactoren bij onderwijsinnovaties in de beroepskolom [Space for change: Success factors of educational innovation in vocational education]. De Bilt, the Netherlands: Het Platform Beroepsonderwijs. Retrieved December 8, 2003 from http://www.hetplatformberoepsonderwijs.nl/download/Ruimte_voor_verandering_ HPBjun03.pdf Wetenschappelijk Technische Raad SURF (21 januari 2003). Advies tender 2003 en aanpak proefvelden. Zhao, Y., Pugh, K., Sheldon, S., & Byers, J. L. (2002). Conditions for classroom technology innovations. Teachers College Record, 104, 482-515.

100


Appendix I - SURF projects (Tender year 1999 up to 2001) Tender year

Project

Topica

1 2 3 CASCADE X Didiclass X Digital Didactics Digital Language Classroom E-Folio X GIPSY IbL X X Competency management system and X digital portfolio 2001 MusicBase 2001 SPINE X 2001 Com2know 2001 VPRO2 X 2001 ZAP (Highly Interactive Psychology) X 2000 World Wide Writing X 2000 ICT X 2000 K-Web X 2000 MultiVLA 1999 POLARIS 2 X 1999 Task-based teamlearning with ICT X 1999 TeleTOP® 1999 VOICE X X 1999 Walhalla X a 1=Competencies/Portfolio 2=Collaborative learning 3=Interactive teaching materials 4=Learning Content Management Systems (LCMS)/Communities 5=New Media

4 X

2001 2001 2001 2001 2001 2001 2001 2001

5 X

X X X X

X X

X X

X X X X

(for more information see http://www.surf.nl/en/download/SURF_OVP_UK.pdf)

101


Appendix II - CATWOE CATWOE is a Soft Systems Methodology (SSM) developed at Lancashire University (Checkland, 1981) for the analysis. Point of departure is that problems can be seen differently by different participants in a process. This has to be prominent in mind for the design and development of an effective and efficient system. The method used is called CATWOE, which is an acronym for: - Clients / Customers (Who is the system for? / Who uses it?) - Agents / Actors (Who are involved in/with the system?) - Transformations (What does the system do? / What should the system do?) - World View (What is the idea behind it?) - Owners (Who pays for it? / Who has commissioned it?) - Environment (Where should it work? / How does the environment influence its use?) Gebaseerd op CATWOE-uitgangspunten, moet een aantal vragen beantwoord worden zoals: - Wie zijn de gebruikers of “actoren”? - Wie zijn er nog meer bij betrokken zonder gebruiker te zijn? - Wat zijn voor ieder van de gebruikers de doelstellingen, taken, acties, en operaties? - Wat wordt er gedaan en hoe wordt het gedaan? - Wat zijn de rollen die horen bij die taken? - In welke omgeving werken de actoren? - Hoe moet het systeem in die omgeving ingepast worden? - Welke eisen worden er gesteld door de opdrachtgevers? (Dit zijn niet de gebruikers!) - Binnen welk tijdsbestek moet het beschikbaar zijn? - Welke (soorten) data (format)vereisten zijn er? - Welke usability vereisten zijn er? - Leerbaarheid (Wat is de moeilijkheidsgraad van het systeem?) - Throughput (Hoe snel/langzaam/kort/lang zijn de /langzame handelingen?) - Flexibiliteit (Hoe aanpasbaar moet het zijn?) - Attitude (Met welke instelling zit de gebruiker achter het systeem?) - Wat kan niet / hoeft niet in de omgeving gedaan te worden? - Wat kan alleen maar via de omgeving gedaan worden? - Waarom zou men het willen gebruiken? - Welke datastromen zijn te identificeren? (Wat van waar/wie naar/via waar/wie?)

103


Example of a CATWOE (Brainstorm version – in Dutch) 1 Cliënten (Wie gebruikt het?) 1.1 Studenten Het gaat om een beperkt aantal 4e jaars studenten, ca. 4. Later kunnen dat er misschien 10-20 worden, maar veel meer waarschijnlijk niet. Deze studenten hebben verhoudingsgewijs vrij veel ervaring met de computer. Ze werken er veel mee. Gedurende de eerste 3 jaar hebben ze ook al allerlei projecten gedaan waarin de computer een belangrijk hulpmiddel is. 1.1.1 Taken Gedacht wordt aan de volgende taken: • Bedrijfsoriëntatie • Probleemdefinitie • Projectplan schrijven • Thesis schrijven • Meedraaien in het bedrijf • Literatuur zoeken / lezen / ordenen 1.1.2 Werkmethoden • Uitwisselen van ideeën met andere studenten • Uitwisselen van ideeën met vakdocent (en andere docenten) • Individueel werken aan verslagen e.d. • Samenwerken met iemand uit het bedrijf • Zoeken op Internet • Zoeken in bibliotheek (via Internet) 1.2 Docenten Het gaat hier om docenten KT. Ze hebben verhoudingsgewijs veel ervaring met de computer, maar er zijn grote individuele verschillen. Bovendien zijn er verschillen in computergebruik tussen Informatica en Wiskunde. 1.2.1 Taken Gedacht wordt aan de volgende taken: • Hulp bij de probleemdefinitie • Aanwijzingen voor het vinden van literatuur • Helpen bij vinden van richting voor de oplossing 1.2.2 Werkmethoden • Uitwisselen van ideeën • Lezen en mening geven over concepten • Spreekuur 1.3 Expert coach (vakdocent) Dit zijn dezelfde als de docenten, behalve dat ze een aparte taak hebben. Zij zijn de centrale docent voor de inhoud van het project. De expert coach zal zich vooral buigen over de inhoud van het project. 1.3.1 Taken Gedacht wordt aan de volgende taken: • Hulp bij de probleemdefinitie • Aanwijzingen voor het vinden van literatuur • Helpen bij vinden van richting voor de oplossing • Beoordelen van de gekozen oplossing

105


1.3.2 Werkmethoden • Gezamenlijk overleg • Uitwisselen van ideeën • Lezen van concepten (plannen, rapportages etc.) • Lezen en beoordelen van definitieve producten • Monitoren van voortgang inhoudelijk en tijdplanning • Incidenteel aanspreken (op welke manier dan ook) • Werkbezoek • Spreekuur 1.4 Bedrijfscoach Dit zijn mensen uit het bedrijf waar de studenten het project uitvoeren. Over de computervaardigheid van deze mensen kunnen we vooraf geen uitspraken doen. Gedacht wordt aan de volgende taken: 1.4.1 Taken • Wegwijs maken in het bedrijf • Probleemruimte schetsen • Hulp bij de probleemdefinitie • Aanwijzingen voor het vinden van literatuur, experts en andere belanghebbenden • Helpen bij vinden van richting voor de oplossing • Beoordelen van de gekozen oplossing 1.4.2 Werkmethoden • Gezamenlijk overleg • Uitwisselen van ideeën • Lezen van concepten (plannen, rapportages etc.) • Lezen en beoordelen van definitieve producten • Monitoren van voortgang inhoudelijk en tijdplanning • Incidenteel aanspreken (op welke manier dan ook) • Werkbezoek • Spreekuur 1.5 Mentor (procedureel begeleider) 1.5.1 Taken • Controleren van tijdplanning • Controleren van eisen waaraan moet worden voldaan • Niet inhoudelijke problemen (helpen) oplossen 1.5.2 Werkmethoden • Regelmatig voortgang kunnen bekijken • Gezamenlijk overleg met student, docent en bedrijfsbegeleider 1.6 Systeembeheerder 1.6.1 Taken • Bijhouden van de gebruikers en hun rol • Hulp bij technische problemen 1.6.2 Werkmethoden • Op afstand kunnen bedienen van het systeem • Overleg op afstand voor afhandelen van een probleem.

106


1.7 Administratieve medewerkers 1.7.1 Taken • Bijhouden van administratieve zaken (WELKE?) 1.7.2 Werkmethoden • Werken vanuit de universitaire werkplek • (ANDERE METHODEN?) 2 2 Agents (Andere betrokkenen) 2.1 Medewerkers van de universiteit Incidentele andere experts vanuit de universiteit? 2.2 Medewerkers van bedrijf Incidentele andere experts vanuit het bedrijf? 3 3 Transformaties (Wat moet er gedaan worden?) 3.1 Probleemoriëntatie Verzamelen van informatie voor de probleemoriëntatie. Waarschijnlijk in de vorm van allerlei documenten: papier en digitaal. Maken van notities. Literatuurbestand 3.2 Probleemdefinitie Een (digitaal) document 3.3 Projectplan Een (digitaal) document, als dan niet in de vorm van MS Project, Excel work sheet. 3.4 Thesis schrijven, conceptversies maken Een (digitaal) document 3.5 Literatuur verzamelen Een literatuurbestand met abstract? 3.6 Commentaar verzamelen / Ideeën bespreken/bediscussiëren • In de vorm van documenten die bekeken zijn door anderen • Uitgewisselde e-mail • Discussieberichten • Notities van gesprekken 3.7 Doelen stellen / Taken formuleren Waarschijnlijk in de vorm van een projectplan of plan van aanpak. Mijlpalen vaststellen. Aantekenen in de agenda Ook opdrachten formuleren en data stellen voor de oplevering. 3.8 Taken bijhouden Voortgangsbewaking 4 4 Weltanschauung (Wat zijn de ideeën er achter?) 4.1 Individuele opdrachten in projectvorm Het idee achter het Duale Leren is leren op de werkplek in de vorm van een project. Echter, het gaat om individuele projecten. 4.2 Samenwerkend leren Het individuele werken/leren wordt afgewisseld met gezamenlijke opdrachten en gezamenlijk brainstormen. 4.3 Docent is domeinexpert De student moet zoveel mogelijk zelf doen. De docent bewaakt het leerproces inhoudelijk.

107


4.4 Mentor is procesbewaker De mentor is verantwoordelijk voor de algehele voortgang. Lukt het? Worden de deadlines gehaald? Hoe gaat de samenwerking met andere studenten, docenten, het bedrijf? Zijn er motivatieproblemen? 5 Eigenaren (Wie is opdrachtgever en betaalt er voor?) 5.1 Opleiding 1 Opleiding 1is de opdrachtgever. De omgeving is bedoeld voor ZXC. Het moet later groot inzetbaar zijn voor Masters en Bachelors opleiding, en voor verschillende afstudeerrichtingen 5.2 Opleiding 2 Opleiding 1is de opdrachtgever. De omgeving is bedoeld voor VBN. Het moet later groot inzetbaar zijn voor Masters en Bachelors opleiding, en voor verschillende afstudeerrichtingen 6

Omgeving (Waar moet het straks gebruikt worden?) 6.1 Bedrijfswerkplek De bedrijfswerkplek is de meest belangrijke werkplek. Het grootste probleem is daar waarschijnlijk: • Besturingssysteem (MS Windows, Apple Mac, Unix) • Mate van beveiliging van de toegang tot internet (proxy servers) 6.2 Thuis-werkplek De thuiswerkplek neemt de tweede belangrijke plaats in. Met name in de aanvang periode. Beperkingen zijn dan voornamelijk: verbinding met internet: • Bandbreedte • verbindingskosten 6.3 Universiteit De universiteit is als werkplek voor de studenten veel minder belangrijk, maar wel voor de docenten, mentoren e.d. Potentiële problemen daar: • Integratie met bestaande software (MS Windows, Outlook, Exchange)

108


Appendix III - Factor ratings Factors ranked by mean (displayed are rank, factor description in Dutch, mean, standard deviations, number of experts that scored the factor, range scores, minimum score and maximum score on a five point Likert scale. Rank 1 2

Factor Zichtbaarheid toegevoegde waarde / Hoe zichtbaar zijn de voordelen van een innovatie; kwestie van PR, maar ook keuze van accenten. Hoe zichtbaarder hoe beter. In hoeverre lost een innovatie een probleem op, hoe meer men denkt dat dit zo is des te beter (de innovatie is niet alleen nieuw maar ook beter)

Mean

SD

N

Range

Min

Max

4,54

0,66

13

2

3

5

4,46

0,52

13

1

4

5

3

De projectleider moet competent zijn

4,46

0,66

13

2

3

5

4

Vier je successen

4,46

0,78

13

2

3

5

5

Betrokken en competent management

4,38

0,65

13

2

3

5

6

Volledig toegewijde, kundige projectleider

4,38

0,87

13

3

2

5

7

Positieve ambitie bij betrokkenen

4,38

0,96

13

3

2

5

8

Deskundigheid binnen team

4,33

0,78

12

2

3

5

9

Betrokkenheid afnemers (docenten, studenten, bestuurders)

4,31

0,75

13

2

3

5

10

Open cultuur bij medewerkers van project

4,31

0,95

13

3

2

5

11

Aantoonbare relatie tussen innovatie en verbetering kwaliteit en kwantiteit leeractiviteiten van studenten

4,23

0,60

13

2

3

5

12

Inhoudelijk / organisatorisch etc. competente medewerkers binnen projecten

4,23

0,73

13

2

3

5

13

Informeren

4,23

1,01

13

3

2

5

14

Sterke sleutelpersonen op cruciale posities

4,17

1,03

12

3

2

5

15

Duidelijke frequente communicatie voor, tijdens en na project

4,15

0,69

13

2

3

5

16

Gedeelde visie over het wat en het hoe

4,15

0,69

13

2

3

5

17

Toegevoegde waarde moet snel (liefst vanaf het begin) duidelijk zijn voor een ieder

4,15

0,69

13

2

3

5

18

Blijvende managementaandacht en monitoring

4,15

0,69

13

2

3

5

19

Succesvolle tussenproducten / zichtbare resultaten

4,15

0,80

13

3

2

5

20

Participatie van betrokkenen tijdens traject

4,15

0,90

13

2

3

5

21

Betrokkenheid in de organisatie

4,15

0,90

13

2

3

5

22

(vanaf het begin) bewustwording van verandernoodzaak en werken aan conceptwisseling / paradigmashift

4,15

0,90

13

2

3

5

23

Teamvorming bij de direct betrokkenen

4,15

0,90

13

3

2

5

24

Doelen van het project liggen in lijn met de activiteiten van de organisatie (succesfactor)

4,15

0,90

13

3

2

5

25

Autoriteit, gezag, gaat wel/niet actief en zichtbaar achter de innovatieactiviteiten staan

4,15

0,90

13

3

2

5

26

Duidelijk bijdragen aan de performance (verbetering) van de lerende / de werkende (1)

4,15

0,90

13

3

2

5

27

Ervaren meerwaarde voor betrokkenen: wat wordt student, docent, management 'beter' van de betreffende innovatie?

4,15

0,99

13

3

2

5

28

Probleemeigenaar, ownership: wie heeft er een probleem, hoe urgent is het probleem voor de probleemeigenaar?

4,15

0,99

13

3

2

5

109

Determinants for Success and Failure of Innovation Projects 29

Tijd om van het innoveren te leren / Tijd om fouten te mogen maken en te zoeken

4,15

0,99

13

3

2

5

30

Analyse van hoe zaken nu gaan; en waarom ze gebeuren zoals ze gebeuren en niet zoals gewenst

4,15

1,07

13

3

2

5

31

Analyse van wat men wil bereiken

4,08

1,04

13

3

2

5

32

De kwaliteit van de innovatie, de meerwaarde t.a.v. bestaande oplossingen

4,08

1,04

13

3

2

5

33

Duidelijke noodzaak en nut van de innovatie waardoor "energie" van mensen vrijkomt

4,08

1,12

13

4

1

5

34

Bespreekbaar maken van weerstand, politiek en retoriek

4,08

1,26

13

3

2

5

35

Ontbreken van daadkracht om in te grijpen als resultaten achterblijven, personen nalatig zijn (faalfactor)

4,08

1,32

13

4

1

5

36

Studenten en "klanten" betrekken bij innovaties

4,00

0,71

13

2

3

5

37

Facilitering van innovatie: (a) tijd, (b) geld, (c) deskundigheid, (d) middelen

4,00

0,71

13

2

3

5

38

Topmanagement is niet alleen facilitator, doch ook voorvechter

4,00

0,74

12

2

3

5

39

Overeenstemming m.b.t. doelen van project

4,00

0,82

13

2

3

5

40

Duidelijke strategische koers, ambitie

4,00

0,82

13

3

2

5

41

Flexibiliteit in organisatie om tijd en middelen vrij te maken voor innovatie-investeringen

4,00

0,91

13

3

2

5

42

Doelstellingen helder

4,00

1,00

13

3

2

5

43

Eisen dat er een opdrachtgever is die de functionele eisen kan beoordelen / wil hebben (succesfactor)

4,00

1,08

13

3

2

5

44

Een heldere opdracht

4,00

1,08

13

3

2

5

45

Projectplan dat flexibel is en nadenkt over verschillende scenario's (innovatie is niet voorspelbaar)

4,00

1,08

13

4

1

5

46

Goede consensus wat bereikt/ geïnnoveerd/ opgelost moet worden

4,00

1,15

13

3

2

5

47

Formuleren van een helder idee / doel (succesfactor)

4,00

1,15

13

4

1

5

48

Durven en willen leren als organisatie (personen en bijbehorende leercultuur)

3,92

0,86

13

2

3

5

49

Goede draagvlakanalyse (succesfactor)

3,92

0,86

13

2

3

5

50

Een op te lossen probleem

3,92

0,95

13

2

3

5

51

Draagvlak gebruikers

3,92

0,95

13

3

2

5

52

Daadwerkelijke betrokkenheid van docenten, van het begin af aan

3,92

0,95

13

3

2

5

53

Contact met financierder / opdrachtgever / subsidiegever

3,92

1,04

13

3

2

5

54

Cultuur/klimaat van verandering

3,92

1,04

13

3

2

5

55

Projectplan is wel richtinggevend doch geen dwangbuis

3,92

1,12

13

3

2

5

56

Is de benodigde expertise voor implementatie aanwezig?

3,92

1,19

13

4

1

5

57

In hoeverre kan een innovatie gaande de implementatie worden aangepast op wensen (optimalisatievraag!)

3,92

1,26

13

4

1

5

58

Kwaliteit van omgaan met weerstanden en opportuniteit

3,92

1,32

13

4

1

5

59

Weten voor wie het is bedoeld en wat er mee kan worden gedaan (succesfactor)

3,85

0,69

13

2

3

5

60

Motieven van leerlingen om innovaties te gebruiken: aansluiting bij hun ervaringen, meerwaarde

3,85

0,69

13

2

3

5

61

Voortrekkersrol voor management

3,85

0,80

13

2

3

5

62

Aansluiting op de praktijk

3,85

0,80

13

2

3

5

63

Nut en motief van de betrokkenen in primair proces

3,85

0,80

13

3

2

5

64

Helderheid in besluitvorming van (bijgestelde) plannen

3,85

0,90

13

2

3

5

65

Ontbreken van / onvoldoende rekening houden met de directe omgeving van de organisatie (faalfactor)

3,85

0,90

13

3

2

5

110

The Road to Sustainable Educational Innovation 66

Expliciete veranderstrategie en veranderaanpak

3,85

0,99

13

3

2

5

67

Het gemeenschappelijk gevoel dat "men" met een innovatief, vernieuwend project bezig is

3,85

0,99

13

3

2

5

68

Positief kriticisme. Anders denkenden. "challengers"

3,85

0,99

13

3

2

5

69

Goed gestructureerde projectorganisatie

3,85

0,99

13

3

2

5

70

Veel aandacht voor "collective concept formation"; gezamenlijk inhoud geven aan "het nieuwe"

3,85

1,07

13

4

1

5

71

Zicht op veranderbereidheid verandervermogen van organisatie, betrokkenen

3,85

1,14

13

3

2

5

72

Een team maken van een willekeurige startgroep (succesfactor)

3,85

1,14

13

4

1

5

73

Institutioneel management moet minimaal begrijpen waar het om gaat

3,85

1,46

13

4

1

5

74

Projectleider moet naast verantwoordelijkheid voor het proces ook gezien worden als inhoudelijk op de hoogte (minimaal)

3,77

0,73

13

2

3

5

75

Toereikende middelen

3,77

0,93

13

3

2

5

76

Invloed geven en laten nemen van en door betrokkenen

3,77

0,93

13

3

2

5

77

Ontbreken van contacten met (eind)gebruikers en geen inzamelen van hun terugkoppeling

3,77

0,93

13

3

2

5

78

Integrale blik op het project: strategisch, organisatie, systemen en cultuur

3,77

0,93

13

3

2

5

79

"Follow up" innovatie, duurzamer maken; minder aan project gebonden

3,77

0,93

13

3

2

5

3,77

0,93

13

3

2

5

80

Speelruimte bij uitvoering / In hoeverre kunnen betrokkenen de innovatie uitproberen, er mee spelen? Hoe meer des te beter. Geldt vooral voor complexe software

81

'verkopen' / uitleggen aan belanghebbenden vanaf de start (succesfactor)

3,77

1,01

13

3

2

5

82

Participatief ontwerpen: de gebruiker als medeontwerper

3,77

1,01

13

3

2

5

83

Zorgen dat er een 'champion' is voor het project, de projectleider of een 'boegbeeld' (succesfactor)

3,77

1,01

13

3

2

5

84

Relatie met visie instelling

3,77

1,09

13

3

2

5

3,77

1,09

13

3

2

5 5

85

Hoe goed past een project binnen de instellingscultuur? Hoe beter des te groter de slaagkans. Cultuur moet je breed opvatten, opvattingen van mensen, de instelling als geheel (beleid), behoeften (idem), verwachtingen

86

Poor / Strong project management performance

3,77

1,09

13

3

2

87


3,77

1,09

13

4

1

5

88

Gebrek aan belangstelling van instellingsmanagement voor 'facultair' project (faalfactor)

3,77

1,09

13

4

1

5

89

Aandacht voor percepties van betrokkenen: hoe wordt innovatie gewaardeerd en gebruikt

3,77

1,24

13

3

2

5

90

Heldere taken en bevoegdheden

3,77

1,24

13

4

1

5

91

Doing a job just because it must be done (faalfactor)

3,77

1,30

13

4

1

5

92

Maak middel niet tot doel

3,77

1,36

13

3

2

5

93

Having no / an incompetent / indecisive owner / principal (faalfactor)

3,77

1,36

13

4

1

5

3,77

1,36

13

4

1

5

94

Hoe betrouwbaar is de politieke steun voor een innovatietraject? Matchen de agenda's van de innovatie-uitvoerders en de fondsverschaffers voldoende? Zijn er verborgen agenda's?

95

Kennis en vaardigheid m.b.t. organisatieontwikkeling en veranderingsprocessen

3,69

0,75

13

2

3

5

96

"Beter ten halve gekeerd dan ten hele gedwaald". M.a.w. goede kwaliteitszorg

3,69

0,75

13

3

2

5

3,69

0,85

13

2

3

5

3,69

0,95

13

3

2

5

97 98

Wat levert innovatie op voor "de buitenwereld": klanten, arbeidsorganisaties, educatieve partners? Meer succes als er meer winst is voor anderen. Goede informatievoorzieningen; "verkopen van het project"; imago building

111


Een rechte rug

3,69

1,11

13

3

2

100

Ontwerp- en ontwikkelbenadering combineren

3,69

1,11

13

3

2

5

101

Open en heldere relatie en betrokkenheid tussen de participerende organisaties

3,69

1,11

13

4

1

5

102

Functioneren techniek / Kwaliteit technische infrastructuur

3,69

1,18

13

3

2

5

103

Procesmanagement

3,69

1,18

13

4

1

5

104

Inspirerende prototypen van de voorgenomen innovatierichting

3,69

1,25

13

3

2

5

105

Acceptatie

3,67

1,44

12

4

1

5

106

Plan strak, en ga daar vervolgens soepel mee om

3,62

0,65

13

2

3

5

107

Diagnose "stavaza" (stand van zaken; red.) en deze delen met betrokkenen

3,62

0,87

13

3

2

5

108

Consensus: in welke mate is een overeenstemming bij de betrokkenen over het probleem en de oplossingsrichting?

3,62

0,96

13

3

2

5

109

Multidisciplinair naar innovatie kijken, denken en doen

3,62

1,04

13

3

2

5

110

Ruimte voor experimenten (geen afhankelijkheid van onderwijsorganisatie)

3,62

1,12

13

3

2

5

3,62

1,12

13

3

2

5

Innovatietrajecten dienen scherp onderscheid te maken tussen piloteren en implementeren. Consequenties voor de

111

bemensing, sturing, implementatietempo, …

5

112

Politiek bewustzijn / management bewustzijn en beïnvloeding en betrokkenheid

3,62

1,19

13

3

2

5

113

Organisatieontwerp en -ontwikkeling passend bij innovatie primair proces

3,62

1,26

13

3

2

5

114

Infrastructuur: (a) technisch, (b) professionalisering, (c) financieel, (d) medezeggenschap

3,54

0,88

13

2

3

5

115

De "bekende" factoren uit de innovatie diffusie literatuur (Rogers: relative advantage, complexity, e.d.)

3,54

0,88

13

3

2

5

3,54

0,88

13

3

2

5

3,54

1,05

13

3

2

5 5

"Infrastructuur"moet adequaat zijn, diverse factoren zoals: (a) kwaliteit van de organisatie, (b) kennis van

116

problematiek/analyse, (c) kennis/vaardigheid m.b.t. oplossingen, (d) financiële middelen, (e) technische ondersteuning, (e) … Ruimte voor en/of stimuleren van "bottom-up" initiatieven; kleinschaligheid; individueel-cultureel i.p.v. technologisch

117

perspectief.

118

Onvoldoende balans in ICT-Onderwijskundige en organisatie doelstellingen (faalfactor)

3,54

1,13

13

3

2

119

Support management

3,54

1,13

13

3

2

5

120

Globaal overzicht innovatietraject en plan op hoofdlijnen

3,54

1,13

13

4

1

5

121

Onvoldoende inzicht / ervaring met ontwikkelprocessen en fasering/methodieken (faalfactor)

3,54

1,13

13

4

1

5

122

Doing a project in a project aversive environment (faalfactor)

3,54

1,33

13

4

1

5

123

Niet voldoende controle over middelen (mensen!) van participerende partijen (docenten bijvoorbeeld)

3,50

1,51

12

4

1

5

3,46

0,66

13

2

2

4 5

Mate van gedetailleerdheid van het projectplan (te veel details: bureaucratie / star; te weinig: slecht meetbaar /

124

onvoorspelbaar)

125

Benut onvoorziene kansen die zich voordoen

3,46

0,88

13

3

2

126

Betekenis van innovatie voor loopbaan van student en opbrengst voor de buitenwereld

3,46

0,97

13

3

2

5

127

Positionering van het project binnen de totale onderwijswaardeketen

3,46

1,05

13

3

2

5

128

Onderscheid maken tussen (a) strategische, (b) tactische en (c) operationele innovatie

3,46

1,05

13

3

2

5

129

Baseren op goede deskresearch / bewezen benefits uit onderzoek (succesfactor)

3,46

1,05

13

4

1

5

130

Balans tussen innovatie, visie, organisatie en cultuur

3,46

1,13

13

3

2

5

131

Globale / vage noties over wat er ontwikkeld moet worden (faalfactor)

3,46

1,13

13

3

2

5

112

The Road to Sustainable Educational Innovation 132 133

Creativiteit van deelnemers HRD beleid in organisaties: individueel leren verbinden met organisatieontwikkeling (individuele en collectieve zones van de naaste ontwikkeling)

3,46

1,20

13

3

2

5

3,46

1,20

13

4

1

5 5

134

Verankering knowhow in eigen organisatie

3,46

1,20

13

4

1

135

Creatief denken

3,46

1,27

13

3

2

5

136

Persoonlijk belang projectleden bij de projectuitkomsten

3,46

1,27

13

4

1

5

137

Omgaan met kritiek uit de omgeving: terechte beantwoorden, onterechte afsluiten

3,46

1,27

13

4

1

5

138

Denken voorbij de grenzen van het bestaande

3,46

1,27

13

4

1

5

139

Streven naar duurzame innovatie (2e orde verandering)

3,46

1,27

13

4

1

5

140

Het leerproces van de docent t.a.v. nieuwe opvattingen, kennis, vaardigheden

3,46

1,33

13

4

1

5

141

Oplossen van een niet bestaand probleem is heel vaak reden voor het mislukken van het implementatietraject

3,46

1,45

13

4

1

5

142

Cultuur van de onderwijsorganisatie en docenten/studenten/leiding (bijvoorbeeld lerende organisatie, ondernemend, …)

3,38

0,96

13

3

2

5

143

Effectieve projectsystematiek (geen "papieren tijger")

3,38

1,04

13

4

1

5

144

Complexiteit van de innovatie (bijv. beamer en powerpoint versus digitale portfolio in universitair systeem)

3,38

1,12

13

3

2

5

3,38

1,19

13

4

1

5

3,38

1,19

13

4

1

5

3,31

0,85

13

3

2

5

145 146 147

Te gretig te blijven innoveren, (i.p.v. inbedding of evaluatie pilots nog een stapje erboven op; niet afbakenen v/h concrete resultaat) (faalfactor) Sterke sturing door leidinggevende Duidelijke taken verdeling tussen inhoudelijk projectbetrokkenen en organisatorische projectbetrokkenen (niet teveel dynamiek hiertussen).

148

Voldoende creatief intern en extern gericht PR beleid (succesfactor)

3,31

0,95

13

3

2

5

149

Aandacht voor professionele identiteit docent: zien ze zichzelf als "leraar" of als "innovator"

3,31

1,03

13

4

1

5

150

Lange adem en ruimte voor iterativiteit en dynamiek van het proces

3,31

1,11

13

3

2

5

3,31

1,18

13

3

2

5

151

Bij het genereren van oplossing voor problemen is het noodzakelijk ook te bepalen welke additionele problemen deze oplossingen met zich meebrengen

152

Perceptie van de docent in het nieuwe onderwijs

3,31

1,25

13

4

1

5

153

Onvrede met bestaande gang van zaken of met trend in ontwikkeling (toekomstige toestand)

3,31

1,25

13

4

1

5

154

Student zien als mede-ontwerper / change-agent

3,31

1,32

13

4

1

5

155

Scheid innovatie van het lopend bedrijf: innovatie verstoort

3,31

1,44

13

4

1

5

156

'mooi weer' rapportages die (uit angst voor stopzetten subsidie) de problemen verzwijgen / toedekken (faalfactor)

3,31

1,44

13

4

1

5

157

Gerelateerd aan de business

3,27

0,90

11

3

2

5

3,23

0,93

13

3

2

5

158

Innovaties die kleine stapjes nemen, niet veel decentrale sturing onder centrale regie, waarbij probleemeigenaren het gevoel hebben en houden mee te denken en werken aan de oplossing, hebben meer kans

159

Druk van buiten af helpt bij het duidelijk krijgen van prioriteiten

3,23

0,93

13

3

2

5

160

Relevante kennis van leren / Onderwijspsychologische kennis

3,23

0,93

13

3

2

5

161

Voldoende (doch niet teveel) middelen (succesfactor) / Een overdaad aan beschikbare middelen schaadt

3,23

1,09

13

3

1

4

162

Goede band met support van opdrachtgever (SURF platform management) (succesfactor)

3,23

1,09

13

4

1

5

163

Duidelijkheid over onderscheid staande versus nieuwe organisatie en innovatie en innovatieonderwerp

3,23

1,17

13

4

1

5

113


Veranderen is een constante en dat moet je meenemen in het personeelsbeleid

3,23

1,17

13

4

1

165

(not) agreeing at the start what success really is

3,23

1,24

13

4

1

5

166

Mate waarin 'leren van studenten' prioriteit heeft bij betrokkenen boven 'onderwijzen door docenten'

3,23

1,24

13

4

1

5

167

Ambitie versus haalbaarheid op korte, middellange termijn

3,23

1,30

13

4

1

5

168

Organisatieverandering

3,23

1,36

13

4

1

5

169

(in)sufficient focus

3,23

1,42

13

4

1

5

170

Institutioneel management dient zich niet bezig te houden met operationele zaken

3,23

1,42

13

4

1

5

171

Zijn de technische/technologische risico's voldoende bekend en uitgeschreven?

3,17

1,11

12

3

1

4

172

Weak / Strong alliance

3,17

1,47

12

4

1

5

173

Duidelijke organisatiestructuur

3,15

0,90

13

3

2

5

174

Project niet geïsoleerd, maar in een 'stream' van vergelijkbare projecten: =aanspraak, =gezamenlijke profilering (succesfactor) 3,15

1,14

13

4

1

5

175

Ervaring van betrokkenen met innovatie in onderwijs en andere professionele praktijken: beroep/ discipline/ onderzoek …

3,15

1,14

13

4

1

5

176

Toepassen van praktische inzichten over groepsdynamica in het projectteam, en naar individuen in hun rol (succesfactor)

3,15

1,14

13

4

1

5

3,15

1,21

13

4

1

5

Innovaties die de flexibiliteit v.h. systeem vergroten hebben meer kans van slagen (massa-individualisering en maatwerk is

177

belangrijk in het onderwijs)

5

178

Fit op type organisatie (WO, HBO, BVE) en cultuur

3,08

0,95

13

3

1

4

179

Een doorwrocht projectplan

3,08

1,32

13

4

1

5

180

Kennis operationele processen

3,00

1,00

13

3

1

4

181

Zone van de naast nabije ontwikkeling (d.w.z. niet te ver op de troepen vooruit)

3,00

1,15

13

3

1

4

182

Hoewel innovatie, al beeld hebben van product, afnemers en exploitatie model (succesfactor)

3,00

1,22

13

4

1

5

183

Metafactor: fase van innovatieproces en invoering van de innovatie

3,00

1,47

13

4

1

5

184

Transformatief leiderschap is voorwaarde

2,92

1,12

13

3

1

4

185

Ruime financiering

2,92

1,19

13

3

1

4

186

Starting / finishing at the wrong place / moment (faalfactor)

2,92

1,44

13

4

1

5

187

Doing too many projects simultaneously (faalfactor)

2,92

1,44

13

4

1

5

188

Niet praten, maar doen

2,92

1,50

13

4

1

5

2,92

1,55

13

4

1

5 5

Schaalgrootte van de innovatie implementatie (bijv. 1 vak in 1 universiteit voor 100 studenten versus alle Bachelors voorzien

189

van gebruik van digitale portfolio's voor 25000 studenten in grote universiteit

190

Project door niet teveel intensief samenwerkende partijen. (commitment)

2,92

1,24

12

4

1

191

Wishful thinking / not aware of insufficient knowledge / means / funds (faalfactor)

2,92

1,51

12

4

1

5

192

IT'ers -> !! meenemen !!

2,91

1,22

11

4

1

5

193

Toegevoegde waarde in opgeleverde knowhow bij studenten als trigger voor vakdocenten

2,85

1,14

13

4

1

5

2,85

1,14

13

4

1

5

Past bij doelstellingen van subsidiegever (OCW), betekent exposure op relevante plekken (kenniscirculatie, "e-learning",

194

transformatie) (succesfactor)

195

Als projectleider heb je niet meer ruimte dan de organisatiecultuur je toelaat

2,85

1,28

13

4

1

5

196

Proces van (a) bottom up, (b) "expansive learning" en (c) "cross boundary"

2,85

1,34

13

4

1

5

197

Constructieve frictie

2,85

1,34

13

4

1

5

114


Teveel sociaal aspect; roesteren in het projectteam (faalfactor)

2,85

1,34

13

4

1

5

199

Accent leggen op beheersaspecten i.p.v. de doelen en inhouden (faalfactor)

2,85

1,41

13

4

1

5

200

Af is af

2,82

1,40

11

4

1

5

201

Minstens vijf jaar de innovatie ongewijzigd uitvoeren

2,77

1,36

13

4

1

5

202

Blikverruiming, bredere oriëntatie van betrokkenen op mogelijkheden en problemen MAAR OOK analyse toekomstige situatie

2,75

1,06

12

3

1

4

203

(in)sufficient momentum

2,75

1,42

12

4

1

5

204

Aansluiting bij andere projecten / ontwikkelingen

2,69

0,85

13

3

1

4

205

Betrekken van de 'distributed knowledge' / "cross boundaries"

2,69

1,44

13

4

1

5

206

ICT: professionele tools; authenticiteit, ook in opdrachten

2,64

1,12

11

3

1

4

207

Spanning tussen regulier onderwijs en nieuwe onderwijs: keuzes maken en doen

2,62

1,26

13

4

1

5

208

In hoeverre is een innovatie een revolutie versus een evolutie. Eerst is veel risicovoller, lastiger te sturen ook dan tweede

2,54

1,05

13

3

1

4

209

Innovaties hebben meer kans van slagen als innovatieagenda wordt gekoppeld aan kennisagenda

2,54

1,27

13

4

1

5

210

Uitschakelen middenmanagement

2,54

1,56

13

4

1

5

211

Relatie tussen discrepantie 'nu' en toekomst en daarvan de relatie tussen taakverdeling, cultuur, regels en procedures

2,50

1,45

12

4

1

5

2,46

1,33

13

4

1

5

212

Projectorganisatie over laten lopen in de exploitatie - organisatie levert op korte termijn voordelen en op langere termijn nadelen op

213

Grens stapelgedrag af

2,38

1,19

13

4

1

5

214

Onderwerp / doel past bij de hype van het moment (in 2000: Eportfolio; in 2004: wireless?) (succesfactor)

2,38

1,26

13

4

1

5

215

Vooral niet meer spreken over draagvlak en weerstand

2,31

1,11

13

3

1

4

216

Mobiliteit van personeel, verjonging van personeel

2,31

1,25

13

3

1

4

217

Onderwijs versus kennisontwikkeling door docenten

2,25

1,14

12

3

1

4

218

Huidige (nu) 'tools' en mogelijke tools

2,25

1,14

12

3

1

4

219

Gezamenlijke training / reflectie van projectleiders (bij SURF: EXchange programma)

2,23

0,83

13

3

1

4

220

Laboratories for innovation, e.g. Honours Programmes

2,17

1,34

12

3

1

4

115


Appendix IV - Clusters Clusters, factors per cluster, rating values, bridging values, cluster rating values and cluster bridging values. Cluster 1 Expertise and creativity of the team Factor ID Factor

Rating

Bridging

60

Deskundigheid binnen team

4,33

0,29

36

Open cultuur bij medewerkers van project

4,31

0,44

215

Inhoudelijk / organisatorisch etc. competente medewerkers binnen projecten

4,23

0,27

88

Teamvorming bij de direct betrokkenen

4,15

0,36

155

Bespreekbaar maken van weerstand, politiek en retoriek

4,08

0,47

94

Cultuur/klimaat van verandering

3,92

0,64

195

Positief criticisme. Anders denkenden. “challengers”

3,85

0,43

208

Ruimte voor en/of stimuleren van “bottom-up” initiatieven; kleinschaligheid; individueel-cultureel i.p.v. technologisch perspectief.

3,54

0,43

1

Creativiteit van deelnemers

3,46

0,31

46

Creatief denken

3,46

0,34

63

Complexiteit van de innovatie (bijv. beamer en Powerpoint versus digitale portfolio in universitair systeem)

3,38

1,00

72

Relevante kennis van leren / Onderwijspsychologische kennis

3,23

0,43

66

3,15

0,47

2,92

1,00

163

Ervaring van betrokkenen met innovatie in onderwijs en andere professionele praktijken: beroep/ discipline/ onderzoek … Schaalgrootte van de innovatie implementatie (bijv. 1 vak in 1 universiteit voor 100 studenten versus alle Bachelors voorzien van gebruik van digitale portfolio’s voor 25000 studenten in grote universiteit) IT’ers -> !! meenemen !!

2,91

0,48

212

Teveel sociaal aspect; rosteren in het projectteam (faalfactor)

2,85

0,40

179

Huidige (nu) ‘tools’ en mogelijke tools

2,25

0,62

3,53

0,49

116

Average: Cluster 2 Competence of the project manager Factor ID Factor

Rating

Bridging

82

De projectleider moet competent zijn

4,46

0,17

213

Volledig toegewijde, kundige projectleider

4,38

0,17

91

Is de benodigde expertise voor implementatie aanwezig?

3,92

0,43

181

Kwaliteit van omgaan met weerstanden en opportunities

3,92

0,38

90

Een team maken van een willekeurige startgroep (succesfactor)

3,85

0,32

119

Projectleider moet naast verantwoordelijkheid voor het proces ook gezien worden als inhoudelijk op de hoogte (minimaal)

3,77

0,17

98

Kennis en vaardigheid m.b.t. organisatieontwikkeling en veranderingsprocessen

3,69

0,36

150

Een rechte rug

3,69

0,34

117


‘mooi weer’ rapportages die (uit angst voor stopzetten subsidie) de problemen verzwijgen / toedekken (faalfactor)

3,31

0,52

19

Kennis operationele processen

3,00

0,31

149

Niet praten, maar doen

2,92

0,35

4

Wishful thinking / not aware of insufficient knowledge / means / funds (faalfactor)

2,92

0,71

201

Als projectleider heb je niet meer ruimte dan de organisatiecultuur je toelaat

2,85

0,58

219

ICT: professionele tools; authenticiteit, ook in opdrachten

2,64

0,64

203

Gezamenlijke training / reflectie van projectleiders (bij SURF: EXchange programma)

2,23

0,37

3,44

0,39

Average: Cluster 3 Focus and complexity Factor ID Factor

Rating

Bridging

40

(vanaf het begin) bewustwording van verandernoodzaak en werken aan conceptwisseling / paradigmashift

4,15

0,38

110

Analyse van hoe zaken nu gaan; en waarom ze gebeuren zoals ze gebeuren en niet zoals gewenst

4,15

0,39

162

4,15

0,42

3,77

0,36

96

Betrokkenheid in de organisatie Speelruimte bij uitvoering / In hoeverre kunnen betrokkenen de innovatie uitproberen, er mee spelen? Hoe meer des te beter. Geldt vooral voor complexe software Open en heldere relatie en betrokkenheid tussen de participerende organisaties

3,69

0,44

3

Multidisciplinair naar innovatie kijken, denken en doen

3,62

0,37

27

Consensus: in welke mate is een overeenstemming bij de betrokkenen over het probleem en de oplossingsrichting?

3,62

0,41

48

112

Diagnose “stavaza” (stand van zaken; red.) en deze delen met betrokkenen

3,62

0,40

174

De “bekende” factoren uit de innovatie diffusie literatuur (Rogers: relative advantage, complexity, e.d.)

3,54

0,43

37

(in)sufficient focus

3,23

0,30

78

Project door niet teveel intensief samenwerkende partijen. (commitment)

2,92

0,37

151

Vooral niet meer spreken over draagvlak en weerstand

2,31

0,48

3,56

0,39

Average: Cluster 4 Start small and then go for it! Factor ID Factor

Rating

Bridging

193

In hoeverre kan een innovatie gaande de implementatie worden aangepast op wensen (optimalisatievraag!)

3,92

0,24

142

Zicht op veranderbereidheid verandervermogen van organisatie, betrokkenen

3,85

0,43

126

Maak middel niet tot doel

3,77

0,25

29

Innovatietrajecten dienen scherp onderscheid te maken tussen piloteren en implementeren. Consequenties voor de bemensing, sturing, implementatietempo, …

3,62

0,19

173

Baseren op goede deskresearch / bewezen benefits uit onderzoek (succesfactor)

3,46

0,34

202

Druk van buiten af helpt bij het duidelijk krijgen van prioriteiten Innovaties die kleine stapjes nemen, niet veel decentrale sturing onder centrale regie, waarbij probleemeigenaren het gevoel hebben en houden mee te denken en werken aan de oplossing, hebben meer kans Zone van de naast nabije ontwikkeling (d.w.z. niet te ver op de troepen vooruit)

3,23

0,42

3,23

0,25

3,00

0,26

209 64

118


Starting / finishing at the wrong place / moment (faalfactor)

2,92

0,27

136

Constructieve frictie

2,85

0,30

30

Af is af

2,82

0,29

152

Grens stapelgedrag af

2,38

0,39

3,25

0,30

Rating

Bridging

Average: Cluster 5 Participation, information, communication Factor ID Factor 41

Vier je successen

4,46

0,32

55

Positieve ambitie bij betrokkenen

4,38

0,32

24

Informeren

4,23

0,25

33

Duidelijke frequente communicatie voor, tijdens en na project

4,15

0,25

218

Participatie van betrokkenen tijdens traject

4,15

0,22

44

Goede draagvlakanalyse (succesfactor)

3,92

0,35

103

Daadwerkelijke betrokkenheid van docenten, van het begin af aan

3,92

0,17

196

Het gemeenschappelijk gevoel dat “men” met een innovatief, vernieuwend project bezig is

3,85

0,30

10

Participatief ontwerpen: de gebruiker als medeontwerper

3,77

0,35

34

‘verkopen’ / uitleggen aan belanghebbenden vanaf de start (succesfactor)

3,77

0,49

111

Invloed geven en laten nemen van en door betrokkenen

3,77

0,22

93

Goede informatievoorzieningen; “verkopen van het project”; imago building

3,69

0,25

220

Acceptatie

3,67

0,29

156

Omgaan met kritiek uit de omgeving: terechte beantwoorden, onterechte afsluiten

3,46

0,50

187

Persoonlijk belang projectleden bij de projectuitkomsten

3,46

0,45

70

Aandacht voor professionele identiteit docent: zien ze zichzelf als “leraar” of als “innovator”

3,31

0,34

74

Student zien als mede-ontwerper / change-agent

3,31

0,32

115

Voldoende creatief intern en extern gericht PR beleid (succesfactor)

3,31

0,32

42

Betrekken van de ‘distributed knowledge’ / “cross boundaries”

2,69

0,36

3,75

0,32

Average: Cluster 6 Stakeholder involvement Factor ID Factor

Rating

Bridging

50

Betrokkenheid afnemers (docenten, studenten, bestuurders)

4,31

0,25

153

Studenten en “klanten” betrekken bij innovaties

4,00

0,23

16

Draagvlak gebruikers

3,92

0,32

49

Veel aandacht voor “collective concept formation”; gezamenlijk inhoud geven aan “het nieuwe”

3,85

0,26

53

Ontbreken van contacten met (eind)gebruikers en geen inzamelen van hun terugkoppeling

3,77

0,42

119


Aandacht voor percepties van betrokkenen: hoe wordt innovatie gewaardeerd en gebruikt

3,77

0,35

190

Doing a job just because it must be done (faalfactor)

3,77

0,51

137

Het leerproces van de docent t.a.v. nieuwe opvattingen, kennis, vaardigheden

3,46

0,49

186

Onderwijs versus kennisontwikkeling door docenten

2,25

0,52

3,68

0,37

Rating

Bridging

Average: Cluster 7 Position of innovation within the organisation Factor ID Factor 123

Doelen van het project liggen in lijn met de activiteiten van de organisatie (succesfactor)

4,15

0,41

67

Doelstellingen helder

4,00

0,39

122

Formuleren van een helder idee / doel (succesfactor)

4,00

0,36

188

Ontbreken van / onvoldoende rekening houden met de directe omgeving van de organisatie (faalfactor)

3,85

0,43

86

Relatie met visie instelling

3,77

0,40

148

“Follow up” innovatie, duurzamer maken; minder aan project gebonden

3,77

0,34

158

Onvoldoende balans in ICT-Onderwijskundige en organisatie doelstellingen (faalfactor)

3,54

0,42

172

Doing a project in a project aversive environment (faalfactor)

3,54

0,37

7

Streven naar duurzame innovatie (2e orde verandering)

3,46

0,41

38

Positionering van het project binnen de totale onderwijswaardeketen

3,46

0,32

105

Te gretig te blijven innoveren, (i.p.v. inbedding of evaluatie pilots nog een stapje erboven op; niet afbakenen v/h concrete resultaat) (faalfactor)

3,38

0,36

69

Bij het genereren van oplossing voor problemen is het noodzakelijk ook te bepalen welke additionele problemen deze oplossingen met zich meebrengen

3,31

0,42

183

Ambitie versus haalbaarheid op korte, middellange termijn

3,23

0,39

206

(in)sufficient momentum

2,75

0,39

178

Spanning tussen regulier onderwijs en nieuwe onderwijs: keuzes maken en doen

2,62

0,39

160

Innovaties hebben meer kans van slagen als innovatieagenda wordt gekoppeld aan kennisagenda

2,54

0,48

168

In hoeverre is een innovatie een revolutie versus een evolutie. Eerst is veel risicovoller, lastiger te sturen ook dan tweede

2,54

0,37

131

Onderwerp / doel past bij de hype van het moment (in 2000: Eportfolio; in 2004: wireless?) (succesfactor)

2,38

0,34

85

Laboratories for innovation, e.g. Honours Programmes Average:

2,17

0,37

3,29

0,39

Rating

Bridging

Cluster 8 Added value Factor ID Factor Zichtbaarheid toegevoegde waarde / Hoe zichtbaar zijn de voordelen van een innovatie; kwestie van PR, maar ook keuze van accenten. Hoe zichtbaarder hoe 35 beter. 107 In hoeverre lost een innovatie een probleem op, hoe meer men denkt dat dit zo is des te beter (de innovatie is niet alleen nieuw maar ook beter)

4,54

0,16

4,46

0,05

11

Toegevoegde waarde moet snel (liefst vanaf het begin) duidelijk zijn voor een ieder

4,15

0,17

14

Gedeelde visie over het wat en het hoe

4,15

0,28

120


Ervaren meerwaarde voor betrokkenen: wat wordt student, docent, management ‘beter’ van de betreffende innovatie?

4,15

0,20

43

Duidelijk bijdragen aan de performance (verbetering) van de lerende / de werkende

4,15

0,07

39

De kwaliteit van de innovatie, de meerwaarde t.a.v. bestaande oplossingen

4,08

0,00

118

Analyse van wat men wil bereiken

4,08

0,38

170

Duidelijke noodzaak en nut van de innovatie waardoor “energie” van mensen vrijkomt

4,08

0,07

65

Goede consensus wat bereikt/ geïnnoveerd/ opgelost moet worden

4,00

0,32

120

Overeenstemming m.b.t. doelen van project

4,00

0,27

52

Motieven van leerlingen om innovaties te gebruiken: aansluiting bij hun ervaringen, meerwaarde

3,85

0,01

141

Nut en motief van de betrokkenen in primair proces

3,85

0,27

211

Weten voor wie het is bedoeld en wat er mee kan worden gedaan (succesfactor)

3,85

0,32

177

Inspirerende prototypen van de voorgenomen innovatierichting

3,69

0,32

182

Wat levert innovatie op voor “de buitenwereld”: klanten, arbeidsorganisaties, educatieve partners? Meer succes als er meer winst is voor anderen.

3,69

0,06

20

Denken voorbij de grenzen van het bestaande

3,46

0,44

83

Betekenis van innovatie voor loopbaan van student en opbrengst voor de buitenwereld

3,46

0,09

109

Onvrede met bestaande gang van zaken of met trend in ontwikkeling (toekomstige toestand)

3,31

0,13

180

Perceptie van de docent in het nieuwe onderwijs

3,31

0,28

132

Mate waarin ‘leren van studenten’ prioriteit heeft bij betrokkenen boven ‘onderwijzen door docenten’

3,23

0,18

214

Toegevoegde waarde in opgeleverde knowhow bij studenten als trigger voor vakdocenten

2,85

0,15

84

Blikverruiming, bredere oriëntatie van betrokkenen op mogelijkheden en problemen MAAR OOK analyse toekomstige situatie

2,75

0,40

185

Relatie tussen discrepantie ‘nu’ en toekomst en daarvan de relatie tussen taakverdeling, cultuur, regels en procedures

2,50

0,24

3,73

0,20

Rating

Bridging

Average: Cluster 9 Relationship between project and real problem / core business Factor ID Factor 89

Aantoonbare relatie tussen innovatie en verbetering kwaliteit en kwantiteit leeractiviteiten van studenten

4,23

0,20

113

Probleemeigenaar, ownership: wie heeft er een probleem, hoe urgent is het probleem voor de probleemeigenaar?

4,15

0,30

101

Een op te lossen probleem

3,92

0,10

165

Aansluiting op de praktijk

3,85

0,25

17

Globale / vage noties over wat er ontwikkeld moet worden (faalfactor)

3,46

0,28

108

Oplossen van een niet bestaand probleem is heel vaak reden voor het mislukken van het implementatietraject

3,46

0,15

217

Gerelateerd aan de business

3,27

0,19

124

(not) agreeing at the start what success really is

3,23

0,38

184

Innovaties die de flexibiliteit v.h. systeem vergroten hebben meer kans van slagen (massa-individualisering en maatwerk is belangrijk in het onderwijs)

3,15

0,23

117

Hoewel innovatie, al beeld hebben van product, afnemers en exploitatie model (succesfactor)

3,00

0,30

127

Past bij doelstellingen van subsidiegever (OCW), betekent exposure op relevante plekken (kenniscirculatie, “e-learning”, transformatie) (succesfactor)

2,85

0,30

3,51

0,24

Average:

121


Cluster 10 Organisational culture Factor ID Factor

Rating

Bridging

2

Duidelijke strategische koers, ambitie

4,00

0,32

125

Een heldere opdracht

4,00

0,34

23

3,92

0,36

3,77

0,29

138

Durven en willen leren als organisatie (personen en bijbehorende leercultuur) Hoe goed past een project binnen de instellingscultuur? Hoe beter des te groter de slaagkans. Cultuur moet je breed opvatten, opvattingen van mensen, de instelling als geheel (beleid), behoeften (idem), verwachtingen Organisatieontwerp en -ontwikkeling passend bij innovatie primair proces

3,62

0,29

198

Ruimte voor experimenten (geen afhankelijkheid van onderwijsorganisatie)

3,62

0,32

121

Verankering knowhow in eigen organisatie

3,46

0,34

159

Balans tussen innovatie, visie, organisatie en cultuur

3,46

0,38

167

Onderscheid maken tussen (a) strategische, (b) tactische en (c) operationele innovatie

3,46

0,33

171

HRD beleid in organisaties: individueel leren verbinden met organisatieontwikkeling (individuele en collectieve zones van de naaste ontwikkeling)

3,46

0,36

47

Cultuur van de onderwijsorganisatie en docenten/studenten/leiding (bijvoorbeeld lerende organisatie, ondernemend, …)

3,38

0,33

26

97

Organisatieverandering

3,23

0,28

216

Duidelijkheid over onderscheid staande versus nieuwe organisatie en innovatie en innovatieonderwerp

3,23

0,33

57

Weak / Strong alliance

3,17

0,39

54

Fit op type organisatie (WO, HBO, BVE) en cultuur

3,08

0,32

22

Minstens vijf jaar de innovatie ongewijzigd uitvoeren

2,77

0,38

133

Aansluiting bij andere projecten / ontwikkelingen

2,69

0,27

143

Mobiliteit van personeel, verjonging van personeel

2,31

0,42

3,37

0,34

Average: Cluster 11 Quality of institutional management Factor ID Factor

Rating

Bridging

71

Betrokken en competent management

4,38

0,30

92

Autoriteit, gezag, gaat wel/niet actief en zichtbaar achter de innovatieactiviteiten staan

4,15

0,36

114

Blijvende managementaandacht en monitoring

4,15

0,19

87

Topmanagement is niet alleen facilitator, doch ook voorvechter

4,00

0,25

169

Flexibiliteit in organisatie om tijd en middelen vrij te maken voor innovatie-investeringen

4,00

0,21

45

Institutioneel management moet minimaal begrijpen waar het om gaat

3,85

0,15

210

Voortrekkersrol voor management

3,85

0,24

58

Gebrek aan belangstelling van instellingsmanagement voor ‘facultair’ project (faalfactor) Hoe betrouwbaar is de politieke steun voor een innovatietraject? Matchen de agenda’s van de innovatie-uitvoerders en de fondsverschaffers voldoende? Zijn er verborgen agenda’s? Having no / an incompetent / indecisive owner / principal (faalfactor)

3,77

0,22

3,77

0,33

3,77

0,36

164 192

122


3,54

0,24

3,54

0,24

129

Infrastructuur: (a) technisch, (b) professionalisering, (c) financieel, (d) medezeggenschap “Infrastructuur”moet adequaat zijn, diverse factoren zoals: (a) kwaliteit van de organisatie, (b) kennis van problematiek/analyse, (c) kennis/vaardigheid m.b.t. oplossingen, (d) financiële middelen, (e) technische ondersteuning, (e) … Institutioneel management dient zich niet bezig te houden met operationele zaken

3,23

0,15

145

Veranderen is een constante en dat moet je meenemen in het personeelsbeleid

3,23

0,23

28

Duidelijke organisatiestructuur

3,15

0,24

176

Project niet geïsoleerd, maar in een ‘stream’ van vergelijkbare projecten: =aanspraak, =gezamenlijke profilering (succesfactor)

3,15

0,27

6

Uitschakelen middenmanagement

2,54

0,24

3,65

0,25

75

Average: Cluster 12 Project and process management Factor ID Factor

Rating

Bridging

12

Tijd om van het innoveren te leren / Tijd om fouten te mogen maken en te zoeken

4,15

0,33

207

Succesvolle tussenproducten / zichtbare resultaten

4,15

0,27

102

Ontbreken van daadkracht om in te grijpen als resultaten achterblijven, personen nalatig zijn (faalfactor)

4,08

0,32

95

Projectplan dat flexibel is en nadenkt over verschillende scenario’s (innovatie is niet voorspelbaar)

4,00

0,25

191

Eisen dat er een opdrachtgever is die de functionele eisen kan beoordelen / wil hebben (succesfactor)

4,00

0,28

81

Contact met financierder / opdrachtgever / subsidiegever

3,92

0,33

134

Projectplan is wel richtinggevend doch geen dwangbuis

3,92

0,26

135

Goed gestructureerde projectorganisatie

3,85

0,20

205

Helderheid in besluitvorming van (bijgestelde) plannen

3,85

0,20

5

Zorgen dat er een ‘champion’ is voor het project, de projectleider of een ‘boegbeeld’ (succesfactor)

3,77

0,44

13

Poor / Strong project management performance

3,77

0,16

154

Heldere taken en bevoegdheden

3,77

0,16

9

“Beter ten halve gekeerd dan ten hele gedwaald”. M.a.w. goede kwaliteitszorg

3,69

0,30

100

Procesmanagement

3,69

0,17

161

Functioneren techniek / Kwaliteit technische infrastructuur

3,69

0,52

200

Plan strak, en ga daar vervolgens soepel mee om

3,62

0,25

189

Onvoldoende inzicht / ervaring met ontwikkelprocessen en fasering/methodieken (faalfactor)

3,54

0,31

21

Benut onvoorziene kansen die zich voordoen

3,46

0,25

104

Mate van gedetailleerdheid van het projectplan (te veel details: bureaucratie / star; te weinig: slecht meetbaar / onvoorspelbaar)

3,46

0,25

68

Sterke sturing door leidinggevende

3,38

0,43

76

Effectieve projectsystematiek (geen “papieren tijger”)

3,38

0,14

25

Duidelijke taken verdeling tussen inhoudelijk projectbetrokkenen en organisatorische projectbetrokkenen (niet teveel dynamiek hiertussen)

3,31

0,22

32

Lange adem en ruimte voor iterativiteit en dynamiek van het proces

3,31

0,19

80

Goede band met support van opdrachtgever (SURF platform management) (succesfactor)

3,23

0,35

51

Toepassen van praktische inzichten over groepsdynamica in het projectteam, en naar individuen in hun rol (succesfactor)

3,15

0,29

123


Een doorwrocht projectplan

3,08

0,17

130

Transformatief leiderschap is voorwaarde

2,92

0,46

147

Proces van (a) bottom up, (b) “expansive learning” en (c) “cross boundary”

2,85

0,22

3,61

0,28

Average: Cluster 13 Means to support the project Factor ID Factor

Rating

Bridging

146

Sterke sleutelpersonen op cruciale posities

4,17

0,33

61

Facilitering van innovatie: (a) tijd, (b) geld, (c) deskundigheid, (d) middelen

4,00

0,25

99

Expliciete veranderstrategie en veranderaanpak (zie ook 56)

3,85

0,19

31

Integrale blik op het project: strategisch, organisatie, systemen en cultuur

3,77

0,30

56


3,77

0,24

204

Toereikende middelen

3,77

0,26

139

Ontwerp- en ontwikkelbenadering combineren

3,69

0,20

73

Politiek bewustzijn / management bewustzijn en beïnvloeding en betrokkenheid

3,62

0,37

79

Globaal overzicht innovatietraject en plan op hoofdlijnen

3,54

0,29

140

Support management

3,54

0,27

77

Niet voldoende controle over middelen (mensen!) van participerende partijen (docenten bijvoorbeeld)

3,50

0,35

128

Scheid innovatie van het lopend bedrijf: innovatie verstoort

3,31

0,23

144

Voldoende (doch niet teveel) middelen (succesfactor) / Een overdaad aan beschikbare middelen schaadt

3,23

0,34

15

Zijn de technische/technologische risico’s voldoende bekend en uitgeschreven?

3,17

0,30

166

Metafactor: fase van innovatieproces en invoering van de innovatie

3,00

0,23

8

Ruime financiering

2,92

0,31

157

Doing too many projects simultaneously (faalfactor)

2,92

0,32

194

Accent leggen op beheersaspecten i.p.v. de doelen en inhouden (faalfactor)

2,85

0,25

197

Projectorganisatie over laten lopen in de exploitatie - organisatie levert op korte termijn voordelen en op langere termijn nadelen op

2,46

0,27

3,42

0,28

Average:

124


Appendix V - Interview protocol Dit interview zal ongeveer 45 minuten in beslag nemen. Algemene gegevens Naam: Projectleider vanaf begin tot eind: ja/nee Ervaring als projectleider: Ervaring als projectmedewerker bij onderwijsinnovatieprojecten: Algemene vragen 1. Was uw project, in uw ogen, een succes? Waarom wel of niet. 2. Was de innovatie zelf – dus na het project – een succes? Waarom wel of niet Lessons learned SURF project Wat hebt u in dit project geleerd ga hierbij kort in op o.a.: 3. Positieve en negatieve ervaringen van uzelf als projectleider en eventuele feedback van de projectleden / anderen. 4. Uit een literatuurstudie is een aantal faal- en succesfactoren naar voren gekomen. Deze factoren hebben we geclusterd, hieruit is een veertiental hoofdclusters of aspecten naar voren gekomen. Ga per aspect na welke rol dit aspect heeft gespeeld binnen uw project? -

Afbakening inhoud project: de inhoud van het project (e.g. probleem, doelstelling etc) was duidelijk gedefinieerd.

-

Afbakening proces project: het proces van het project (e.g. projectplanning) was duidelijk gedefinieerd.

-

Projectleider: er was een competente, ervaren projectleider aanwezig.

-

Projectmedewerkers: er was een competente groep medewerkers betrokken bij het project en de rollen van hen waren duidelijk verdeeld.

-

Institutioneel management / organisatiemanagement: er was competent management, dat het project steunde en erbij betrokken was.

-

Organisatie waarbinnen wordt geïnnoveerd

125


-

Communicatie

-

Betrokkenheid bij project / innovatie

-

Projectmanagement (proces / activiteiten)

-

Verwachtingen

-

Meerwaarde / noodzaak / nut van de innovatie

-

Projectcultuur: open, flexibel, kritisch, etc.

-

Schaalgrootte / complexiteit

-

Duurzame implementatie van het project

Succesfactoren en faalfactoren eigen project 5. Gezien de bovengenoemde succesfactoren, welke zijn vooral bij dit project aan de orde geweest? Welke andere factoren maakten dit project tot een succes? 6. Wat zou er verbeterd kunnen worden en hoe? 7. Beargumenteer welke van de bovenstaande factoren u als de belangrijkste succesfactor(en) beschouwt en waarom. Reflectie eigen project 8. Kunt u aangeven op een schaal van 1 tot 10 hoe succesvol uw project in uw ogen was? Waarom? 9. Kunt u aangeven op een schaal van 1 tot 10 hoe succesvol SURF uw project volgens u vindt? Waarom? 10. Kunt u aangeven op een schaal van 1 tot 10 hoe succesvol de eindgebruiker uw project volgens u vindt? Waarom? 11. Kunt u aangeven op een schaal van 1 tot 10 in hoeverre de procedurele begeleiding (projectplan, formats etc) van SURF heeft bijgedragen aan het succes van uw project? Wat was er goed en wat kan er verbeterd worden? 12. Kunt u aangeven op een schaal van 1 tot 10 in hoeverre de inhoudelijke begeleiding (feedback, overleg etc) van SURF heeft bijgedragen aan het succes

126


van uw project? Wat was er goed en wat kan er verbeterd worden? 13. Als u opnieuw de kans zou hebben om het project uit te voeren (met wat u nu al weet en hebt ervaren): • Zou u het opnieuw doen? • Zo ja, wat zou u anders doen om het beter te laten verlopen? • Zo neen, waarom niet? Duurzaamheid innovatieve projecten 14. Moet duurzaamheid of duurzame implementatie een belangrijke rol spelen in een innovatief project? 15. Hebt u ideeën over hoe de duurzaamheid van innovatieve projecten gestimuleerd zou kunnen worden? Samenvattend 16. Wat maakte uw project tot een succes en wat was een faalfactor Huidige werkzaamheden? 17. Houdt u in uw huidige werkzaamheden ook rekening met de ‘lessons learned’ van uw SURF project? 18. Indien voor SURF wat gaat er nu goed en wat kan er verbeterd worden? Hartelijk dank voor uw medewerking. Ik zal u binnenkort het verslag van dit interview toesturen. Hierbij stuur ik ook een lijst met factoren mee die tijdens dit interview besproken zijn. Graag zou ik u willen vragen deze factoren te waarderen met een rapportcijfer. Dit zal ongeveer 5 minuten in beslag nemen. Bent u bereid hier ook aan mee te werken? Dan zijn we nu klaar.

127


Appendix VI - Interviews Verslag interview Projectleider 1 – Project 1 Dinsdag 11 januari 2005 Algemene gegevens Projectleider vanaf het begin tot het einde van het project. Hiervoor had hij al ervaringen opgedaan als projectleider en projectmedewerker bij onderwijsinnovatieprojecten. Algemene vragen: Het project heeft veel teweeggebracht en op allerlei vlakken succes geboekt. Er is veel ervaring opgedaan mbt ICT gebruik, samenwerking etc. Het project heeft echter door diverse oorzaken de rechtstreekse doelen (het steeds op twee plekken aanbieden van twee complete curricula) niet volledig behaald. Als gevolg van de prettige samenwerking bij dit project werken veel van de partners nog steeds samen. Lessons learned SURF project: SURF was in eerste instantie niet zo gelukkig met de zware nadruk op het gebruik en de ontwikkeling van ICT-tools maar door de gedeelde financiering kon dit opgevangen worden en is er veel geleerd over de gebruikte instrumenten en hun sterktes en zwaktes. Zo heeft het project ook een groot gedeelte (van met name het ICT gedeelte) gefinancierd en kon SURF zich meer richten op de onderwijskundige kant van het project. De didactische inbedding is belangrijk volgens SURF en de projectleider is het hier voor een deel mee eens, maar het didactische gedeelte moet in een dergelijk project samengaan met het technologische gedeelte en dat is goed gelukt in dit project. De rol van faal- en succesfactoren uit literatuurstudie bij dit project: Bij een project speelt het dilemma van enerzijds helder aangeven van je doelen en anderzijds genoeg ruimte laten voor aanpassingen. Bij dit project is ervoor gekozen om wel te definiëren welke dingen er gedaan moesten worden maar dit niet in detail uit te splitsen hoe dit precies gedaan moet worden. Deze flexibiliteit was belangrijk gezien de verschillen tussen de deelnemende opleidingen. De projectplanning is goed gegaan. Het project is verdeeld in 5 blokken waardoor het project goed verliep ondanks kleine vertragingen. De stuurgroep (vertegenwoordigers van het management van de verschillende organisaties) speelde een belangrijke rol en leverde belangrijke input. De steun van het management (door deze stuurgroep) was doorslaggevend bij het kunnen afronden van het project. De projectleider en 2 subhoofden (een onderwijskundige en een technoloog) hadden twee maal per jaar overleg met deze stuurgroep waarbij problemen werden besproken en eventuele bijsturingacties gepland. Maar ook bij tussentijdse problemen kon teruggevallen worden op de stuurgroep. Hieruit bleek ook de betrokkenheid en steun van het organisatiemanagement. De rapportages naar SURF waren een goede externe stimulans. Het projectteam wisselde van samenstelling maar bestond doorgaans uit 30 – 35 personen. Een zwak punt bij onderwijsinnovatie projecten waarbij ICT te pas komt is volgens de projectleider de inbreng van docenten. Deze is onmisbaar, maar docenten zijn veelal niet gewend om

128


projectmatig te werken. Onderwijsvernieuwing is ook voor veel docenten ondergeschikt aan onderwijs verzorgen en de begeleiding van docenten zal hierdoor ook meer tijd en energie vergen. De communicatie in een project is van doorslaggevend belang en moet op verschillende niveaus gevoerd worden: op regionaal niveau, binnen de instelling, binnen het project, naar SURF toe en binnen subgroepjes en alle vormen en niveaus waren belangrijk. Een communicatieplan is van essentieel belang bij het succes van een innovatief project. De verwachtingen van de docenten speelden een rol en hiermee werd getracht zoveel mogelijk rekening te houden. Dit project was / is bedoeld om vooruit te komen, maar hoe dit gerealiseerd moest worden was bij aanvang nog onduidelijk. De doelen waren groot en een externe evaluatie leverde oa op: je moet zorgen dat je eerst kunt lopen voordat je kunt gaan hardlopen. Dit was een groot leereffect van het project; de doelen waren wellicht te groot; het project was erg complex wat zeker een faalfactor zou kunnen zijn. De duurzame implementatie speelde een belangrijke rol en werd ook in het projectplan opgenomen. De projectcultuur kon omschreven worden als open; er was veel ruimte voor opmerkingen, aanpassingen, creativiteit, etc. Succes- en faalfactoren eigen project: Sterke punten (succesfactoren) waren: betrokken partners, goede communicatie, goede projectgroep (hecht team). Verbeter punten zijn (faalfactor): het was een te groot, complex project. Te ambitieus. Reflectie eigen project: SURF vond net als de projectleider dat het project succesvol was. Het rechtstreekse product zou een onvoldoende krijgen aangezien de directe doelen niet voldoende gehaald zijn en geen van beide opleidingen meer bestaat. De opbrengsten voor de eindgebruikers (docenten) verschillen in nut en de opgeleverde faciliteiten worden ook niet voldoende gebruikt. Toch was dit niet het belangrijkste, er is een heleboel ervaring opgedaan mbt de samenwerking en het gebruik van ICT in het onderwijs. De procedurele begeleiding van SURF heeft zeker bijgedragen aan het succes. De formats van SURF zijn iets aangepast om de rapportages aan alle partijen te kunnen integreren. De inhoudelijke overleggen werden zeer gewaardeerd, streng maar rechtvaardig. De projectleider zou het project wel over willen doen, maar op kleinere schaal. Duurzaamheid innovatieve projecten Een belangrijke factor bij duurzame implementatie van een innovatief project is de commitment van het management, het is echter niet gemakkelijk om uit te vinden hoe sterk deze commitment is. Verder moet de innovatie goed aansluiten op de behoeften van de organisatie, dient het een eigen onderwijskundige verbetering. Een luxe speeltje wordt hoogstwaarschijnlijk niet gebruikt. Dus moet de noodzaak van het project en het product duidelijk zijn.

129


Verslag interview Projectleider 2 – Project 2 Donderdag 13 januari 2005 Algemene gegevens De geïnterviewde is niet van het begin tot het einde projectleider geweest bij dit project. Na advies van de SOS is er in de loop van het project een andere projectleider aangesteld. De geïnterviewde had ervaring als deelprojectleider en projectmedewerker. Algemene vragen: Volgens de projectleider was het project een succes binnen de eigen organisatie. Hier raakte de projectleider toen het project afgerond was na verloop van tijd ook steeds meer van overtuigd aangezien steeds meer mensen gebruik van het product (gaan) maken. Het product was succesvoller dan het proces. Het product was klaar voor gebruik en wordt ook gebruikt. Lessons learned SURF project: De afbakening van de inhoud was erg duidelijk en goed vastgelegd, hier is ook zo veel mogelijk aan vastgehouden. In de loop van het project is er wel een lijst van do’s en don’ts gemaakt die gaandeweg werd aangepast. Veel docenten hadden geen of weinig ervaring met projectmatig werken en dit leverde problemen op bij dit project. Door dit gebrek aan ervaring van de docenten konden zij ook de studenten in het projectmatig werken niet goed begeleiden. Deze twee problemen leverden een vertraging op en vergden extra aandacht wat een nadelig effect had op het proces binnen dit project. De communicatie binnen het project liep erg goed en volgens het vooraf opgestelde communicatieplan. De communicatie is zeer belangrijk volgens de projectleider. Er was nagenoeg geen betrokkenheid en steun vanuit het CvB en (organisatie)management. Dit bemoeilijkte de mogelijkheden om pilots te organiseren, men was niet bereid om hier tijd en moeite in te steken. Sommige docenten waren wel betrokken en zeer geïnteresseerd. Het was van enkele partijen zeer duidelijk wat hun verwachtingen waren en deze lagen ook in de lijn van het project. De projectleider probeerde zo open mogelijk te zijn richting alle partijen, dit zorgde soms voor problemen. Succes- en faalfactoren eigen project: Succesfactoren bij dit project waren onder andere: de geloofwaardigheid van het product, dit product was nuttig, had een duidelijke meerwaarde bij het huidige onderwijs en sloot goed op de behoefte van zowel de studenten als de docenten aan. De goede aansluiting op deze gecombineerde doelgroep (docenten én studenten) versterkte de positieve effect bij beide doelgroepen. Een faalfactor was de veronderstelling dat de docenten zelf projectmatig kunnen werken en studenten in hún projectmatig werken kunnen ondersteunen en sturen. In het begin is vooral gelet op de ICT-vaardigheden en hier werd ook voldoende ondersteuning vanuit het project geregeld. Dit gebrek aan ervaring in projectmatig werken werd ook gerapporteerd in de tussenrapportages en de projectleider heeft een cursus projectmatig werken voor de docenten geregeld.

130


Reflectie eigen project: De projectleider denkt dat er binnen SURF altijd wel een aantal mensen in het project geloofd heeft, anderen waren meer sceptisch ten opzichte van het slagen van het project en enkelen waren vrij negatief ten opzichte van het project. De tussentijdse reviews waren erg nuttig maar de conclusies van de SOS waren soms te bont en te negatief, hier was de projectleider het ook niet mee eens. Een van de adviezen van de SOS was dat er een andere projectleider zou komen, die is er ook gekomen. Volgens de projectleider werd de projectleider alleen in naam veranderd, maar bleef de oorspronkelijke projectleider in functie nog steeds projectleider. Ook had de projectleider het idee dat de spelregels veranderden tijdens de wedstrijd: de evaluatie door de SOS haalde aspecten erbij die niet in de overeenkomst stonden en die wel verwacht werden. Dit was echter geen groot probleem. Maar de verandering van projectleider was niet nodig geweest. De manier waarop de SOS met een grote bezetting zich opstelde tegenover de projectleider was erg overdreven, niet in evenwicht en werd niet als prettig ervaren. Verder werd het commentaar van de SOS op het (door de WTR) goedgekeurde concept ook niet gewaardeerd. Het bezoek van de reviewers werd wel als zeer prettig ervaren. De projectleider zou het project wel opnieuw willen doen, maar hij zou minder open zijn. Hij zou in bijvoorbeeld de tussenrapportages niet te eerlijk de problemen vermelden om zo de voortgang niet te willen verstoren en alleen bij écht grote problemen dit melden naar SURF toe. Inhoudelijk gezien had er meer rekening gehouden moeten worden vanaf het begin met het gebrek aan ervaring van de docenten met projectmatig werken, hij zou beter naar de omgeving en de mogelijkheden hiervan kijken. Duurzaamheid innovatieve projecten Dit project is duurzaam geïmplementeerd, dit komt mede door de duurzame infrastructuur van zowel het project als de organisatie waarbinnen het is geïmplementeerd. Verder is ‘boerenverstand’ een belangrijke factor bij duurzame implementatie. Je moet helder voor ogen hebben wat je wilt, helder de omgeving en organisatie zien en de behoefte van de doelgroep helder voor ogen hebben. Er wordt bij te veel projecten gefocused op de output in de vorm van een product, zonder te kijken naar de behoeftes van de eindgebruiker. Het is cruciaal om vanuit de doelgroep te redeneren.

131


Verslag interview Projectleider 3 – Project 3 Dinsdag 25 januari 2005 Algemene gegevens De geïnterviewde is projectleider geweest van het begin tot het einde van dit project en had al ervaring als projectleider. Algemene vragen: De meeste aspecten van het project waren in de ogen van de projectleider een succes. De producten waren een groot succes, de spin-off was een groot succes, het proces en de efficiëntie van het project hadden soepeler kunnen lopen. Lessons learned SURF project: De samenwerking met SURF was erg goed. De projectleider vond achteraf en tegen zijn eigen verwachting in, de verplichte samenwerking met een andere instelling zeer prettig. Hij heeft hier veel van geleerd. Vooraf verwachte de projectleider geen meerwaarde van de samenwerking. Deze samenwerking resulteerde er onder andere in dat het product op meerdere plaatsen inzetbaar is geworden en zich niet te veel richt op één instelling. De rol van faal- en succesfactoren uit literatuurstudie bij dit project: De afbakening van de inhoud van het project is belangrijk, hier moet wel flexibel mee worden omgegaan en aangepast kunnen worden aan de wensen van de eindgebruiker. De administratieve handelingen zijn door SURF goed afgebakend en worden goed ondersteund, hiervan werd niet veel vanaf geweken. Er was een competente groep medewerkers aanwezig, met name binnen de eigen instelling. De ontwikkelaars hadden daar niet altijd voldoende inhoudelijke expertise. Door de grootte van de twee instellingen kan niet iedereen op de hoogte zijn van het project. Diegene die wel op de hoogte waren, waren ook betrokken en positief. Het management was zeer betrokken en positief. Ook binnen de divisie van de projectleider zelf waren geen punten van kritiek. Er was geen vooraf gemaakt communicatieplan op papier, hier was wel over nagedacht, maar niet als zodanig genoteerd. Dit had achteraf wel gemaakt kunnen worden. Een communicatie plan kan het proces vergemakkelijken maar niet noodzakelijk. Op de website werden alle notulen, rapportages, casussen geplaatst. Dit werd mede gedaan om eventuele invallers goed in te kunnen lichten bij een eventuele uitval van een medewerker. De betrokkenheid van verschillende partijen werd onder andere gerealiseerd door de ruime kennisdisseminatie die mogelijk wordt gemaakt binnen SURF projecten. Er werd dmv artikelen voor bekendheid gezorgd binnen de verschillende instellingen. Verder werd het project op nationale en internationale congressen gepresenteerd. Vooraf is er een informele behoefte analyse onder docenten gehouden, met de resultaten van deze analyse is rekening gehouden bij het opstellen van het projectplan. Bij aanvang van het project is er rekening gehouden met de duurzame implementatie van het project. Dit is echter moeilijk bij een dergelijk innovatief project. Succes- en faalfactoren eigen project: Succesfactoren waren onder andere: het idee, de behoefte en maatschappelijke belangrijkheid van het project, de authenticiteit van de casussen (real-time verlopen van een

132


casus, dit vergrootte de betrokkenheid van de studenten enorm). Een faalfactor was: de uitval van leden van het team (capaciteit van de instelling). Dit kan helaas van te voren niet goed ingedekt worden, wel werd er gezorgd voor goede documentatie om dit zo goed mogelijk op te vangen. Reflectie eigen project: Het project krijgt van de projectleider een 8. SURF zal volgens de projectleider een hoger cijfer geven aangezien er veel administratieve methodes van dit project terugkomen in de standaard methodes van SURF. SURF heeft waarschijnlijk geleerd van de methodes van dit project. De eindgebruiker zal volgens de projectleider ook een 8 geven aangezien de studenten ook erg enthousiast waren en ze doen in de praktijk extra hun best en zijn meer betrokken bij hun onderwijs. De procedurele begeleiding van SURF was erg goed. Het was duidelijk wat er van je verwacht wordt, hoe dit van je verwacht wordt en SURF was zeer bereid op mee te denken bij problemen en boden hierbij goede ondersteuning. De inhoudelijke begeleiding ontbrak. De projectleider vindt het ook jammer dat de voortgangsrapportages nooit echt inhoudelijk becommentarieerd werden. Er was bij SURF een gebrek aan inhoudelijke expertise. De projectleider heeft deze lacune zelf ingevuld door een fase in het project toe te voegen, iets waar SURF erg welwillend mee omgegaan is. De projectleider zou het project zeker opnieuw doen maar zou proberen het efficiënter te regelen (bijv financieel gezien), een communicatieplan opstellen, zorgen voor voldoende inhoudelijk experts binnen het projectteam en externe inhoudelijke experts erbij betrekken voor reviews en feedback. Duurzaamheid innovatieve projecten De duurzame implementatie van een innovatief project zou gestimuleerd kunnen worden door rekening te houden met de behoefte van de eindgebruiken, het nut van de innovatie en door gebruik te maken met een standaard technologie die over een aantal jaar nog steeds gebruikt of geconverteerd kan worden.

133


Verslag interview Projectleider 4 – Project 4 Maandag 24 januari 2005 Algemene gegevens De geïnterviewde is vanaf het begin betrokken geweest bij het project, in het tweede deel van het project als projectleider. Iemand anders was de projectleider gedurende de eerste helft van het project en opsteller van het projectplan. Algemene vragen: Volgens de projectleider was het project een succes. Het product wordt nog zeer veel en succesvol gebruikt en heeft als een versneller gewerkt bij de introductie van de leeromgeving. Zowel het product als het proces waren een succes. Het procesgedeelte was een succes omdat er veel aandacht was besteed aan de implementatiestrategie en de ondersteuning van de docenten die veelal één-op-één. Met het management waren hierover goede afspraken gemaakt en daarna met docenten doorgesproken waardoor de docenten niet het gevoel hadden dat dit project van bovenaf werd opgelegd maar dat ze zelf er ook bij betrokken waren. Het product zelf is succesvol doorontwikkeld doordat ze zelf producten aan het ontwikkelen waren waarin wensen van eindgebruikers konden worden ingepast. Lessons learned SURF project: De rol van faal- en succesfactoren uit literatuurstudie bij dit project: De afbakening van de inhoud moet goed gedefinieerd zijn, je moet een duidelijk plan hebben, maar je moet vooral ook open staan en bereid zijn om de plannen en de needs assessment bij te stellen. De afbakening van het proces van het project moet helder zijn maar zeker ook flexibel zijn. De rol van de projectleider was belangrijk, deze moet met iedereen communiceren, enthousiasmeren, motiveren en het proces in de gaten houden. Door de grote vraag vanuit de opleidingen was de oorspronkelijke projectteam niet groot genoeg. Het projectteam was een competente groep mensen met voldoende expertise. De capaciteit bleek echter te weinig te zijn voor de extra vraag, hierdoor is het team ook uitgebreid. Het organisatiemanagement was betrokken, zorgde voor voldoende ondersteuning en had het laatste woord bij beslissingen. Men was zich er vanaf het begin van bewust dat het project tijdelijk was en dat het product duurzaam geïmplementeerd moest worden. Men had echter niet verwacht dat het project binnen zo’n korte doorlooptijd al Universiteitsbreed effect zou hebben. Deze snelle implementatie was o.a. een gevolg van goede mond op mond reclame. Het leverde voor de docenten een duidelijke tijdswinst op en de opleidingen zagen de urgentie van dit project in en meldden zich zelf aan bij het project. Dit gevoel van urgentie was een duidelijke succesfactor. Dit product bood een duidelijke oplossing voor een duidelijk en aanwezig probleem. Voor de docenten viel het project en de belasting ervan uiteindelijk mee door de goede ondersteuning en was de meerwaarde van het project ook duidelijk. De ondersteuning was positief gemotiveerd om dit product als nieuwe taak in hun werk op te nemen. Het management van de verschillende betrokken partijen was zo enthousiast dat ze andere opleidingen tipten over dit project.

134


Vantevoren was er geen uitgebreid communicatieplan opgesteld, wel verliep veel communicatie via mond op mond reclame en informele wegen. In alle fasen van het project is er zeer veel gecommuniceerd, gepubliceerd en gepresenteerd. Informele en formele communicatie was hier een succesfactor. Succes- en faalfactoren eigen project: Succesfactoren: gevoel van urgentie bij betrokkenen, goede ondersteuning voor docenten, tijdswinst voor docenten, goede projectorganisatie en commitment van de opleidingen voordat het project startte. Faalfactor: te weinig uitstraling naar andere faculteiten bij de andere instelling waardoor er geconcurreerd moest worden met andere producten waardoor waarschijnlijk overgestapt moest worden naar een standaardproduct. Reflectie eigen project: Rapportcijfer volgens projectleider: 9. Rapportcijfer van SURF volgens projectleider: 8 / 9 Rapportcijfer van eindgebruikers volgens projectleider: 8 De procedurele begeleiding en formats van SURF heeft bijgedragen aan het succes van het project doordat het dwong om vantevoren over bepaalde zaken na te denken en tot betere planning. De tussenrapportages dwingen je tot reflectie maar leverde veel overhead op. De inhoudelijke begeleiding was in de tijd van dit project nog niet zo breed en uitgebreid en waren soms nuttig. Het zou verbeterd kunnen worden door er een expertreview van te maken in plaats van een soort huiswerkoverhoring waar het indertijd op leek. Meer inhoudelijke review was beter geweest. De projectleider zou het project zeker opnieuw doen. Grotendeels zat het project goed in elkaar en behoeft geen verbetering. Wel zou het bij de andere instelling vanaf het begin breder ingezet kunnen worden zodat de strategische insteek beter zou zijn en de duurzaamheid beter gegarandeerd kan worden en de concurrentiestrijd voorkomen had kunnen worden. Duurzaamheid innovatieve projecten Bij het opstellen van het projectplan moet er zeker rekening gehouden worden met de duurzame implementatie van het product. Bij de duurzame implementatie van innovatieve projecten in het algemeen is het heel belangrijk om vanaf het begin helder te hebben of het een haalbaar idee is waar behoefte aan is. Verder moet duidelijk zijn voor welk probleem dit een oplossing is, wat de opbrengsten zijn voor de eindgebruikers, kunnen ze het product ook gebruiken, wat moeten ze er voor doen en voor laten (kosten – baten analyse maken) en hoe zien de verwachtingen eruit en is dit reëel. Binnen zijn huidige werk houdt de projectleider rekening met de lessons-learned van dit project: beter vooruit denken en plannen, projectorganisatie is erg belangrijk en betrokkenheid van alle partijen verzorgen.

135


Verslag interview Projectleider 5 – Project 5 Donderdag13 januari 2005 Algemene gegevens De geïnterviewde is vanaf het begin tot het einde projectleider geweest van het project. Hij heeft veel ervaring als projectleider bij verscheidene grotere en kleinere (onderwijsvernieuwings)projecten. Algemene vragen: Het project was volgens de projectleider gedeeltelijk een succes. Aan het project werkte 2 hogescholen mee en per hogeschool 4 opleidingen, dit leverde 8 deelprojecten waarvan er enkele zeer succesvol waren en anderen relatief weinig hebben bereikt. Kijkend naar de reacties op het project in zijn totaliteit lijkt het voor de projectleider dat de buitenwereld het een erg geslaagd project vindt. Dit project was een echt innovatief project met als doelstelling het onderwijs te transformeren zodat het onderwijs door de student zelf gestuurd kan worden. Uitgangspunt daarbij is dat het leren gebaseerd moet worden op sociaal constructivisme en te verwerven competenties. Het denken in termen van competenties als doelstelling bij de 8 deelprojecten en andere opleidingen is een doorlopend proces geworden. De aanzet en de eindmanifestatie van het project werden door SURF ook als geslaagd gezien. Lessons learned SURF project: De afbakening van de inhoud van het project was erg belangrijk. Binnen dit project is geprobeerd goed naar deze inhoud te kijken en te voorzien wat haalbaar zou zijn: in hoeverre is een organisatie in staat om deze veranderingen te maken. Er zit een fundamentele fout in het kijken naar innovaties: vaak wordt vergeten te kijken naar de consequenties van de veranderingen, van de innovatie. Echte innovatie projecten zijn veel meer veranderingstrajecten (organisatieverandering, attitude verandering, etc) dan men vaak verwacht en omvatten meer veranderingsprocessen dan de introductie van bijvoorbeeld een nieuw software pakket. De organisatiecultuur waarbinnen de verandering of innovatie zich afspeelt is cruciaal. Enkele deel-projectleiders die verantwoordelijk waren voor de innovatie hadden te weinig leiderschap (dit was soms niet mogelijk, was niet toegestaan, niet genoeg middelen beschikbaar, etc) en konden hierdoor niet veel bereiken ondanks dat ze hard werkten. De rest van de organisatie had soms ook geen boodschap meer aan het project of had er been band (meer) mee waardoor zij (de deel-projectleiders) zich afzonderden en niet succesvol konden zijn. De communicatie richting de rest van de organisatie is daarbij onvoldoende geweest. En indien de projectleider in de beoordelingscommissie zou zitten zou hij elk nieuwe aanvraag voor een project willen beoordelen op a) de veranderingscapaciteit van de organisatie waarbinnen de innovatie moet gebeuren, b) een compleet en concreet uitgewerkt communicatieplan, c) er moet een sponsor aanwezig zijn: een verantwoordelijke met macht en geld en met het besef van de consequenties en persoonlijk verlies indien het project niet zal slagen, op deze manier zorg je voor échte commitment. Bij veel van de projecten (in het algemeen) wordt aan deze punten te weinig aandacht besteed en wil men de innovatie in de

136


organisatie laten inbedden dan moet men zich bewust zijn van bovenstaande punten en realiseren wat de veranderingen teweeg zal gaan brengen. Voor aanvang is er bij dit project geen behoefte analyse gedaan bij de participerende opleidingen. Een behoefte analyse zou meer nut hebben indien je een product gaat maken. Bij een onderwijskundige innovatie zoals bij dit project is dit moeilijker. Wel is er achteraf en te laat geanalyseerd of de organisaties inderdaad écht dit project wilden, bij aanvang gaven zij dit wel aan maar dit bleek achteraf niet zo te zijn. Een aantal organisaties wilde mee doen om het mee doen zelf, voor de subsidie of vanuit ideologische invalshoek. Helaas zorgt dit niet voor een kritische massa om het project te gaan ondersteunen. Om dit te voorkomen moeten er vooraf hogere en strengere eisen komen. Bij de koplopers van de deelprojecten was het nut van het project zeer sterk aanwezig en duidelijk en zij hadden ook zeer duidelijk voor ogen wat ze wilden bereiken met dit project. Dit zorgde voor een sterke intrinsieke motivatie. Door deze betrokkenheid die hier door de gehele organisatie zichtbaar was, waren ze ook ruimschoots op tijd klaar met de innovatie en was deze zeer geslaagd. Ook voor de studenten leverden dit voordeel op en deze maken hier nu ook nuttig gebruik van. Het project was achteraf gezien te groot en te complex (de helft was beter geweest) of er zou een strenge selectie vooraf moeten zijn zodat alleen zeer betrokken organisaties die ook veranderingsbereid en bekwaam zijn mee zouden doen. Er zou een selectie moeten komen waarbij gekeken wordt naar de aanwezigheid van een sponsor en change-agents, hoe groot de kritieke massa is, wat voor capaciteit beschikbaar is en of de organisatie bereid is deze beschikbaar te stellen. SURF zou er eigenlijk op moeten staan dat deze punten ook in de projectdocumenten terugkomen en ook door de projectleiders wordt aangetoond dat ze hiertoe ook in staat zijn. Dit kan ook de projectleiders kunnen helpen. Er is teveel vrijblijvendheid: loopt een project niet goed, dan is het jammer, en loopt het project goed dan is het prima. Zolang je maar binnen de tijd en het budget blijft. Hier moet meer naar gekeken worden. Succes- en faalfactoren eigen project: De volledige toewijding en betrokkenheid van een aantal deelprojecten zorgde ervoor dat deze een succes waren, terwijl anderen een stuk minder succesvol waren vanwege verschillende redenen. Verder moet de organisatie een lerende organisatie zijn en bereid zijn tot veranderingen in denken en doen, tot leren van nieuwe dingen. Ook de hiërarchische structuur van een organisatie heeft soms invloed op het gemak waarmee dingen veranderd kunnen en mogen worden wat het succes van een project kan beïnvloeden. Een van de allergrootste problemen is (en daarmee is dit dus een succesfactor) de capaciteit van een organisatie. Bij innovatie of veranderingsprojecten moet je helaas altijd op een ‘rijdende trein’ springen, dit bemoeilijkt het project. De betrokken partijen moeten dus rekening houden met het huidige onderwijs dat normaal moet doorlopen, en tegelijkertijd moet er ruimte, tijd en menskracht zijn om de innovatie te ontwikkelen, testen en implementeren. Een soort (tijds)buffer zou een uitkomst zijn. Een mogelijke oplossing hiervoor zou het betrekken van externe medewerkers (dus niet per se de docent of inhoudsdeskundige zelf) die niet met het

137


lopende onderwijs zelf bezig zijn, dit wordt helaas niet vaak gedaan. Ook studenten moeten beschikbaar zijn om het product te kunnen testen zonder dat dit hun extra tijd gaat kosten. Reflectie eigen project: SURF beschouwt dit project waarschijnlijk als meer succesvol dan de projectleider zelf, omdat dit project als eerste bezig was met een transformatie en een aantal deelprojecten was inderdaad zeer succesvol en deze successen zijn ook naar buiten gebracht. De eindgebruikers waren erg enthousiast. De kwartaal projectrapportages naar SURF kostten teveel tijd en waren niet altijd nodig, vooral niet in de zomervakantie. Drie keer per jaar zou ook meer dan genoeg zijn aangezien er ook tussentijdse reviews plaatsvinden. De inhoudelijke begeleiding was goed. Meer overleggen zouden op prijs gesteld worden. De projectleider zou het project opnieuw willen doen maar zou dan graag een aantal dingen willen veranderen: een andere invalshoek om mee te starten die meer gericht is op veranderingsmanagement met andere woorden, het veranderingsproces zou beter benadrukt moeten worden. Verder zou het project minder complex moeten zijn of er zou een betere selectie vooraf moeten zijn zodat de deelnemende organisaties ook echt in staat zijn en bereid zijn deze innovaties door te voeren en te implementeren en zou de projectleider meer ‘macht’ moeten krijgen om de toegezegde deelname en bereidheid tot veranderen ook ‘af te kunnen dwingen’ bij deelnemende organisaties. Ook de communicatie zou beter geregeld moeten worden op alle vlakken (binnen het project zelf, binnen de deelnemende organisaties, tussen de organisaties en naar buiten toe). Over dit gebrek aan uitvoering van het beloofde communicatieplan heeft SURF ook nooit wat gezegd, dit vindt de projectleider jammer en dit had wel gedaan moeten worden. Het projectmanagement (zowel binnen SURF als binnen het project) zou verbeterd moeten worden en zou er op toe moeten zien dat iedereen van het project blijft meewerken en nakomt wat er beloofd is en in de plannen staat. Er zal dan ook iemand aangewezen moeten worden die kan optreden indien nodig, en die hiertoe ook de autoriteit en zeggenschap voor heeft (gekregen). Deze zeggenschap zou bij voorkeur bij de projectleider moeten liggen. Tot slot moeten de projectleden voldoende tijd krijgen om aan het project te besteden, anders kan er wel toewijding en goede wil zijn, maar indien er niet genoeg tijd is (dit is afhankelijke van de sponsor en commitment van de organisatie om hiervoor te zorgen) kan het project ook geen succes worden. Op voorhand zou ook dit goed gedocumenteerd moeten worden.

138


Verslag interview Projectleider 6 – Project 6 Woensdag 19 januari 2005 Algemene gegevens De geïnterviewde is vanaf het begin (1 oktober 2001) tot het einde (1 januari 2005) projectleider geweest van dit project (het loopt nu door met een vervolgproject tot 1 januari 2007). De projectleider heeft ervaring als projectleider in kleine en grote projecten. Algemene vragen: Het succes was volgens de projectleider een succes omdat de doelgroep het product in grote getale gebruikt. Het project loopt al drie jaar en gaat nog 2 jaar door. Het project is in 2001 begonnen met een procesmatige aanpak met onderwijskundige fasen (oriëntatie en ontwerpfase, pilotfase, implementatie en evaluatie) waarbij er per instelling een contactpersoon verantwoordelijk was. Hierna is het project in 2004 verder gegaan met een meer productmatige aanpak waarbij gekeken werd naar de deliverables en waarbij er werkpakketten zijn gedefinieerd met elk een werkpakketleider. Deze werkpakketleider is verantwoordelijk voor de werkzaamheden in zijn werkpakket en over de personen binnen zijn werkpakket verdeeld over meerdere instellingen. Deze vernieuwde aanpak leent zich goed voor projecten waarbij de eindresultaten van tevoren vast staan. In innovatieve projecten waarbij de resultaten soms door voortschrijdend inzicht moeten worden bijgesteld voldoet de procesmatige aanpak ook. Lessons learned SURF project: Positieve ervaringen van de projectleider waren o.a. het succesvol samenwerken en creëren van een kennisbasis die dicht aanligt tegen de behoeftes van de docenten. Een aantal van deze docenten was ook erg nauw betrokken bij het project en fungeerde als klankbordgroep in de periode 1 oktober 2001 tot 1 oktober 2003. Er werd periodiek geëvalueerd en overlegd met de klankbordgroep. De nauwe betrokkenheid van de doelgroep was zeker positief en zorgde ervoor dat het product ook gebruikt ging worden. Een mindere kant van dit project was de samenwerking op afstand, waardoor de afstemming met de andere partners af en toe moeizaam kan verlopen. Maar hier werd wel een oplossing voor gevonden. De rol van faal- en succesfactoren uit literatuurstudie bij dit project: De afbakening van de inhoud van het project was volgens de projectleider heel relevant. Hoe beter en hoe concreter de doelstellingen geformuleerd worden, zowel in kwalitatieve als kwantitatieve zin, hoe doelgerichter je kunt werken binnen een project en hoe gemakkelijker je afspraken kunt maken en des te duidelijker is het wat je aan het einde van het project gaat bereiken. Door goede procesafbakening van het project kan de voortgang goed gepland worden en in de gaten gehouden worden om zo de einddatum te halen. De projectleider was zowel sturend, motiverend, monitorend als inhoudelijk betrokken. De overkoepelende rol van de projectleider richtte zich op zowel procesmatige als inhoudelijke aspecten. Het projectteam (10 mensen) was een goede competente groep mensen. Er is van tevoren veel aandacht aan besteed om de juiste competenties bij elkaar te krijgen door gericht mensen te vragen voor het project en in samenwerking met de andere onderwijskundige centra te selecteren.

139


Het organisatiemanagement was betrokken bij het project. Er werd ervoor gezorgd dat het project binnen de verschillende gremia geregeld onder de aandacht kwam en dat de resultaten verspreid werden. Tijdens het project is een communicatieplan opgesteld waaraan in grote lijnen is vastgehouden. Dit communicatieplan hangt samen met de complexiteit en het doel van het project en veranderde met het voortschrijden van het project. Bij aanvang van het project is een behoefte onderzoek uitgevoerd dat op de eindgebruikers gericht was. Deze eindgebruikers werden ook in de loop van het project bij het project betrokken. Het management was op de hoogte van de inhoud van het project, maar hier werd geen behoefte analyse uitgevoerd. De projectcultuur was open en kritisch binnen de projectgroep en zo zou het ook moeten zijn. De samenstelling was zodanig dat deze openstond voor goede discussies en kritische geluiden. De duurzame implementatie van het project was in het begin van het project niet belangrijk, maar dit groeide naarmate het project bewees succesvol en nuttig te zijn. Het plannen van een duurzame implementatie van een innovatief project is erg moeilijk omdat je niet weet in hoeverre het succesvol en gewenst zal zijn. Gedurende het project is de duurzame implementatie belangrijker geworden en is het plan hiervoor ontwikkeld. Succes- en faalfactoren eigen project: Succesfactoren bij dit project waren o.a. de betrokkenheid van de eindgebruikers, de praktische en concrete manier van vormgeven van de inhoud en het product, de aansluiting bij de behoeftes van de gebruikers en goede afbakening van het project. Een verbeterpunt is de balans tussen het halen en brengen van de kennis. Hier had meer aandacht aan besteed moeten / kunnen worden. Reflectie eigen project: Zowel de projectleider, als SURF als de eindgebruiker waren positief over de resultaten. De bijeenkomsten van SURF waren wel nuttig, maar kan niet echt begeleiding genoemd worden. SURF stond wel open voor problemen en wilden dan ook zeker helpen. De EExchange bijeenkomsten leverden soms goede casussen voor discussie en handzame instrumenten voor de projectleiders. De projectleider had geen behoefte aan meer begeleiding van SURF. Wel waren de administratieve procedures soms te / erg uitgebreid en relatief tijdsintensief. De projectleider zou dit project zeker opnieuw willen doen, waarbij de organisatorische structuur veranderd zou kunnen worden zodat de projectmedewerkers meer verantwoordelijkheden zouden krijgen zoals nu in het tweede gedeelte van het project het geval is. Duurzaamheid innovatieve projecten Het plannen van een duurzame implementatie bij een innovatief project is erg moeilijk aangezien je van tevoren niet weet of deze succesvol of nuttig zal zijn. Het is wel belangrijk om hier rekening mee te houden bij het projectplan en dat de medewerkers zich hiervan bewust zijn om het project na afsluiting ook in leven te houden. De duurzame implementatie zou als concrete activiteit gedefinieerd moeten worden in het projectplan. Om de duurzame implementatie te stimuleren zou het nut voor de gebruikers

140


duidelijk moeten zijn, zij moeten er ten slotte beter van worden en zij zouden bij voorkeur lid moeten zijn van de projectgroep. Verder moet er commitment zijn vanuit de opdrachtgever. In het algemeen Een succesvol innovatief project in het algemeen moet voldoen aan oa: het moet gebruikt en geïmplementeerd en het moet leiden tot verbetering van de processen en producten. De eindgebruikers moeten er duidelijk nut van hebben.

141


Verslag interview Projectleider 7 – Project 7 Dinsdag 11 januari 2005 Algemene gegevens De geïnterviewde heeft het project als projectleider overgenomen van iemand anders na ongeveer 9 maanden. Hij heeft al ervaringen als projectleider en als projectmedewerker bij innovatieve projecten. Algemene vragen: Het eindresultaat was in de ogen van de projectleider een succes. De gestelde projectdoelen zijn gehaald en de organisatie heeft ingezien dat ze bezig zijn (geweest) met een verandertraject wat ook een toegevoegde waarde heeft. De organisatie heeft ervan geleerd. Lessons learned SURF project: De afbakening van de inhoud was vrij helder en scherp, dit was geen probleem. Aan de projectbeschrijving is vastgehouden, alleen de manier waarop de resultaten behaald moesten worden is wel veranderd in de loop van het project. Het is opgezet als productgericht project, dit veranderde in een procesgericht project. Producten maken zonder dat je de processen daarvoor doorloopt is onmogelijk. De ingestelde processen van het project lopen nu nog steeds door en hiermee worden de projectresultaten geborgd. De afbakening van de processen van het project heeft een kritieke rol gespeeld in het slagen van het project. De projectleider was competent en niet te sterk inhoudelijk betrokken of expert, deze afstand was een groot voordeel binnen dit project. Het was echter wel moeilijk om halverwege in een project in te stappen tijdens een ‘crisismoment’. Een moeizaam project doorstarten is meestal niet aan te raden. Goede afspraken zijn zeer belangrijk en de concrete invoering moet goed zijn vastgelegd bij aanvang van het project. Niet alleen het organisatiemanagement moet betrokken zijn, maar ook moet vaststaan wie wanneer welke tijd/moeite gaat investeren. Er was een sterke betrokkenheid van het management, dit was zeker ook een succesfactor. Ook de (kleine groep) docenten waren zeer betrokken. Het was niet geheel duidelijk wat de verwachtingen van de betrokkenen waren met betrekking tot het project. De omvang en impact van het project was niet voor iedereen duidelijk, dit werd pas gaandeweg duidelijker. De omslag van kijken naar het project als project en kijken naar het project als duurzame implementatie in de organisatie heeft een rol gespeeld. Deze verandering van visie speelde een rol in het project. De projectcultuur was open, flexibel, en het werd / was een lerende omgeving. Deze openheid leverde soms ook frustraties op aangezien er soms teveel ruimte was (en werd genomen) voor discussie zonder conclusies en besluiten nemen. Hier was het handig dat de projectleider wat verder van de inhoud stond dan de rest van het team. De innovatie was erg belangrijk voor de opdrachtgevers en de noodzaak van het project was duidelijk aanwezig. Er is door de procesbenadering en doordat met dit project andere opleidingen ook op deze wijze leermodules kunnen maken rekening gehouden met de duurzame implementatie van het project.

142


Succes- en faalfactoren eigen project: Succesfactoren zijn oa het commitment van het management geweest, evenals de verandering van inzicht van product gericht naar procesgericht werken, de flexibele opstelling van SURF binnen de kaders (werkt zeer motiverend), de veerkracht van de projectgroep en een externe projectleider die iets meer afstand heeft van de inhoud (vergemakkelijkte de besluitvorming). De samenwerking met de andere instituten was deels een succes door het deels invullen van de intenties, maar was deels ook een faalfactor geweest doordat deze had beter geregeld moeten worden bij aanvang van het project. Een andere faalfactor is het gebrek aan onvoldoende inzicht in de consequenties van het project. Een intake bij het begin van het project zou nuttig zijn, waarbij ingegaan moet op worden op aspecten als: zijn de deelnemende instituten zich bewust van de consequenties van het project op hun manier van werken, hebben ze de afspraken voldoende diep in de organisaties vastgelegd, heeft de organisatie ook de ‘massa’ om de innovatie te realiseren? De projectleider van een dergelijk groot project zou via een wervingstraject gekozen moeten worden ipv ‘vriendjespolitiek’. Reflectie eigen project: Het project zou een 7 krijgen van de projectleider. Wat SURF voor cijfer zou geven weet de projectleider niet, maar hij denkt wel dat ze tevreden waren. De eindgebruikers zullen het project wel een mooi cijfer geven (8). De algemene evaluaties waren positief. De procedurele begeleiding was strak, maar met ruimte voor aanpassingen. De inhoudelijke begeleiding van SURF was erg goed, op het juiste moment en goed gedoseerd waarbij geen valse beloftes werden gedaan maar wel goed tips gegeven en hulp geboden. De projectleiders werden goed begeleid door de medewerkers van SURF. Een kritische intake-gesprek zou een verbetering zijn van het proces kunnen zijn vanuit SURF. En een blijvende klankbord groep van collega-projectleiders om ervaringen uit te wisselen zou ook een verbetering zijn. De projectleider zou dit project zeker opnieuw willen doen met intake-gesprek. Duurzaamheid innovatieve projecten De organisaties waarbinnen de implementatie plaats vindt zullen ervoor moeten zorgen dat zij een aantal processen starten waarbij ze vooral kijken naar haalbaarheid van de implementatie binnen de organisatie om de duurzaamheid te garanderen. Een pilot vooraf kan zeer nuttig zijn hierbij. Processen zijn belangrijk.

143


Verslag interview Projectleider 8 – Project 8 Woensdag 12 januari 2005 Algemene gegevens De geïnterviewde is vanaf het begin projectleider geweest en heeft ervaring als projectleider (van kleinere projecten) en ervaring als projectmedewerker. Algemene vragen: Het project was in de ogen van de projectleider een succes en leverde meer op dan oorspronkelijk gedefinieerd was en waar voor meer dan de oorspronkelijke projectpartners nuttig kan zijn en in de toekomst gebruikt zal worden. De innovatie (product) zelf was wel een succes, het proces echter niet. Het proces is anders gelopen dan het oorspronkelijk gedefinieerd was. Lessons learned SURF project: De afbakening van de inhoud van het project (opzetten van kennisnetwerk voor communicatievak) was goed afgebakend, maar anders ingevuld. Deze verandering van invulling vond plaats na 1 jaar na een ‘schrikeffect’. Na dit schrikeffect werd eerst gedivergeerd en na een doorloopperiode kon er geconvergeerd worden na de nodige extra moeite, geld en tijd. De verandering van technologie resulteerde onder andere in een aanpassing van de projectplanning Alle delen van de planning waren op deze ‘verkeerde’ start gebaseerd en moesten veranderd worden. De aangenomen ‘zekerheden’ werden veranderd. Bij innovatieve projecten of processen is een flexibele afbakening van het project heel belangrijk, te strak vastleggen van projectwegen kan belemmeren en extra werk opleveren. In het projectteam zaten gaten in de expertise, dit had voornamelijk te maken met de samenwerking met de projectpartners. Er ontbrak het nodige besef wat nodig was voor dit project wat ook resulteerde in vervroegde uittreding van een van de partners. Vanuit de projectpartners zijn de verkeerde competenties ingezet in het project. Het management van de twee betrokken partners waren betrokken bij het project en ondersteunden dit ook. De druk vanuit het management om de eigen organisatie er beter bij te betrekken had beter gekund. Dit had gekund door meer spontane betrokkenheid af te dwingen. Delen van de organisatie waren niet erg gewillig om mee te werken, hier had vanuit het management meer aandacht aan besteed kunnen worden. De verwachtingen die de betrokken partijen hadden maar zeker ook met de vertragingen die het project heeft opgelopen droegen ook bij aan deze ‘onwil’. Deze vertragingen (als gevolg van verandering van technologie) riep ongeduld een sceptische houding binnen de organisatie op. Er is te weinig rekening gehouden met de verwachtingen van de betrokken partijen, dit was echter moeilijk te veranderen omdat het eindproduct niet helder was bij aanvang van het project. Ook bleken de wensen van de partners en eindgebruikers anders te liggen dan in eerste instantie was aangegeven. De communicatie verliep in dit project niet optimaal en er waren duidelijke fouten gemaakt die op dat moment moeilijk te voorkomen waren. Met name mbt de vertraging is er te weinig en niet duidelijk genoeg gecommuniceerd. Communicatie was bij de implementatie wel een belangrijke factor.

144


De meerwaarde van het project speelde een dominante rol. Het is nog steeds niet af en blijft een organisch geheel, het product ontwikkeld zich nog steeds. De projectcultuur was open en flexibel omdat er spontane medewerking was van het projectteam om een andere weg in te slaan. Ook tussen de partners was een open sfeer. Succes- en faalfactoren eigen project: Bovengenoemde factoren zijn allemaal relevant maar vooral dissatisfiers: als deze factoren niet aanwezig zijn, dan bemoeilijkt dat de kans op succes. Maar indien ze goed meewerken wil dat nog niet automatisch zeggen dat het project een succes wordt. De juiste keuze van technologie, het technologische hart in dit project, is een typisch voorbeeld van een satisfier. De beschikking over een state-of-the-art technologie was dus een succesfactor en zorgde voor een goed afronding van het project. Een verbeterpunten achteraf gezien is: meteen de goede keuze maken voor de juiste technologie. De juiste technologie kon echter pas opgemerkt en herkend worden bij gebruik van de technologie. Reflectie eigen project: De projectleider zou het product een 8 geven en de implementatie een 5. SURF denkt er volgens de projectleider ook ongeveer zo over. De eindgebruiker is tevreden over het product. Maar omdat het product nog niet in volle potentie wordt gebruikt en de gebruiker nog niet alle voordelen kan zien, zal deze geen 8 krijgen van de eindgebruiker. De klachten mbt gebruik en fouten van het begin zijn verbeterd en zo blijft het product zich ontwikkelen en verbeteren. Doordat er meer actoren in het netwerk komen wordt dit ook sterker en krijgt het meer potentie. De procedurele begeleiding van SURF werd als nuttig ervaren. Het programma voor projectleiders was nuttig. Ook de inhoudelijke begeleiding was goed en de overleggen met SURF werden zeker gewaardeerd en als prettig ervaren. De flexibiliteit van SURF heeft bijgedragen aan het slagen van het project. De projectleider zou het project zeker opnieuw willen doen, hierbij zou hij de communicatie beter aanpakken, betere afspraken maken en direct de juiste technologie kiezen. Echter een innovatief project is altijd een lerend proces waarbij ruimte moet zijn voor aanpassingen. Duurzaamheid innovatieve projecten Duurzame implementatie van een innovatief project is zeker belangrijk. Je moet geen project starten voor het project, maar om het product wat er uit gaat komen. Dit project had weinig waarde gehad als het product niet duurzaam was geweest. Duurzame implementatie kan gestimuleerd worden door goed te kijken of het product aansluit bij de eindgebruikers, of er echt behoefte is aan dit product (bijna alsof het van ‘levensbelang’ is), of het een urgent product is. Maar wellicht is het ook een goede factor indien het leuke dingen kan creëren. De duurzame implementatie kan versterkt worden door vooraf goede afspraken te maken en vastleggen wat de implementatie beteken voor de organisatie: wat moet de organisatie doen om zo’n product in stand te houden, in tijd, menskracht, etc. Hier zit ook de moeilijkheid: je kunt het eindproduct in het begin niet goed schetsen omdat het een innovatief (dus nieuw) product is, je kunt dus ook niet goed de kosten schetsen. Duidelijkheid met een zekere mate van flexibiliteit is belangrijk samen met de commitment en het besef van de consequenties voor de organisatie.

145


Appendix VII - Projecttoets Projecttoets - Innovatiesubsidie Samenwerkingsprojecten •

Past mijn projectidee binnen Innovatiesubsidie Samenwerkingsprojecten?

•

Gegevens aanvrager: Naam bedrijf/organisatie: Postadres: Postcode: Plaats: Sector: MKB: ja/nee (doorhalen wat niet van toepassing is)

•

Naam contactpersoon: Dhr/Mevr. (doorhalen wat niet van toepassing is) (mobiele) telefoon contactpersoon: E-mail contactpersoon:

Heeft u al contact gehad met een projectadviseur van SenterNovem? Naam:………………………

ja/nee

•

Gegevens project Wat is de titel van het project?

•

Welk projecttype is mogelijk van toepassing op uw projectidee? (Kruis aan wat van toepassing is) O Haalbaarheidsproject O Onderzoeks- of ontwikkelingsproject O Nog niet bekend Gaat u internationaal samenwerken: O Ja O Nee O Nog niet bekend

•

Geef een korte omschrijving van het project.

•

Invulling beoordelingscriteria

147


Bij O&O-projecten zijn technologische innovativiteit, duurzaamheid, samenwerking en het economisch perspectief de criteria waarop uw project wordt beoordeeld. Om u goed bij uw aanvraag te adviseren vragen wij u aan te geven hoe uw project invulling geeft aan de criteria. Op pagina 8 van de handleiding worden de criteria toegelicht. Ook bij een haalbaarheidsstudie zijn de criteria van belang. Na de haalbaarheidstudie moet immers duidelijk zijn of u het project wilt vervolgen met een Onderzoeks- en/of Ontwikkelingsproject. Als u nog niet concreet weet hoe uw project invulling kan geven aan deze criteria mag u dit onderdeel overslaan en pas bij de bespreking van het project met de projectadviseur toelichten. •

Welke innovatieve technieken worden toegepast of ontwikkeld in uw project?

•

Levert uw project een bijdrage aan duurzaamheid? Zo ja, op welke manier?

•

Met welke partijen (bedrijven, instanties) wordt binnen het project samengewerkt? Geef hierbij ook de specifieke inbreng per partij en de wekverdeling aan.

•

Wat is het economisch perspectief van het project voor de betrokken partijen?

•

Wat is de vermoedelijke startdatum en looptijd van het project? Start , duur (maanden)

•

Heeft u al een indicatie van de projectkosten? EUR

E-mail, stuur of fax deze projecttoets naar: SenterNovem Den Haag Innovatiesubsidie Samenwerkingsprojecten Gebouw CentreCourt, Juliana van Stolberglaan 3 Postbus 93144 2509 AC Den Haag Telefax : (070) 373 51 00 E-mail : [email protected] Heeft u nog vragen of wilt u meer informatie, neem dan contact op met een van de projectadviseurs van Innovatiesubsidie Samenwerkingsprojecten, telefoon: (070) 37 35 111

148


Appendix VIII – Original checklists Initiation phase Nr Initiation phase 1.

The problem is clearly 14 defined

2.

The project goal is clearly defined The project objectives are clearly defined

3.

4.

The project’s scope is clearly defined

-

5.

The project’s scope is realistic -

6. 7.

14

Realistic expectations are created The project planning is specified

-

Description / criteria The problem has been stated simply, realistically and to-the-point It is evident who experiences the problem It is evident who is responsible for the problem; who the owner of the problem is The desired end result of the project has been stated in the goal The goal statement is smart (specific, measurable, assignable, realistic, time-related) Criteria for the deliverables is included All objectives are complete and necessary to achieve the stated goal All objectives are stated simply, realistically and to-the-point All objectives have a definable and measurable end result All objectives are smart (specific, measurable, assignable, realistic, time-related); compared to the project goal, the objectives are described in a more correct manner (i.e. ‘smarter’) The boundaries regarding: problem statement, project planning, project objectives and project goal are stated The scope is stated in such a way that the level of changes needed within the different parts of the organisation on the route from the present to the new situation can be clearly envisioned All deliverables are explicitly coupled to project activities and people. In other words all contributors know what their responsibilities are. Minimised/realistic scope: Time and resources (people/money) are crucial factors for many projects, and since scope affects time or project duration and resources required, minimising scope increases a project’s chances of success Realistic change: The greater the number of changes intended, the more factors there are which influence the chances of success and therefore there is an increase in the risks of the project. Expectations regarding to scope, planning, goal, objectives, etc. Time schedule Who does what and when

Two things are asked here: is it defined and is it defined clearly. If it is not defined please mark this item as “not”, this includes that it is also not defined clearly. If it is defined, but not clearly defined please mark this item as “–“.

149


8. 9. 10.

11.

Clear responsibilities of team members are defined Clear accountabilities of team members are defined The project’s prior / limiting conditions are clearly defined

Return-On-Investment is taken into account

-

Who is responsible for what

-

Who has been given the authority to do and decide upon what

-

The cost limits the project will not exceed are stated A list of project deliverables is included Specific due dates of the deliverables are specified Specific due dates of the deliverables are realistic The responsibilities and the authorities have been defined Work packages with responsibilities of the managers of those packages have been defined The costs are deducted from the benefits Not only the visible, but also the transaction costs15 are calculated. Experts (or at the least knowledgeable crafts persons) were used to help estimate and review each activity’s budget and potential benefits All cost categories and potential benefits for each activity have been estimated (labour, equipment, travel, supplies, etc) Estimates for all internal resource costs and potential benefits were estimated before the need or use of external resources was estimated Cost and potential benefits estimates are realistic and conservative Cost and potential benefits estimates are comprehensive for each activity; there are no hidden costs or surprises Who has a stake in what will be done (e.g., problem owner, target group, government, industry, educational institutions, students, teachers, tutors, society in general) The groups that must be interacted with to accomplish project goals have been identified Strategic interests / concerns of each stakeholder has in the project are identified in order to negotiate common interests

12.

Stakeholders are identified

-

15

The costs, other than the money price, that are incurred in trading goods or services. Elementary versions of economic theorizing often make the simplifying assumption that information and other transaction costs are zero. But realism nevertheless demands that we keep in mind the fact that the benefits to the participants in an exchange have to be high enough to cover their transaction costs if the trade is to take place at all. Indeed, many otherwise mutually advantageous trades do not take place because of the very high transaction costs that would be involved. High transaction costs are very often at the root of the problems discussed under the heading of externalities, especially in those situations where the external costs or benefits accrue to very large numbers of third parties and therefore a contractual agreement to internalize the externality is extremely costly to negotiate.

150


13.

Formal feedback channels are created -

14.

There is a competent / experienced project manager

-

15.

The project manager is not the organisational manager -

16. 17. 16

The project manager has the necessary responsibilities The project manager has the necessary authority

Meaningful reporting frequencies have been established The reporting requirements and informed the work package managers accordingly have been specified Variances from plan that will require immediate action (critical path analysis) have been defined and communicated to the project team Responsibilities for completing project status reports have been assigned and communicated to the project team and other interested parties Top management has been briefed on the reporting procedures and they have been approved by them There is attention paid, and opportunities are provided for reporting (orally and in writing) to the organisation at all relevant levels Successful projects have an experienced project manager16. This means that (s)he: - is experienced enough in this project area to successfully commit to complete the project on time, within the budget, and according to specifications - is strategically capable to envision and conceptualise the entire project process through all phases - is technically competent to help design and execute a total project plan - is interpersonally able and experienced to inspire, motivate, discipline, resolve conflicts, negotiate competing interests, and sell the project The purpose of organisational management is to plan, guide and help the organisation work well and survive (i.e., achieve continuity. Project management is to actually actively plan the end of a project (i.e., to break with continuity and achieve change) In contrast to organisational managers, project managers have to start their task from scratch, initiating and bringing an organisation to life. To start something in order to end it within a fixed period of time, according to a budget and according to a number of restrictions distinguishes project managers from organisational managers

-

This includes: having the following competencies and characteristics: flexible, self-motivated, energetic, organises, alert, quick-to-learn, self-disciplined, effective listening skills, excellent problem solver, skilled politician, balanced temperament, sensitive, politically savvy, ethical, able to sell ideas and plans, express enthusiasm, cheerlead, coach, able to effectively interact verbally and nonverbally in the board room and on the shop floor, operates well under pressure demonstrated competency to act effectively and quickly in the following roles: figurehead, leader, liaison, monitor, information disseminator, spokesperson, entrepreneur, disturbance handler, resource allocator, negotiator demonstrated effectiveness and experience in: understanding and interacting with stakeholders, meeting management, contract negotiations, all phases of strategic and technical planning, budgeting, resource acquisition and allocation, initiating and termination professional and personal relationships, understanding complex organisational, group, and interpersonal issues and problems, admitting error and bias, acknowledging and celebrating others’ contributions

151


18.

The project team members are competent -

19.

The project manager and team members are able to clearly explain the project17 There is commitment from executive management There is support from executive management

20. 21.

17

The project team (i.e., the members together) have the needed expertise, experience and skills for the project The project team members: - are committed to planning and executing the project through all phases - are technically experienced in their specific area to help successfully complete the project on time, within budget, and according to specifications - understand and have used project management concepts and tools - are interpersonally capable of sharing information and expertise, solving conflicts jointly, planning and problem solving with others, actively listening, receiving and giving non-judgemental feedback, communication assertively - are able to work with two ore more bosses - are self-motivated and entrepreneurial but not self-oriented - are flexible - are responsible and can own mistakes as well as achievements - can share rewards and failures - are willing to delegate and be delegated changing work assignments - can work well under pressure and changing deadlines

The plan includes the signature of management proving commitment in terms of all aspects of the project planning and execution Resources like time, material, budget, personnel is guaranteed to be available for the project, including implementation at the end of the project

This means that the SURF procedure includes an oral presentation AS WELL AS a written project plan.

152


Implementation phase Implementation phase 22. 23. 24. 25. 26. 27. 28. 29. 30. 31.

User involvement is ensured User expectations about the innovation are managed Stakeholders are engaged Initiator of the change is trusted and respected by the prospective users Adequate informationcommunication channels are created There is a focus on adoption rate of maximally 25% of the institution The is a focus on affecting opinion leaders’ attitudes There is opportunity planned for external criticism There is room for the team continually question its own assumptions There is opportunity in the planning to modify the plan based on realities

- There is continuous reporting to the organisation (end users) - Lack of adequate information communication and fuzzy use of language can cause miscommunications and misconceptions about the innovation - The step (transition) from project to adoption / implementation is not too large - The more opinion leaders that adopt the system, the lower the critical adoption rate for other system members will be - Regular meetings between the project team and the organisation / stakeholders have been planned - Regular meetings between the project work packages have been planned - Feedback loops with the organisation and stakeholders have been planned - There is room in the planning for course corrections

153


Institutionalisation phase Institutionalisation phase 32.

The organisation exists in a stable environment, is supportive and collaborative)

33.

The organisation has a relatively loose, centralized structure with good vertical communication channels

34.

There is competent organisational management

The success of the innovation can be facilitated of inhibited by: - turbulence in the environment: the level of uncertainty in the surroundings can inhibit the innovation’s chances; - level of shock to the system: an extreme event such as a budget crisis can endanger the chances of success; - support or encouragement for the innovation: the capacity of the organisation to meet a new idea with enthusiasm can facilitate the innovation; - collaboration: balancing collaboration between various subsystems can be crucial for the success; - availability of external support for the innovation: support such as external funding can facilitate the innovation; - dedication to implementation. The chances of success of the innovation can be influenced by: - the structure of the organisation: innovations started in relatively loose yet centralised organisational structures with good communication channels have more chance to be successful; - the demographics or level of diversity of its members: innovative teams containing of near peers tend to be more successful; - internal turbulence: such as high staff turnover or frequent re-organisations can disrupt the sustainability of an innovation; - internal boundaries: creating barriers to widespread diffusion of ideas and innovations within a system can inhibited the innovation; - internal resources: such as staff time, funds, office space etcetera facilitate the innovation. Good managers help to keep the organisation on track. Factors that contribute to an innovation success: - Mission management: refers to knowing whom or what the organisation serves (strategies, planning, evaluation, contacts etc) facilitates the innovation; - Pay and personnel: payment and rewarding of the personnel is crucial for the success of an innovation; - Learning: facilitating organisational learning by the group rather than by one person can contribute to the success; - Idea generation: providing establishing systems that provoke new ideas can facilitate the innovation; - Budget: a balance between protecting the organisation’s resources and the investments (Return-onInvestment) increases the chances of success; - Accountability and governance: monitoring, supporting and coaching are important.

154


35.

There is competent implementation leadership

36.

The amount of change is carefully dosed

37.

User expectations are managed

38.

The innovation’s relative advantage as compared with the current practices is communicated Structures and opportunities are planned that promote learning of new practices Observable incentive systems to support implementation are planned Effective communication among all parts of the system Turn the traditional organisation into a “learning organisation”

39.

40.

41. 42.

The competence of a leader can be rated through and influenced by: - the leader’s vision: the leaders ideal and unique image of the future; - temperament of the leader: whether the leader is flexible, intuitive or whatever; - communication: the degree and type of communication between the leader and the organisation; - durability or endurance: prepared for stress and anxiety that will result of the learning process and changes; - innovation skills: associated with management skills, a leader must be able to create space to experiment, develop and innovate. The key to effective change (or innovation) is to carefully dose the amount of change, and therefore of the perceived threat, produced by the disconfirming information to allow the group members to feel safe to unfreeze, change and re-freeze their ideas and concepts. - The more an innovation plans to change, the greater the influence from the environmental will become, the greater the risks will be and the chances of failure will increase - The greater the number of changes is, the more factors there are which influence the chances of success and therefore there is an increase in the risks of the project Innovation projects will fail if the users of a system are dissatisfied with it because it does not meet their expectations. Therefore, project managers should not only manage the development of the system, but also the perception of the system - Return on investment!! - Transaction costs: Am I better off now? - Observability: a degree to which other persons than the innovator see its results as beneficial -

-

A learning organisation that adapts to innovations and restructures itself to accommodate change

155


156

Determinants for Failure and Success of Innovation Projects: The Road to Sustainable Educational Innovation

Recommend Documents