Summary of focus group session on Knowledge Management Impact Assessment

by Louise Clark

On day 2 of the share fair 47 Participants representing 28 institutions from more than 10 different countries came together to discuss their challenges and experiences in measuring the impact of their KM activities. The diversity and dynamism of the group clearly showed the wide ranging interest in this subject and creating spaces to discuss the common challenges we face to provide credible evidence of the changes resulting from KM practice.

The session started by exploring some of the different activities that we describe as KM, this group brainstorming session came up with ideas as diverse as channeling, exploring, documenting, collecting, categorizing, preserving, integrating different types of Knowledge, sharing, communicating, connecting people, learning lessons and making them visible. Someone raised the point that KM could also be about hiding knowledge, a point worth highlighting, as it is important that we are not afraid to talk about negative experiences and things that don’t work. This great diversity of definitions is one of the key challenges we face as KM practitioners; if we do so many different things how can we possibly agree on the best way to measure KM?

Another challenge we face is understanding what we really mean by impact, as for most of us this is something in the stratosphere and a long way from what we all do on a daily basis. From our project activities we generate a range of knowledge products from websites, blogs, tweets as well as manuals, training programs and even library classification systems; we are comfortable to measure our outputs at this level. What is much harder is for us to know how these products are used by our target audiences, and in some cases who the target audiences even are. We are also interested to know how the use of these products leads to changes in behavior amongst these target groups, with some of the participants feeling that this was easier to measure than the use of products as you can ask people while another member of the group questioned this as we also need to know what other factors have influenced this change in behavior. Impact happens even further when different groups change behavior and change happens at a wider level. We feel that KM is having an impact and contributing to these large level changes, we just don’t know how to prove it :{

Despite the great challenge of the KM Impact question the participants dived into lively group debates around 3 questions to think about why it is important for us to be able to explain the changes that result from KM activities, share their experiences of different tools and approaches and work together to think about what we can do better to address this issues.

Responses to the first question highlighted the broad range of reasons of why this question is important including:

Accountability

• To justify the existence of KM and development
• To be accountable to whoever has contributed to the effort and justify use of resources in both time and $$$ to management and donors – demonstrate ROI
• To influence policy and organizational change
• Mainstreaming KM into core activities of orgs
• To reduce possible resistance
• KM takes funds and we need to demystify KM to access new resources

Learning

• To inspire others and assess and evaluation our activities to identify gaps, define new target and continually update and identify new knowledge.
• Knowledge has positive and negative elements, we need to measure impact so that we can see and address the negatives
• Measurement is important to create feedback loops so we can tell what’s working and adapt what isn’t to improve out interventions
• To improve quality of our activities and know that we are reaching our target audiences and whether we are providing the rights products and services in the right formats. This should ensure cultural appropriateness to guarantee sustainability and customer satisfaction
• To facilitate replication of good practices
• To prove value of KM we need indicators, evidence to share and scale up by networking with partners and building relationships with other
• Change researchers’ attitudes and encourage them to disseminate the Knowledge generated by research
• To get sustainability of change
• To demonstrate effectiveness of work
• To improve our own planning

Ownership

• To promote ownership of process – builds commitment and create demand
• To make KM more responsive to local needs
• To get community buy in for KM work
• KM is about finding a way to reach the grassroots but also involves the use of the technologies that draw us away from a focus on the grassroots so it is important to keep balance

Question 2 highlighted a range of different tools and approaches for measuring KM impact

• Find ways to get feedback e.g. letters, email, phone, SMS, film, photos verbal
• Surveys – on usage, customer satisfaction, online, interview or informal feedback
• Identify the right indicators
• Statistics of numbers of new innovations, numbers of partnerships, numbers of documents identified, numbers of websites visited, numbers of radio broadcasts / videos uploaded, changes in content, social network analysis etc.
• Focus groups, seminars, workshops, review meetings, observations, self-documentation, Participatory sessions / dialogue, assess secondary resources, village meetings
• Case studies to tell stories of change (Most Significant Change)
• Outcome Mapping
• Web stats
• Results oriented monitoring using logframe

The groups also highlighted the following points

• Use of visual media such as video / pictures / publications / brochures and manuals to document best practice
• More interaction, communication and networking to encourage upscaling and replication
• Assessment / dialogue with policy makers decisions makers
• We need to see if context / indicators change over time and if best practices are adopted
• Reporting and periodic reviews – reflect lessons learned and identify conclusions / outcomes

And finally there was healthy discussion of what we can do ourselves to improve what we are doing in this area with suggestions focusing around 3 key areas

Feedback

• Make sure you crate feedback mechanisms to generate relevant feedback from all levels
• Feedback should give the audience the change to tell their own stories and should be anonymous
• Process relevant feedback to identify lessons learned from broader actors perspectives
• Systematically convert lessons learned into conclusions leading to action
• Be ready to implement changes needed
• Take lessons from past activities and revise planning based on assessment results

Collaboration and consistency

• Improve institutional and collaborative approaches and continue to improve the way we use new media
• Encourage consistency across organizations
• Allocation of recourses to assessment and gear assessment to audience – make it meaningful to both people you serve and management
• Capacity Building and empowerment at all levels, including audiences and stakeholders
• Map out systems we are working with e.g. policy makers, farmers etc
• Encourage transdiscipinary approaches - multidisciplinary is not enough
• Make sure language is appropriate for audience/context
• Curriculum review

Across sector

• Share experiences and communicate how to measure change in KM forums and networks
• Develop a clear framework for KM activities and integration of KM activities within thematic groups
• Prove causality between knowledge use and behaviour change
• We need more ownership and commitment of government including budget commitment

Next steps
These discussions and inputs will be documented and shared with Share Fair participants and beyond to continue the dialogue started here in Addis. Hopefully over the coming months we can further refine these ideas and share our experiences to reach consensus as to how best to address the issues identified during the KM-Impact Session. We are currently in the process of designing a Knowledge Management Impact Challenge and would love to hear from anyone who has experiences to share or would like to get involved in helping to shape this initiative.