SEA Change CoP Evaluation Conclave and Members Meeting Report 2013

SEA Change Evaluation Conclave report
SEA Change: An Asian Community of Practice for Monitoring and Evaluation of Climate Change Interventions
Report title: Written by: SEA Change Evaluation Conclave report Dennis Bours
Report Covering:
Evaluation Conclave and SEA Change member engagement results 21 March 2013
Report Date:
1
Table of Contents
Executive summary .......................................................................................................................... 3 Introduction ..................................................................................................................................... 5 Section 1. SEA Change Climate Change Adaptation M&E panel session, complexity and attribution .................................................................................................................... 6 Summary of the group work process and results .................................................... 8 Section 2. SEA Change Climate Change Mitigation M&E panel session, M&E of climate investments and REDD+ MRV systems ..................................................................... 10 First theme: M&E of climate investments ............................................................. 10 Second theme: MRV systems in REDD+ programs ............................................... 11 Section 3. Section 4. Draft knowledge generation strategy ......................................................................... 13 Pre-Conclave Climate Change M&E Knowledge Needs survey .................................. 14 Profile of respondents to the M&E Knowledge Needs survey ............................... 14 Knowledge needs responses ...................................................................................15 Knowledge gaps responses ..................................................................................... 17 Section 5. Climate Change M&E Knowledge Needs panel session ............................................. 18
Conclusion .................................................................................................................................... 21 1. Continue doing what SEA Change is good at ..................................................... 21 2. Introduce new elements in line with the concept of knowledge generation ..... 22 3. Include new ideas outside the scope of knowledge generation ......................... 22 Annex 1: Annex 2: Annex 3: The Evaluation Conclave in Kathmandu, Nepal ........................................................ 23 Climate change and the environment, a Conclave sub-theme ........................... 23 List of SEA Change participants ................................................................................. 24 Draft SEA Change knowledge generation strategy..................................................... 26 Introduction / purpose .......................................................................................... 26 Knowledge generation strategy ............................................................................. 27 Annex 4: Annex 5: SEA Change knowledge needs survey full results ................................................... 30 Climate change M&E knowledge needs panel session results ....................................51 Part 1 of the knowledge needs group work: Knowledge needs results ...................51 Part 2 of the knowledge needs group work: Knowledge products results ............. 53
Page 2
Executive summary
With support from the Rockefeller Foundation, the SEA Change Community of Practice sponsored 25 members to join the Second Evaluation Conclave (Evaluation for Development) in Kathmandu, Nepal from February 26 to March 1 2013. More information on the Evaluation Conclave can be found in Annex 1. In collaboration with the SEA Change staff, SEA Change members facilitated three climate change M&E panel sessions at the Evaluation Conclave. The event also served as the third SEA Change Member Meeting. The first Member Meeting took place in New Delhi during the First Evaluation Conclave on October 2010, and it was at this Kick-Off meeting that the SEA Change Community of Practice was formally launched. A second Member Meeting took place in Bangkok in November 2011, marking the one-year milestone for the CoP following the Kick-Off meeting in New Delhi. The 25 SEA Change members who attended the SEA Change Member Meeting at the Evaluation Conclave in Kathmandu consisted of a range of monitoring and evaluation and climate change professionals, consultants and researchers. These members were selected by the SEA Change team lead based on proposals they put forward for presentations and their likelihood to make active contributions towards the Conclave. Annex 2 provides a list of SEA Change members attending. SEA Change members made the most of the opportunity to meet face to face with each other, to discuss progress in the M&E discourse and direction of the SEA Change community of Practice, to take part in the three SEA Change panel sessions (discussed in Section 1, 2 and 5) as well as other sessions and workshops, and to meet a broad range of other participants attending the Conclave. The first panel session was the Climate Change Adaptation M&E panel on day 1 of the Conclave, Tuesday, February 26 2013, focusing on complexity and attribution. A summary of the session can be found in Section 1. The Climate Change Mitigation M&E panel session took place on the second day of the Evaluation Conclave, with the themes M&E of climate investments and REDD+ MRV systems. An overview of this panel session is presented in Section 2. Prior to the Evaluation Conclave event, the SEA Change team developed a draft knowledge generation strategy. The aim of this strategy is to help SEA Change move from being a knowledge dissemination platform towards focusing more on knowledge generation. The draft strategy document is included in Annex 3 and an overview of steps forward is given in Section 3 of this report. An online Climate Change M&E Knowledge Needs survey was launched in advance of the Conclave event, to both inform the development of the knowledge generation strategy and to inform the SEA Change Member Meeting session at the Evaluation Conclave. The survey was distributed to all 500 SEA Change members; 110 members filled out the survey, and 80 of these 110 filled out both the closed as well as open question categories. A summary of the results is provided in Section 4 of this report and full survey results are provided in Annex 4. SEA Change members were invited to join a dinner on Wednesday, February 27, when both the results of the pre-Conclave Climate Change M&E Knowledge Needs survey and the draft knowledge generation strategy were presented. The discussion was continued during the SEA Change Climate Change M&E Knowledge Needs panel session on the third day of the Conclave, Thursday February 28. The SEA Change Climate Change M&E Knowledge Needs panel session was the main session in which members could reflect on the direction of the SEA Change Community of Practice with the members. After a brief review and discussion of the results of the online pre-conclave Knowledge Needs survey, participants engaged in a group work activity that further elicited the concepts and practitioner requirements for knowledge needs and gaps. A recap is given in Section 5 and pictures of the group work results can be found in Annex 5.
Page 3
The results of the SEA Change panel sessions, the SEA Change Member Meeting in Kathmandu and the pre-Conclave survey will be used to finalize the SEA Change knowledge generation strategy. SEA Change members will be asked to reflect on the final strategy following its completion. In the pre-Conclave survey, 77% of SEA Change members indicates that the SEA Change Community of Practice fits their climate change M&E knowledge needs, while 78% agrees that the SEA Change platform makes new, relevant knowledge available.1 Based on discussions, results of the Climate Change M&E Knowledge Needs panel session on day 3 and the pre-Conclave survey, recommendations after triangulating all the feedback are the following: 1. Continue doing what SEA Change is good at, with the biggest selling points mentioned being: Resource sharing Webinars Moderation.
2. Introduce new elements in line with the concept of knowledge generation, with an interest in analysis of existing knowledge products and development of new ones: Analysis and discussion statements Reviews of evaluations Working groups Development of e-learning materials.
3. Include new ideas outside the scope of knowledge generation ideas that introduce new functionality to the online platform and member network beyond knowledge dissemination and generation: Consultant / volunteer database Helpdesk function Indicator database More face-to-face and workshops.
Pre-Conclave SEA Change Climate Change M&E Knowledge Needs survey (see Annex 4 for full results).
1
Page 4
Introduction
The theme of the second Evaluation Conclave, which took place February 26 to March 1, 2013 was Evaluation for Development. The main focus of the Conclave was a very practical, meaningful one how evaluation should make a difference in the lives of people. The organizers and the Steering Committee decided to hold the second conclave in Kathmandu, in part due to its central location in South Asia for people to conveniently converge. More than 316 evaluators from over 22 countries across the world registered and attended the Conclave. The SEA Change Community of Practice sponsored 25 members to join, with financial support from the Rockefeller Foundation. The Conclave sought to attract global thinkers engaged in innovative evaluation research, theorizing, or practice who seek opportunities to push their thinking in new directions and are interested in applying their ideas in a South Asian context. Conclave organizers also sought to include leading development theorists, thinkers, activists and policy makers from South Asia to embed discussions in current development issues and contexts that evaluation must respond to. Partnerships and sponsorships were negotiated to help realize the Conclave. A robust mix of local and international organizations provided financial and/or in-kind support, and coordinated and hosted Conference sessions and strands. The Conclave would not have taken place without the assistance of core funders and other partner organizations, including the Rockefeller Foundation, that made available a range of grants and in-kind support towards the Conclaves core budget, mobilization of expertise, travel funds for participants, and the coordination of sessions and panels.
Page 5
Section 1. SEA Change Climate Change Adaptation M&E panel session, complexity and attribution
The SEA Change Adaptation M&E panel session took place on day 1 of the Evaluation Conclave and was chaired by: Jyotsna Puri (3iE), Tricia Wind (IDRC), and Marina Agpar (World Fish Centre) with panelists: Lucy Faulkner (ICCCAD), Bruce Ravesloot (TANGO International), Ram Chandra Khanal (CoE-Nepal), Albert Salamanca (SEI) and Tuan Doan (Save the Children). The Climate Change Adaptation M&E panel session was built around four short presentations on climate change adaptation M&E frameworks and principles, and two short presentations on complexity and attribution. Panel session opening, introduction and first 4 short presentations [VIDEO1] The first video shows Ms. Jo Puri of 3ie taking the chair with a short introduction, followed by four 5-minute presentations on climate change adaptation M&E frameworks and principles. Climate change adaptation type interventions are seen as more complex than for other development interventions and that evaluators should be mindful of their a) effectiveness did the intervention have the intended effect, b) sustainability, c) whether the intervention intentions would be adopted, d) timing of the measurement of impact/outcomes, e) context, f) measurability (i.e. to what extent they can be measured), and g) complexity in terms of timescales, actors and context. Impacts of climate change will fall disproportionately on developing countries and measuring climate change adaptation interventions in related contexts adds another layer of complexity to the M&E system. The first presentation was provided by Ms. Lucy Faulkner, Researcher on M&E for CBA for the International Centre for Climate Change and Development (ICCCAD), Bangladesh. Lucy presented a new participatory, multi-track monitoring and evaluation strategy designed to contribute to current knowledge gaps on how to plan, deliver and measure the effectiveness of community-based adaptation (CBA). Recently developed in conjunction with IIED for ICCCADs flagship CBA program, Action Research for Community Adaptation in Bangladesh (ARCAB), the ARCAB M&E for CBA methodology sets M&E adaptation priorities that focus on the needs of the climate vulnerable poor while simultaneously responding to the wider range of stakeholders and scales across and for whom information about effective CBA needs to be generated. Rapidly gaining recognition at international levels, the practicability and accessibility of this M&E approach to stakeholders in different adaptation contexts is currently supported through its use in INGO CBA interventions operating across differing ecosystems in Bangladesh and East Africa. Ms. Faulkners presentation can be accessed through the following link. Also, click here to read the study Moving Towards Transformed Resilience: Assessing CommunityBased Adaptation in Bangladesh. The second presentation/speech was provided by Mr. Bruce Ravesloot, Vice President TANGO International and Care Climate Change Adaptation Advisor, Thailand. Bruce discussed current applications of the TANGO framework for measurement and analysis of resilience, and its relevance towards the monitoring and evaluation of adaptation interventions. The framework draws on practical experience in M&E of food, livelihood security and DRR programming. Bruce emphasized that good and tested measurement approaches and indicators towards resilience already exist and measurement challenges in adaptation interventions remain the same. There is Page 6
possibly too much focus currently on the what rather than the how. We talk about what resilience is, what adaptation is, becoming too much an academic discussion on concepts. But we dont focus on how to measure it. Importantly, there are different nuances for particular organizations, context and entry points taken. When looking at the how, we also need to look at the higher level measurements and not get stuck on household level only. The third presentation was provided by Mr. Tuan Doan, Climate Change and Emergency Program Manager, Asia Regional Technical Advisor Save the Children, Vietnam. Tuan presented traditional knowledge, attitude, and practice (KAP) research methods in impact evaluation of climate change adaptation and disaster risk reduction in Vietnam. The fourth presentation was provided by Mr. Ram Chandra Khanal, General Secretary of Community of Evaluators Nepal, Nepal. Mr. Khanal presented a short presentation on current work of some of the programs/projects (e.g. TAMD, PPCR and CDKN) on M&E climate change adaptation in Nepal, and the results of those initiatives on consolidating learning. the the for the
After these four presentations, an opportunity for question and answers was provided for participants: Question [VIDEO2] and answering session
How does one effectively define attitudes at the local level? People at this level often regard these quite differently from the survey developers. Difficulties measuring attribution in longer term CCAI programmes. Is contribution a solution? But contribution to what?! How are local adaptation plans of communities taken into account in the frameworks/principles discussed? Can we get some more developing indicators for CCAI? On complexity and group work session [VIDEO3] Mr. Ram Chandra Khanal started this part of the session with a short discussion of development versus climate change adaptation interventions. He discussed whether complexity is always negative? There is a general change towards more results-based and participatory M&E in Nepal, but efforts and results are fragmented. Mr. Albert Salamanca, Research Fellow, Regional Climate Change Adaptation Knowledge Platform with the Stockholm Environment Institute (SEI), Thailand then asked the participants to think about the following issues (which would lead into the group work exercise) Which M&E framework offers the best potential to address complexity? What have been the experiences so far? What can we claim as the state-of-the-art? How do these experiences inform the planning of future adaptation projects? Page 7 guidance on
Summary of the group work process and results

The groups were asked to discuss the following: Understanding that climate change adaptation monitoring and evaluation must account for complexity. Are there any approaches and ideas you have from your own experience that might help address the complexity and issues like for example the ones Albert and other presenters just talked about? Participants were asked to write on cards the main points from their discussion. The cards were all collected at the end of the session. Groups were not asked to reach consensus to feed back to plenary, and in the time they had, it is unlikely that any group conclusions were reached. The comments shared on cards, therefore, give us a glimpse of what was discussed and of views held by those in the room rather than a representative or collective view from the room. The synthesis offered here aims to pick up on some of the main points of convergence and divergence to feed into on-going discussions through the SEA Change CoP and other forums. The transcribed notes from cards were clustered into eight themes: 1. 2. 3. 4. 5. 6. 7. 8. Terminology and understanding of complexity Contribution and attribution analysis Evaluation through reducing complexity Using systems approaches and recognizing inter connections Using participatory evaluation Supporting learning and considering time scale Using theory of change Specific M&E tools
1. Terminology and understanding of complexity / 2. Contribution and attribution analysis The first two themes reflect discussions regarding tensions that were touched upon during the presentations made by the panelists on understanding complexity, and whether it is a characteristic of the programs or processes being evaluated that influences the approach and methodologies taken and if so how one describes it (or whether evaluators just make things complex). Related to this is the question on whether contribution and/or attribution analyses are appropriate and possible in CCA evaluations. Questions in the earlier part of the session and Lucys, Bruces and Alberts presentations all discussed these inherent tensions, and efforts around measuring resilience are attempting to grapple with the challenge. Some of the comments point to using already articulated distinctions between complicated and complex as one way of clarifying terminology, while others indicate that complexity may be created by evaluators and therefore there is scope to simplify through improved definitions and terminology. Similarly some participants questioned whether attribution is possible in cases of CCA, and others point to the need to use contribution to understand both intended and unintended causal pathways. The question of whether aggregating contribution analyses can support whole program attribution analysis was raised earlier in the session and discussed by some groups.
Page 8
3. Evaluation through reducing complexity / 4. Using systems approaches and recognizing inter connections These two different approaches fall out of different interpretations of complexity either as a problem which can be reduced to its parts and experimental and quasi-experimental approaches are useful for assessing attribution (what some might call complicated), or, as made up of nonlinear interactions between parts that make reduction difficult and require understanding emergent phenomena and not isolating CC from wider development and change processes and can assess contribution (what some call complex). Other comments also seem to indicate that some groups discussed bridging approaches and methodologies as necessary; yet, it is not clear how far these discussions evolved. Nonetheless a message from some participants is that mixed approaches and methods might be useful. 5. Using participatory evaluation / 6. Supporting learning and considering time scale Discussions arose around evaluation approaches that are particularly useful in CCA settings. There were a number of comments on the need to use participatory approaches and take into consideration local and indigenous knowledge in CCA both to ensure buy-in and to address the local-global nature of the problem. There seems to be some consensus around this approach within the group. Related to this is the need to support learning and adaptation both to deal with quick changes as well as to support resilience long term. Comments were made about building M&E systems to support learning beyond a project life cycle and support sustainability and resilience. 7. Using theory of change / 8. Specific M&E tools More specific discussions around M&E frameworks and tools also took place. There was some discussion and agreement on the usefulness of a Theory of Change approach (used here to mean broadly program theory and logic model/framework approaches) to help identify what needs to be monitored and measured and how to do so, as well as helping to undercover causal/impact pathways across stakeholders and goals. Finally, some specific tools were mentioned and deserve some further future discussion on the SEA Change CoP website. To name a few; Including CC related indicators in census questionnaire, agricultural survey, to get data for evaluation; Resilience Indicators; Coping Capacity Index monitor changes of capacity index map, and Learning from the "margins" - i.e. positive deviance method.
Page 9
Section 2. SEA Change Climate Change Mitigation M&E panel session, M&E of climate investments and REDD+ MRV systems
The SEA Change Mitigation M&E panel session took place on day 2 of the Evaluation Conclave and was chaired by: Sanjay Saxena (Total Synergy Consulting), and Anna Williams (Perspectio) with panellists: Christine Roehrer (Climate Investment Funds Administrative Unit), Kapil Thukral (ADB), Kelly Hewitt (ADB), Umi Hanik (UNDP), Tim Larsson (Ross Strategic), Julien Brewster (Pact) and Stefan Bepler (RECOFTC). The climate change mitigation M&E panel session was split into two themes; the first theme discussed the M&E of climate investments, with a panel of five specialists. The second theme focused on MRV (Monitoring, Reporting and Verification) systems in REDD+ programs, with two presentations on community forestry in Cambodia.
First theme: M&E of climate investments

Panel session opening, introduction, presentations and panel discussion on the M&E of climate investments [VIDEO1] Dennis Bours, SEA Change team leader, briefly introduced the mitigation panel session including the chairs and panel members. He then outlined the organization of the meeting into two themes, the first being the M&E of climate investments, and the second being presentations of MRV systems in REDD+ programs. The first theme was chaired by Sanjay Saxena (Total Synergy Consulting), with panelists Christine Roehrer (Climate Investment Funds), Kelly Hewitt (ADB), Umi Hanik (UNDP) and Tim Larsson (Ross Strategic). Ms. Christine Roehrer outlined her work with Climate Investment Funds around the globe. She talked about the three year reform process in which the number of core climate change indicators had been reduced from around 30 to between 3 and 5 core indicators to make the M&E work manageable and doable across the 49 target countries. Streamlined systems are more efficient, and that it is important to establish an evidence base as soon as possible. Ms. Hewitt mentioned the importance of additionality, leverage and sustainability when considering the evaluation of climate investments and trying to accommodate the unknown role/forces of the private sector and market forces (and the fact that the language and terminology of these two worlds can be very different). Ms. Umi Hanik presented the on the Indonesian Climate Change Trust Fund with its overall objectives of a lower carbon economy and to increase government effectiveness. Mr. Tim Larsson outlined his global research work at Climate Works. He mentioned some of the evaluation work being carried out at the macro level which aimed to see whether results were being achieved, resources being allocated, the network functioning and the strategies/policies being modified. He said that due to this level of complexity, mixed method evaluation approaches were required.
Page 10
After the presentations, an opportunity for question and answers was provided for participants: Q&A session on the M&E of climate investments [VIDEO2] The Q&A session provided interesting questions and answers; the questions are provided below and the answers can be found in the video. A first question to Ms. Hewitt focused on project design and boundary lines: how do you define the boundary lines at the ADB Evaluation Office? Question two looked at the link between profit and effectiveness related to private sector involvement. A third question towards the entire panel focused on the role of participatory evaluation towards climate investments. Is there scope to do this due to the complexity? The fourth and final question was on analogies with energy efficiency is there weakness in evaluation because we are talking about markets? Answers of the panel to all four questions can be found in the recording, which can be accessed here [VIDEO2].
Second theme: MRV systems in REDD+ programs

MRV systems in REDD+ programmes, community-forestry examples from Cambodia [VIDEO3] After the presentations on the M&E of climate investments, two presentations on the development of MRV systems in REDD+ programmes were provided. A key challenge is on measuring results i.e., what are the proxies for mitigation and how to establish baselines and ensure the continuous and regular process of data collection through consistent/uniform standards. Measurement, Reporting and Verification (MRV) is used in REDD+ programming and essential in order to be compensated for the emissions reductions. The focus of the work is GHG emissions monitoring, Reference Emissions Levels, forest inventories and remote sensing. This session was chaired by Anna Williams (Perspectio), with presenters Stefan Bepler (RECOFTC) and Julien Brewster (Pact).
Page 11
Mr. Bepler focused on RECOFTCs field work on sentinel sites in Cambodia, using International Forestry Resources and Institutions (IFRI) methodology to establish baselines or community-forestry sites, and presented preliminary results and lessons learned. An important topic was time series data collection using IFRIs method, which can provide a good basis for measuring changes in carbon stock. Mr. Beplers presentation slides can be accessed here. Mr. Julien Brewster, Pact REDD+ monitoring and evaluation officer in Cambodia, delivered a presentation on their REDD+ program as a case study on how local community forest user groups are being engaged in monitoring a REDD+ forest carbon offset project in the North West Cambodian province of Oddar Meanchey. He discussed the overall progress so far, a range of issues being encountered at the government and community levels which are hindering progress, and the challenges and opportunities of taking a participatory approach to monitoring, reporting and verification (MRV) and meeting international standards for carbon monitoring. Mr. Brewsters presentation slides can be accessed through the following link.
Page 12
Section 3. Draft knowledge generation strategy

Prior to the Evaluation Conclave event, the SEA Change team developed a draft knowledge generation strategy in February 2013. The aim of this strategy is to help SEA Change move from being a knowledge dissemination platform towards a greater focus on knowledge generation, in line with the needs of the Community members. The results of the Knowledge Needs survey and the Knowledge Needs panel session at the Evaluation Conclave will help inform the finalization of the SEA Change Knowledge Generation Strategy (it is envisaged that the final strategy will be finalized by April 2013). The strategy involves the following steps to move from knowledge dissemination to more knowledge generation: a) b) c) d) e) Identify the Knowledge Target Population and their Knowledge Needs Knowledge Needs and Knowledge Gaps (2 different entities) Cash In On Member Capacities for Knowledge Generation Aligned Learning Formats and Knowledge Materials Continuous Learning and Improvement Cycle
The draft strategy document can be found in Annex 3. The required steps in the development of the knowledge generation strategy are illustrated in the following diagram:
Draft Knowledge Generation Strategy Evaluation Conclave Climate Change M&E Knowledge Needs
Systematic Reviews
Technical Working Group(s)
Final Knowledge Generation Strategy
Knowledge Generation Plan
Learning Formats / Knowledge materials
Importantly, the knowledge generation strategy will inform a knowledge generation plan which will list the specific knowledge generation activities.
Page 13
Section 4. Pre-Conclave Climate Change M&E Knowledge Needs survey

With SEA Change focusing on the monitoring and evaluation of climate change interventions and related fields, like livelihoods, disaster risk reduction, natural resource management, ecosystems, the climate change M&E knowledge needs of practitioners in these fields are relevant to the community of practice. In advance of the Evaluation Conclave 2013, SEA Change sent out a small survey to 500 members and a number of non-member M&E professionals. SEA Change received 110 responses, of which 80 were used for a complete analysis (which included answers to open questions). Below is a summary of the key survey results; the full survey results are provided in Annex 4.
Profile of respondents to the M&E Knowledge Needs survey

M&E Practitioners in climate change and related fields SEA Change members Male: 68 % Coming from: 13% 17% 67% 3% Female: 32 % Working in: 17% 80 % 88 %
3%
Asia
Europe Africa Americas 19% 61%
What would best describe the organization you are working for? International NGO
8%
9% 9% 11%
5% 30%
University / Think Tank Freelance consultant Multilateral (eg. World Bank, UN)
15% 13%
National NGO Private / Commercial business Government CSO / CBO
Page 14
What are the major program management and M&E areas in which you work? (Multiple answers were possible y-axis shows number of answers per category) 120 100 80 60 40 20 0 Climate Change Mitigation Climate Change Adaptation Agriculture, Livelihoods and Food Security Climate Change-related Policy Climate Change and Gender NRM / Ecosystems Management DRR / DRM Urban Climate Resilience Renewable Energy / Infrastructure Community-based Reponses Monitoring and Evaluation Design, Implementation and Management
Knowledge needs responses

How well do you feel you understand the concept of "M&E Knowledge Needs"? (A 6-point answer scale was used and 84 respondents answered this question)
50% 40% 30% 20% 10% 0%

Completely Very good Good Reasonable Basic Not at all
Do you have specific M&E Knowledge Needs in your day-to-day work? Yes No 85 % 15 %
Page 15
Could you describe or define, in your own words, "M&E Knowledge Needs? EXAMPLE 1 M&E Knowledge Needs, for me are the basic M&E Know how which are essential for any practitioner/organization or individual to facilitate M&E in their work environment. More important, this knowledge need is a must to extract learnings and lessons from your working environment. EXAMPLE 2 The need to expand and retain knowledge base, as well as provide case studies and comparable activities within a given framework. EXAMPLE 3 M&E Knowledge Needs are needs to produce information for strategic decision making. This can range from monitoring information to adjust program implementation to evaluation information to inform program design and organization approaches. Identification of needs must come before designing the M&E. EXAMPLE 4 M&E Knowledge Needs refer to a deficiency in intellectual foundation on monitoring and evaluation processes which has to be filled or improved in order to conduct a relevant, appropriate and effective monitoring and evaluation activity. EXAMPLE 5 M&E knowledge needs to me refers to the gap or lack of particular aggregated know-how, skills, experiences and understanding of M&E application on specific areas of practice such as CC adaptation, mitigation etc that can be met or covered by managed information (i.e. documented, processed, stored etc) which captures both the explicit and tacit (if possible) knowledge gained by various proponents in a particular field of practice. Ranking of most preferred Knowledge Products 8.30% 9.50% 17.90% 19.00% 19.00% 25.00% M&E Frameworks and their comparison Handbooks and manuals on CC M&E E-learning materials Sharing of good practices through discussions Relevant case studies
Webinars on useful M&E topics
To what extend did respondents agree or strongly agree with the following statements: (A 5-point answer scale was used: strongly disagree, disagree, dont agree nor disagree, agree, strongly agree. 84 respondents answered this question) The learning aspect of M&E is often forgotten 73.1 % (16.7 % neutral) Most of the M&E knowledge materials I use come from outside my organization 75.5 % (16.7 % neutral) M&E does not get the attention it deserves in the organization I work for 34.5 % (25.0 % neutral)
Page 16
Knowledge gaps responses

Do you have specific M&E Knowledge Gaps in your day-to-day work? Yes 81 % No 19 %
To what extent did respondents agree or strongly agree with the following Knowledge Gaps arise from: (A 5-point answer scale was used: strongly disagree, disagree, dont agree nor disagree, agree, strongly agree. 80 respondents answered this question) a LACK OF TIME to search for knowledge materials a LACK OF ACCESS to knowledge specialists a LACK OF FREE ACCESS to knowledge materials an ACTUAL LACK OF KNOWLEDGE materials the INABILITY TO FIND knowledge materials 73.8 % (18.8 % neutral) 66.3 % (17.5 % neutral) 57.6 % (20.0 % neutral) 51.3 % (28.8 % neutral) 47.6 % (27.5 % neutral)
Could you indicate particular climate change M&E Knowledge Gaps in your day-today work? Critical mass of clear well documented field based case studies that provides the evidence base for climate adaptation in a landscape context. Most of the information is ad hoc, anecdotal and not systematic and therefore difficult to apply. I couldnt find any specialist whom I could discuss and work with regarding the concepts of measuring climate change related indicators such as establishing baseline of GHG emission. I couldnt find any suitable reference that could best describe how to monitor a climate change project like Im involved in. SEA Change members only: To what extent did respondents (STRONGLY) AGREE with the following statements? (A 5-point answer scale was used: strongly disagree, disagree, dont agree nor disagree, agree, strongly agree. 73 respondents answered this question) The SEA Change platform fits my climate change M&E knowledge needs 76.7 % (16.4 % neutral) The SEA Change platform promotes knowledge exchange between members 74.0 % (19.2 % neutral) The SEA Change platform makes new relevant knowledge available 78.1 % (16.4 % neutral) How can we improve the SEA Change platform to better suit your Knowledge Needs? You are already presenting a lot of information. What would be more useful is an analysis of this (like what was done previously for M&E frameworks). This can be facilitated via user interaction or done in-house by SEA Change. Your opinion / analysis does not need to be perfect, but it will certainly help members forward in their own thinking. Commence online thread discussions with a hard statement on an M&E related topic. Its irrelevant whether SEA Change supports the statement it should be clearly intended to spark and engage constructive professional conversation. Page 17
Section 5. Climate Change M&E Knowledge Needs panel session

The Climate Change M&E Knowledge Needs panel session on day 3 of the Conclave was chaired by Dennis Bours, SEA Change team leader, with facilitators Paul Bacon (SEA Change), Romeo Santos (Workland M&E Institute), Leodegardo Pruna (TSU-CITC Consortium) and Andrew Zubiri (Climate Eval). The aim of the Climate Change M&E Knowledge Needs panel was to help assess the knowledge needs/gaps of climate change M&E professionals and of the SEA Change members in particular, and then look at ways to address and improve responses to these needs and gaps. As previously mentioned, SEA Change has already drafted an initial knowledge generation strategy document and participant feedback from this session will be used in the near future to finalize this document. The draft knowledge generation strategy document can be found in Annex 3. On knowledge needs [VIDEO1] Mr. Dennis Bours opened the Knowledge Needs session with a brief introductory icebreaker exercise to help indicate participants understanding of M&E and knowledge needs. This also enabled participants to introduce themselves and provided some background on their work. He then delivered an introductory presentation, which included a summary of relevant parts of the recent Climate Change M&E Knowledge Needs survey conducted by SEA Change. Some examples were given of survey respondent replies to the question, Could you describe or define, in your own words, M&E Knowledge Needs? and participants were given time to reflect on these examples. The presentation used can be downloaded through the following link. Mr. Bours gave directions for the first group work exercise. Participants were asked to write down their primary knowledge needs after discussing their knowledge needs in small groups, and then post them on a paper map according to the following 4 groupings: Tools/guides and technical knowledge Internal M&E feedback/learning Organizational knowledge needs Capacity development
Participants were asked to reflect on the positioning of all knowledge need cards on the map and were then asked to place 3 small dots on those cards most relevant to their M&E work, creating a scatterplot of what is seen as the most important knowledge needs of climate change M&E professionals.
Page 18
The results were, in order of frequency: Grouping 1 2 3 4 5 6 7 8 Tools/guides and technical knowledge Tools/guides and technical knowledge Tools/guides and technical knowledge Internal M&E feedback/learning Tools/guides and technical knowledge Capacity development Capacity development Organizational knowledge needs Knowledge needs of participants Information on indicators M&E frameworks related to climate change Methods for complex systems evaluations Balancing learning and accountability Methodologies for measuring resilience Uptake of research; gaps between research outcomes and practical learning Basic knowledge for those new to the topic Planning M&E and M&E design Knowledge gaps products [VIDEO2] and knowledge
The second video starts with a discussion on knowledge needs and commences with the difference between knowledge needs and knowledge gaps. One conclusion is that you can only talk about knowledge gaps if you really know what knowledge is out there. The split between needs and gaps might be of more importance to knowledge brokers than it is to climate change M&E professionals. The second part of the group work commenced with group discussion on the most preferred knowledge products. After ten minutes, each participant was asked to write down their single number one desired knowledge product on a card. The facilitators then collected these cards and Mr. Bours briefly read out some of the responses before closing the knowledge needs session and thanking all the participants for their contributions.
Page 19
An overview of preferred knowledge products is presented below, which will inform the SEA Change knowledge strategy: Answers given on preferred knowledge products Provision of online publications / media (6 cards) Case studies, best practice and failures documents available online (5 cards) Opportunities for sharing knowledge & discussion, i.e. online or face to face (4 cards) Knowledge and mentoring helpdesk / peer coaching / voluntary specialists to contact (4 cards) Practical application, field learning, exposure visits and training (4 cards) Training materials / guidelines / manuals (3 cards) Comparative reviews of different M&E approaches / frameworks (2 cards) Thematic discussion forum (1 card) Debates on core conceptual issues (1 card) Knowledge map with guidance and examples (1 card) Technology transfer (1 card) Could translate into: Resource library Resource library / webinars / discussion papers / technical working groups Discussion forum / webinars / e-meeting / technical working groups / conferences E-meeting sessions / helpdesk environment
E-learning environment / field trips Resource library / webinars / e-learning environment Webinars / discussion papers / working papers Discussion forum / e-meeting / technical working groups Improvement of resource library search function and visualizing inter-linkages between resources - not entirely sure -
Page 20
Conclusion
The Climate Change Adaptation M&E panel session on day 1 provided a range of useful discussion and presentations on background adaptation subjects for participants, which prompted some good all round thought provoking questions. The Complexity subject took a while to get going after the initial discussions and clarifications, but group exercise results proved to be thorough and revealing. Enabling continuity in the complexity/attribution debate via the SEA Change platform would be beneficial and recommendations towards ways of engaging in discussions and knowledge exchange are provided below. The panelists for the Climate Change Mitigation M&E panel session on day 2 provided some very relevant and useful up to date information on ongoing macro level work in the area of climate investments which included information on global budgets, indicator sets and current research initiatives. Question to the panel on profitability/links to the private sector were interesting, but moved away a little from the focus of the panel, being the M&E of climate investments. The REDD+ MRV presentations provided good technical background for participants on ongoing implementation efforts and issues arising from community/government engagement, and more information sharing on climate change mitigation M&E was also indicated by members and participants in the Climate Change M&E Knowledge Needs panel session as an area of interest. The results of the SEA Change Member Meeting in Kathmandu and the pre-Conclave survey will be used to finalize the SEA Change knowledge generation strategy, after which the SEA Change members will be asked to reflect on the final strategy. SEA Change does seem to add a real value-added in the daily work of the climate change M&E professional, especially given that 75% of professionals / SEA Change members find their M&E knowledge materials outside of their own organization. 77% of SEA Change members indicate that the SEA Change Community of Practice fits their climate change M&E knowledge needs, while 78% agree that the SEA Change platform makes new relevant knowledge available.2 Based on discussions, results of the Climate Change M&E Knowledge Needs panel session on day 3 and the pre-Conclave survey, recommendations after triangulating all the feedback are the following:
1. Continue doing what SEA Change is good at

Resource sharing: The resources shared are great, but some members indicate they want to see more resources being shared on the following topics: Climate change mitigation M&E CDM / GHG emission baselines National-level climate change M&E frameworks and systems Learning from M&E and knowledge management Adaptation/resilience indicators and linkages with existing agriculture/livelihoods indicators The M&E of climate change policy
It was also indicated that the search option of the online resource database needs to improve, which is something that is being worked on at current.
Pre-Conclave SEA Change Climate Change M&E Knowledge Needs survey (see Annex 4 for full results).
2
Page 21
Webinars: Webinars were frequently cited as a strength of SEA Change. Both member feedback and post-webinar surveys show how much these webinars are appreciated, and SEA Change members are asking for more webinars, including webinars with critical analysis by SEA Change. Moderator: The SEA Change online platform is well maintained and moderated, and continuously updated, which is appreciated by the members.
2. Introduce new elements in line with the concept of knowledge generation

Analysis and statements: Analyze knowledge products and give a SEA Change opinion on these resources, which can be done through either webinars or working / discussion papers. The analysis does not need to be perfect and members and other professionals in the climate change M&E field do not necessarily need to agree, but it should cultivate discussion on the topics being analyzed in order to forward members own thinking. Make statements on topics of interest to SEA Change members. The statements should be challenging and enthuse members to engage in discussion on the topics on which statements are made. This can be done either through the forum, open e-meetings or by blogging. Reviews: In line with the analysis of resources, SEA Change members indicate a need to learn from other members evaluations. Often such evaluations are not shared because they are internal to the organization, but there might be a role for SEA Change to anonymize the findings of such evaluations and develop evaluation reviews, describing the particular situation and lessons learned. Working groups: SEA Change members would be interested to be part of working groups that focus on specific climate change M&E topics, come together in an e-meeting environment to discuss the topics and develop discussion papers on these topics. Topics mentioned a number of times were: community-based adaptation, indicators, M&E learning and outcome mapping. Development of e-learning materials: Numerous SEA Change members indicated interest in e-learning materials or an e-learning environment, and even to participate in workshops / training being organized on location. There will need to be more needs analysis on the actual content and format of such e-learning and workshop materials, but it is certainly a direction to be explored within the knowledge generation strategy.
3. Include new ideas outside the scope of knowledge generation

Consultant database / Volunteerism: The SEA Change online platform currently allows members to review who is working in a certain climate change field and/or geographical area, and then contact that person. Members indicate they would like to have the option to search the database for members who have indicated to be available as consultants. One drawback might be that SEA Change cannot guarantee the quality of consultants, so a disclaimer would be necessary. Other members indicated they would like a similar kind of search option, but for members who have indicated to be available for volunteerism. Helpdesk function: SEA Change members would like the option to submit issues they have in their work, which could then be discussed during a one-hour open helpdesk session once a week. Professionals could be invited to take part in the helpdesk sessions, depending on the topics being discussed. Indicator database: Develop an indicator database, where members can search on type of intervention, level and scale and then get an overview of indicators that others have used in the past. More face-to-face: Everyone likes to meet face-to-face. The need for travel grants to attend SEA Change workshops and meetings was mentioned a good number of instances. Page 22
Annex 1: The Evaluation Conclave in Kathmandu, Nepal
The Community of Evaluators (CoE) in collaboration with the Community of Evaluators Nepal (CoE-Nepal) hosted the Second Evaluation Conclave from 26th February to 1st March 2013 in Kathmandu, Nepal. The objective of the conclave was to bring together over 400 thinkers, commissioners and practitioners of evaluation in an interactive forum to discuss, deliberate and share advancements in the theory and practice of evaluation. The Theme of the Second Conclave was Evaluation for Development emphasizing that evaluation should ultimately make a difference in the lives of people. Evaluation is particularly critical in the context of South Asia, home to complex social structures, high rates of poverty, gender discrimination, dynamic forces of globalization sweeping traditional societies and numerous development projects for the large populations of this region. Innovative evaluation approaches and practices are particularly important in such complex contexts. Though our focus is on the situation in South Asia, we will include broader experiences where comparative learning and lessons are offered. There are many reasons for which evaluations are commissioned for accountability, for learning, to understand what works and what doesnt, for utilization and influence. These are not mutually exclusive and indicate the varied potential uses and value of evaluation. The theme Evaluation for Development was aimed to draw out and share innovations in methods, approaches, capacity building and use with a special focus on the participation of civil society, local institutions and persons who are affected by the various projects being evaluated. The Conclave provided a platform for South Asian evaluators as well as others to share and learn from each other and contribute to evaluation field building in the region.
Climate change and the environment, a Conclave sub-theme

Evaluation of climate change (both adaptation and mitigation), agriculture and environment are important areas for evaluation in Asia that cross-cut with poverty, food security and gender issues.
Page 23
Annex 2: List of SEA Change participants

Name Albert Salamanca (Mr.) Andrew Zubiri (Mr.) Andry Napitupulu (Mr.) Aniessa Delima Sari (Ms.) Anna Williams (Ms.) Bruce Ravesloot (Mr.) Chanin Chiumkanokchai (Tom) (Mr.) Dennis Bours (Mr.) Jo (Jyotsna) Puri (Ms.) Julien Brewster (Mr.) Kapil Thukral (Mr.) Kelly Hewitt (Ms.) Kimsan Meng (Mr.) Leodegardo Pruna (Mr) Lucy Faulkner (Ms.) Marina Apgar (Ms.) Paul Bacon (Mr.) Function Research Fellow, Regional Climate Change Adaptation Knowledge Platform Junior Professional Associate Organization Stockholm Environment Institute (SEI) Global Environment Facility - Evaluation Office / Climate-Eval MercyCorps - SCALE-R project MercyCorps - ACCCRN project Perspectio Care Climate Change / TANGO Winrock / USAID LEAF Country Thailand
USA
M&E Specialist M&E Specialist ACCCRN Indonesia President Advisor / Consultant USAID LEAF Monitoring and Evaluation Specialist Team Leader Deputy Executive Director and Head of Evaluation REDD monitoring and evaluation officer Principal Evaluation Specialist
Indonesia Indonesia USA Thailand Thailand
SEA Change CoP International Initiative for Impact Evaluation (3ie) Pact Independent Evaluation Department, Asian Development Bank Independent Evaluation Department, Asian Development Bank SEA Change CoP Philippine Business for Social Progress-Business Advisory Program BCAS
Cambodia India Cambodia Philippines
Evaluation Specialist
Philippines
Project Officer Professor, retired.
Cambodia Philippines
Researcher on M&E for CBA and Research Coordinator for ARCAB Postdoctorate Fellow in Innovation and Evaluation M&E Expert
Bangladesh
World Fish Center SEA Change CoP
Malaysia Cambodia
Page 24
Name Ram Chandra Khanal (Mr.)
Function General Secretary Senior Evaluation Consultant and Nepal coordinator M&E Expert PM&E Consultant Deputy Director - Programme, Policy and Campaigns Capacity Development Advisor Managing Director M&E Officer Senior Associate Senior Program Officer, Evaluation Unit Climate Change and Emergency Program Manager | Asia Regional Technical Advisor M&E Specialist
Organization Community of Evaluators Nepal / GERES / CDKN
Country Nepal
Romeo Santos (Mr.) Rudini Nick Baoy (Mr.) Sajid Raihan (Mr.) Samnang Chum (Mr.) Sanjay Saxena (Mr.) Stefan Bepler (Mr.) Tim Larson (Mr.) Tricia Wind (Ms.) Tuan Doan (Mr.)
Workland M&E Institute Independent consultant ActionAid Cord, South & East Asia Office Total Synergy Consulting Private Limited RECOFTC Ross Strategic / ClimateWorks International Development Research Centre (IDRC) Save the Children Vietnam
Philippines Philippines Bangladesh Cambodia India Thailand USA Canada Vietnam
Umi Hanik (Ms.)
World Bank
Indonesia
Page 25
Annex 3: Draft SEA Change knowledge generation strategy

Introduction / purpose
The overall objective of the knowledge generation strategy is to set out the direction for the SEA Change Community of Practice moving from knowledge dissemination to more knowledge generation, in line with the needs of the Community members. The goal of moving towards knowledge generation is to add value to existing knowledge, informed by members knowledge needs, and developing knowledge materials to fill knowledge gaps and improve M&E knowledge and capacity of climate change M&E professionals to assist them in carrying out their work more effectively. Through this ongoing process, SEA Change will become a leading knowledge broker for climate change M&E knowledge development and exchange. This particular draft version leads into the Evaluation Conclave SEA Change member meeting that will take place in February 2013 in Kathmandu. The Conclave meeting will involve the participation of 23 SEA Change members and other interested professionals and will focus on climate change M&E knowledge needs. Feedback from this meeting and a Conclave climate change M&E knowledge needs panel session will inform the final version of this document.
Draft Knowledge Generation Strategy Evaluation Conclave Climate Change M&E Knowledge Needs
Systematic Reviews
Technical Working Group(s)
Knowledge Generation A preliminary knowledge generation Knowledge Generation Plan plan will be developed which will Strategy detail particular knowledge generation activities. In April 2013, Learning Formats / Knowledge materials after the knowledge generation work has commenced and a series of three systematic reviews has taken place, the knowledge generation strategy will be finalized.
So far the SEA Change platform and Community of Practice has achieved a respectable level of knowledge dissemination, involving the sharing of resources and communication of knowledge between the growing numbers of climate change and M&E Community members. The highest levels have occurred in the areas of M&E capacity development, climate change adaptation M&E frameworks & practices, the M&E of climate change policies and related fields like sustainable development, livelihoods, natural resource management and urban resilience.
Final
Page 26
Knowledge generation strategy

The SEA Change knowledge generation strategy feeds into the knowledge generation plan, which will address specific knowledge gaps of Community members more efficiently and new knowledge materials will be generated to achieve this. The SEA Change knowledge generation strategy should be seen as a conceptual blueprint that informs a set of required activities, detailed in the separate knowledge generation plan. The knowledge generation strategy consists of the following five steps: (i) Identify the Knowledge Target Population and their Knowledge Needs The initial target population comprises of the SEA Change Community of Practice members and other interested professionals in the field of climate change and related monitoring and evaluation. Apart from existing Community members there will be others outside who may (or may not) eventually join SEA Change CoP, including emerging professionals, partners and potential knowledge brokers in the field of climate change and related monitoring and evaluation. Identifying the target population is fairly easy at the onset, but its growth requires monitoring to check how the emerging knowledge generation can continually serve the correct knowledge target population with available resources. The identified target population will have specific climate change M&E knowledge needs. Some of these needs have already been met and can continue to be met by the SEA Change knowledge resource base and the ongoing dissemination of knowledge materials. However, knowledge needs have arisen/will arise that cannot be met by these current knowledge means and resources and these are the knowledge gaps. (ii) Knowledge Needs and Knowledge Gaps (2 different entities) Knowledge needs are defined as needs that can be met or fulfilled by existing knowledge resources contained in documents, materials, toolkits, journals, webinars, recordings etc (often referred to as explicit knowledge). The SEA Change CoP knowledge resource library are made up of the various documents, materials and tools that have been gathered and collected since the inception of SEA Change and are now available online for members and other interested parties. These knowledge products have already been identified as useful to our members and the wider community of practitioners in their work and have already helped in fulfilling knowledge needs Knowledge gaps are defined as identified knowledge needs that cannot be currently fulfilled by the existing knowledge resource base as part of SEA Change CoP or elsewhere (for example - there are no real case studies available on the implementation of well known climate change adaptation frameworks). Knowledge must be generated to meet these gaps via reviewing, recapturing, re-analysing, synthesizing and discussing existing knowledge resources. Importantly, there may be a crossover between Page 27
knowledge needs and knowledge gap categories especially at the individual level and for some organisations (i.e. depending on capacity levels, access to information issues and others). Identifying gaps between the existing knowledge base and whats required will require continual discussion and collaboration with SEA Change Community members, other practitioners and their emerging knowledge needs. Initially, their knowledge gap might only become apparent in a particular work or research situation. However, some more general knowledge gaps may be anticipated and periodic knowledge needs type questionnaires can be used to help inform this. As part of the knowledge gap identification process, it is important to help practitioners better define and articulate their knowledge needs. (iii) Cash In On Member Capacities for Knowledge Generation SEA Change CoP has a considerable range of expertise in its member base including climate change experts, M&E practioners and researchers across Asia and the world. If these resources are drawn upon and combined into appropriate online task type entities, particular knowledge gaps can be addressed. Also, individual experts may be identified outside the Community, and can be requested to discuss specific knowledge requirements with fellow SEA Change experts to help elicit more knowledge. To strengthen these human resources there should be a continual focus on the further developing linkages between SEA Change members and others (including other communities of practice, knowledge networks, etc.). Importantly, when these collaborate, they have the opportunity to generate more complex or tacit knowledge to help fill knowledge gaps (some of this type of collaboration between members has already previously been recorded via the SEA Change platform discussion threads, forums etc). (iv) Aligned Learning Formats and Knowledge Materials After the particular knowledge gap has been defined, and the technical/expert resources selected to address the knowledge gap, identifying the appropriate learning media, formats and materials must then be be considered to help the knowledge recipient(s) then solve their knowledge gap. In attempting to develop and generate knowledge materials, there are various possible learning media/formats and knowledge materials that need to be considered and aligned correctly to ensure effective delivery. E- Learning media/formats should always be considered at this stage, as they are a particularly effective to transfer knowledge. Once the particular learning media/formats have been decided upon, the actual knowledge materials can then be developed as required. The knowledge materials comprise the actual knowledge content - this is the implementation step. The available resources, costs and time to develop the selected knowledge materials should always be carefully considered. The following are possible established medium/formats that should be considered for the knowledge materials: One on One discussion / Expert sessions via skype or web interfaces E-learning mediums Webinars/Video streaming Page 28
E-reading and listening materials, like podcasts, pdf, e-paper etc. Typical e-paper guidelines, toolkits, discussion papers, background notes etc.
(v) Continuous Learning and Improvement Cycle The knowledge generation system has to be able to respond to new knowledge requirements and the system as a whole needs to be able to generate knowledge readily to meet demands. A suitable monitoring and learning feedback mechanism must be built into the knowledge generation system to ensure that its feedback informs its functioning adequately. The knowledge generation system should be reviewed periodically to assess its performance in terms of knowledge gaps successfully bridged, average response times to user knowledge needs, cost and other resources required to sustain and further develop the system. The knowledge generation system should aim to continuously improve and become more effective over time.
Page 29
Annex 4: SEA Change knowledge needs survey full results

Question 1 .Where do you work and where are you from? Currently working in Afghanistan Australia Bangladesh Belgium Belize Brazil Burma Cambodia Cameroon Canada China Ethiopia Finland France Germany Greece India Indonesia Iran Iraq Italy Japan Jordan Kazakhstan Kenya Malaysia Mauritius Mexico Morocco Nepal Netherlands New Zealand 2 6 10 1 1 1 1 8 1 1 2 1 0 0 1 0 4 12 1 0 0 0 1 1 5 1 1 0 0 4 0 1 Coming from 2 5 9 1 0 1 0 5 1 0 0 1 1 2 4 1 7 11 0 1 2 1 0 0 5 0 1 1 1 3 1 2 Page 30
Nigeria Pakistan Peru Philippines Sierra Leone South Africa Tajikistan Thailand Uganda United Kingdom USA Vietnam Zambia TOTAL ASIA EUROPE AFRICA AMERICAS TOTAL
3 7 0 5 0 1 0 10 2 1 9 3 2 110 67 21 19 3 110
3 7 1 7 1 1 1 3 2 2 7 3 2 109 68 14 17 10 109
Question 2. Are you male or female? Answer Options Male Female Response Percent 68.2% 31.8% answered question skipped question Response Count 75 35 110 0
Page 31
Question 3. What would best describe the organization you work for? Answer Options CSO / CBO Government (local or national) I'm a freelance consultant International NGO Multilateral (eg. World Bank, UN, etc.) National NGO Private / Commercial business University / Think Tank Response Percent 4.6% 8.3% 13.0% 29.6% 11.1% 9.3% 9.3% 14.8% answered question Response Count 5 9 14 32 12 10 10 16 108
Question 4. Are you an M&E practitioner working on climate change, or related fields? (eg. NRM, ecosystems, DRR/DRM, renewable energy, agriculture, livelihoods, food security) Answer Options Yes No Response Percent 80.0% 20.0% answered question skipped question Response Count 88 22 110 0
Question 5. What are the major program management and monitoring & evaluation areas in which you work? (Please check all that apply) Answer Options Design, Implementation and Management 49 45 38 30 34 33 30 24 21 16 Monitoring and Evaluation Response Count
Climate Change Adaptation Agriculture, Livelihoods and Food Security Community-based Reponses Climate Change Mitigation NRM / Ecosystems Management DRR / DRM Climate Change-related Policy Climate Change and Gender Renewable Energy / Infrastructure Urban Climate Resilience
54 49 39 38 35 37 24 22 20 14 answered question skipped question
67 61 52 50 48 48 40 33 30 23 102 8
Page 32
Question 5. Other responses: Number 1 2 3 4 5 6 7 8 9 10 11 12 13 Other (please specify) natural resource management and economic development Insurance, asset management (climate impacts focused) Sustainable Development Democratic Governance Water, Sanitation and Hygiene climate change modeling Education Governance - simplifying business regulations governance and climate change including HIV/AIDS Networking evaluation I work on M&E system development within an NGO to monitor their strategic plans Social sector (health, education, child protection, water and sanitation) Rural Infrastructure
Question 6. Are you a member of the SEA Change Community of Practice? Answer Options Yes No Response Percent 88.2% 11.8% answered question skipped question Response Count 97 13 110 0
Question 7. How well do you feel you understand the concept of "M&E Knowledge Needs"?
How well do you feel you understand the concept of "M&E Knowledge Needs"? Answer Options Comple tely Very good understanding 15 Good understanding 35 Reasonable understanding 21 Basic understanding 10 Not at all Rating Average Response Count
1 3.30 answered question skipped question
84 84 26
Page 33
Question 8. Could you describe or define, in your own words, "M&E Knowledge Needs"? No. 1 2 Response Text In fact, I am more guessing what it means, but it seems to be knowledge that is relevant for practitioners to monitor and/or evaluate climate change interventions. M and E Knowledge needs, for me are the basic M and E Know how which are essential for any practitioner/organization or individual to facilitate M and E in their work environment. More important, this knowledge need must be able to extract "learnings" and "lessons" from the working environment. The knowledge that could inform decisions related to the management of CC programs and development of strategies for climate change responses. It is a tool to disseminate the learning from a project that was found through M&E activities Knowledge gaps that I feel when carrying out/ communicating about M&E A framework of information or action to achieve the defined goal or target. It is a continuous process that is used knowing or unknowing in day to life to particular task. What I need to know in order to design, implement and assess monitoring and evaluation efforts for a particular evaluation or context. It is the needs of program to have good knowledge on the importance of M&E and the ability of using it to improve program implementation. The need to expand and retain knowledge base, as well as provide case studies and comparable activities within a given framework To be able to understand the benchmark baselines and then follow through changes resulting from interventions implemented that can then be attributed to the actions / activities done. Critical is the establishment of reliable baselines and then being able to establish causality / attribution in landscape situations where multiple variables are at play. Knowledge, methods, approaches, and data needed to support effective measurement and evaluation practice. Body of knowledge and skills that practitioners (or potential ones) need to develop / acquire so that they can contribute to the advancement and effectiveness of M & E. The needs can be either for understanding M&E concepts or its implementation. The practitioners have to have a sound understanding and only then implementation can be improved. This is the act of investigating/ field study of climate change on ecosystem stability as it affects agricultural production and its impact on the rural poor. It also entails the application of such information to ameliorate its adverse effects. Specific knowledge requirement. Causal attribution. Developing checklists and setting indicators The needs to target the useful indicators of M&E for a complete knowledge Although climate change adaptation and mitigation is emerging as a new sector, it's building on past programming experience, including that in M&E, in various sectors of livelihood, WASH, health, etc. The knowledge body is being built quickly and there is a need to manage and share among practitioners and beyond. Given the multi-discipline nature of climate change adaptation, this should go beyond climate change practitioners but also other related program stakeholders in livelihood, WASH, health, etc. Knowledge is that we may know as a practitioner what is the climate change and what is the program and project designed for that we are going to monitored either as real time are post evaluation the knowledge and information regarding the work that is under process Page 34
3 4 5 6 7 8 9 10
11 12 13
14
15 16 17 18 19
20
and implemented yet may be available and thoroughly studied by the practitioners So the community and implementers knowledge and policy of the country and what are the standards regarding the (science and particularly of the climate change) . Knowledge of the culture and people response and level how they understand the things related to program and project. The project design is the main thing which may open spaces for the stakeholders to monitor the program and project and give their input in the course of the implementation. M & E knowledge needs are what I would like to know in relation to M & E Sampling Techniques, Database (SPSS or similar) Knowledge required for an M &E practitioner to undertake his job efficiently. Setting of indicators is sometimes difficult Awareness of appropriate methods in various circumstances "M&E Knowledge Needs" is a tool essential to depict a comparative picture of pre and post intervention scenario, which facilitates effective design of future intervention adequately. Development of management and adaptation strategies relies heavily on constant monitoring and evaluation of risks. This then relies heavily on measurement and reassessment on a regular basis and updating of risk assessments. Designing logical framework. To ensure the implementation in line with the design plan and the output and outcome achievable at the end of the project by using M&E tools. M&E knowledge needs is the platform for M&E policy/system making, meaning each M&E system or policy should be designed by and to result knowledge M&E needs to start with baseline data for measurables, proxy measures for unmeasurables, budget for the process and a recognition that it needs to be mapped out at the start of the adaptation phase. M & E can learn from case studies but needs to recognize that there is no panacea M&E is to use for monitor the progress of project or organization operation and evaluate the impact of the project and organization. Sample designation, GIS, and advance SPSS. M&E Knowledge Needs means that assessing yourself as to what extent your current knowledge on M&E, identifying the gap, and determine what knowledge you need to enhance your capacity as M&E practitioner The need for knowledge and practice of M & E that will help us during the field. M&E needs to produce information for strategic decision making. This can range from monitoring information to adjust program implementation to evaluation information to inform program design and organization approaches. First and foremost, identification of the needs must come before designing the M&E Gaps in understanding to how utilize M&E methodology that is helpful in your position. Needs for improvements / updating knowledge Preparing result framework, data gap and reliability in the changing context, appropriate indicators, assessment tools M&E basic, intermediate and advanced knowledge areas a practitioner is supposed to be aware of Capacity Building on Monitoring & Evaluation The level of depth of M&E knowledge, further use of the knowledge to be obtained in dayto-day work, areas of particular interest/need in a view of forthcoming challenging changes in requirements to the working areas. M&E Knowledge Needs is the way to understand inputs, output, and outcome (if any) from the overall program design and implementation. This means that the knowledge to Page 35
21 22 23 24 25 26 27
28 29 30 31
32 33 34
35 36
37 38 39 40 41 42
43
44
45
46
47 48 49
50
51 52
53
54
55 56 57
understand each indicator and how far the program has achieved the indicator. Program details and implementation documentations would the most information for ME practitioners to process them into knowledge. The impact and the desired goals. This is the problem as the planning is not holistic and long term. Very, very short period of say 5 years is taken for setting the targets or goals. This is disaster. The systems are not for short term. They must be holistic and allencompassing by involving all the stake holders and not just few. It is a framework that needs by practitioner or program implementers in managing the program or project. The M&E Knowledge needs to safe guarding within the life time of program or project in order to keep on track of the program. In the framework we should set up a long term goal of the program, intermediate result and immediate result of the program or project with refer to five W plus one H (What we are going to do or to achieve within the project life time, where we will address the program to be implemented, When we will implement the program, Who will be the target beneficiaries, Why we are doing that and How we are going to do / program strategy). Those questions will be derived into a logical framework and M&E plan (develop Annual Plan, Semester, Quarterly and monthly plan into the detail activity and target output) set up the indicators , what input needs and what are the mean of verification to ensure what we are doing are on the right track to achieve the immediate objectives and in the long term could contribute positive impact to beneficiaries. General level - the program I work on does not focus on SLR or CCh aspects, though we do get invited to participate in workshops and provide industry feedback. thus my M&E knowledge is basic/elemental M&E Knowledge needs are the gaps that are needed to be closed in order to function effectively in the capacity of M&E. Sharing of information, more than "receipts" Not the faintest idea .. What does M&E stand for? Otherwise I work on interactions and feedbacks of the climatic subsystems (atmosphere- hydrosphere- biosphere- cryosphere and lithosphere) most of the time M& E in development is a relatively new area that has long been overlooked. Most organizations and individuals have quite a bit of catching up to do. Ideally we should attempt to focus our M&E with a specific learning agenda and ensure that we have the knowledge we need to do so. We should also commonly document our learnings so that we can increase our internal knowledge and that of others. This is technical and non-technical information that would equip an M&E practitioner with knowledge that would make them perform their duties better. Appropriate M & E require thorough knowledge of communities and ecosystems. Without grounded knowledge, development specialists can continue to mistake top-down or external policy interventions for real participatory development. To avoid objectifying "beneficiaries" the knowledge gathering needs to start at earlier stages, especially that of planning. M & E would then reflect the extent to which policies and programs are accountable to the expressed wishes of the receiving communities. Knowledge, skills, understanding of concept and tools that would facilitate extracting learning from project implementation in order to inform decision making and be accountable to donors. M&E Knowledge needs refer to a deficiency in intellectual foundation on monitoring and evaluation processes which has to be filled or improved in order to conduct a relevant, appropriate and effective monitoring and evaluation activity. The knowledge and skills required to design and implement M&E systems: conceptual, methodological and practical The tacit and explicit knowledge the an organization or program needs to perform better, which can be fostered and delivered through M&E activities and products PDCA cycle - plan do check and action Page 36
58 59 60
61 62
63
64
M&E Knowledge Needs would refer to the gaps that exist between what exactly the M&E Practitioner knows and practices and what ought to be done. The information and expertise required to monitor and evaluate climate change interventions Knowledge needs describe the information/data and understanding of practices, concepts and processes that are necessary to be able to interpret what is really happening in a specific intervention so that it can be monitored and evaluated. The need for more knowledge in the concepts/ theory and practice of monitoring and evaluation Not sure I would use this term myself but basically it is knowledge in the field that others want or need to strengthen their ability / capacity to design and conduct M&E. This could be data, methods, and results, capacity building such as facilitation techniques for evaluators or ways to conduct professional evaluations in different country/cultural contexts. Knowledge needed to conduct M&E daily work to cover knowledge gaps in the implementation of M&E. Often, we conduct M&E without strong knowledge so that may cause the results to be misleading, offended, not capture the right information, etc. By having information on the knowledge needed for M&E, one can focus to bring the knowledge needed and share it, so M&E practitioners can work better (I hope this is what it really meant). Needs in specific topics within the field of M&E. Lack or low level of understanding/knowledge in some aspects of M&E which needs to be learned. When related to climate change: Gaps in knowledge, information and understanding of climate change to better inform policy and understand effectiveness of programs (including adaptation). Related to this is how best to use the knowledge and information and how best to prioritize addressing M&E knowledge needs are knowledge products needed to facilitate one's work such as information of climate change adaptation or mitigation measures, case studies of good practices in climate change M&E, etc. M & E provides road to project objective success M&E knowledge needs mean the kind of knowledge we require to monitor the progress of a project/programme and to evaluate/assess the impacts (short term and long term). M&E mechanism is a kind of 'programme regulatory framework' which is meant to safeguard programme quality and impact by defining the indicators and methods/processes on the basis of intended outcomes and also to verify the SMARTness of such expected outcomes. However, there is a dire need of knowledge and skills to design quality indicators which are comprehensive in nature and most importantly that are formulated to assess the changes in lives and livelihoods of people living in poverty; e.g. whether we judge climate resilience in terms of "climate resilient agriculture" (a sectoral perspective) or in terms of "climate resilient farmer" (human perspective). There are also significant gaps in knowledge management e.g. how we generate knowledge through M&E and how that knowledge is incorporated in project cycle management. Conceptual and theoretical frameworks on top of practical experience and tools and methodologies in climate change monitoring and evaluation are the bolts and nuts for M&E professionals in the sector. For climate change and issues revolving around are vogue and at nascent stages so is availability of practically and theoretically tested knowledge basis for M&E in the sector. As per my perception, M&E Knowledge Needs can be, that how far M&E person or practitioner can shape the M&E Knowledge according to needs and according to situation. Either his knowledge is wide enough to accommodate all clusters/projects in M&E range and especially in Evaluation stage. Page 37
65
66
67 68
69
70
71 72 73 74
75 76
77
78 79 80 81 82 83 84
I am not aware about the term; it might be the knowledge (documents, training, webinar) which can help M&E work more efficiently and effectively. Knowledge and data/information that M&E professionals need to build up their expertise and need in their field works By my definition, M&E Knowledge Needs means opportunities to learn, share, reflect, and practice the knowledge. This is about the critical gaps/weaknesses in knowledge systems of a particular domain (say climate change) wherein M&E is an integral part. In other words, it is not about M&E as a practice or discipline where knowledge and experience are already abundant but about applying M&E in other development interventions. Sharing of professional information to facilitate M&E - - discussing norms, processes, methods, challenges, and possible solutions. "M&E knowledge needs" to me refers to the gap or lack of particular 'aggregated knowhow, skills, experiences, and understanding of M&E" application on specific areas of practice such as CC adaptation, mitigation, etc. that can be met or 'covered' by managed [documented, processed (analyzed/synthesized), stored, and disseminated] information, which captures both the explicit and tacit [if possible] knowledge gained by various proponents in a particular field of practice. - From knowledge generation (bad and good practices), monitor what works and what doesn't works, why? - identify knowledge gap and corrected measures - participatory actors' assessment (level of interest, power, knowledge generation, etc) - development of e-learning programs Concepts, tools systems that I need to know/have access to Is knowledge for managing for impact of a development intervention project Any resource, experience or lessons learned that would be relevant to M&E and satisfy the needs in my relevant area of work. What needs to be known to ensure development interventions are timely, cost effective and create real benefits (impacts) for communities The information and knowledge requirements to perform monitoring and evaluation. This should include both M&E for program design as well as evaluation. What and how to monitor and evaluate impact of CC interventions on both the natural and human environment The data, information, analytical tools and thus knowledge needed to design and implement an M&E framework for projects/policies.
Question 9. Do you have specific M&E Knowledge Needs in your day-to-day work? Answer Options Yes No Response Percent 84.5% 15.5% answered question skipped question Response Count 71 13 84 26
Page 38
Question 10 - Could you describe your M&E Knowledge Needs? No. 1 2 Response Text Most important one: how can M&E results really inform proactive adaptation and not just be a reporting tool? For me personally, my M and E knowledge need is to "document" the lessons which I am learning from my own experiences and from my/project success/failures. They remain with me and am able to share only when I SPEAK at forums. Still exploring some ways/means to put them in writing and share it with outer/inner world. and more important for your own reference, as I do not want to re-invent the wheel again and again. - Methods, tools and approaches for M&E and learning - Specific indicators for CC adaptation and mitigation at national and program level Website, Fact Sheet Methodology that will help the analysis of qualitative data. To improve the effectiveness/efficiency of work according to objectives/goals and provide the feedback for coming work, M&E knowledge is vital. Because I work on a range of different projects, my M&E knowledge needs vary considerably. I know the basic steps for designing and conducting M&E but depending on the context, I may need to know specific technical information about different kinds of climate change interventions. That's what is so great about SEA Change - it covers a range of topics from which I can choose when I need to. There is an M&E system in place, and it provides information for Program Manager as the input for decision making process and for the program to understand how much they already have achieved. n/a I oversee project / program implementation in different sites and always need to track the connection between our actions and changes to the quality of local community livelihoods, and the changes in the biophysical conditions of the natural environment New frameworks and methods for conducting climate change-related evaluations, addressing program governance (including resource allocation), policy change, and investment project strategy. Creative approaches for presenting evaluation information and results in written reports and presentations. I have fairly good background knowledge in the design and implementation, but not sufficient practical experience as an independent evaluator. My own M&E experience is related to my programme management experience at a UN organization. It is for both academic and action research purpose. The M& E knowledge needs include but not limited to the following: sharing information on known impacts/case studies eg water resources, soil degradation etc Data analysis to fit different information needs. The ability to link the consequences of climate programs with their impact on emissions and resilience. Training on Result bases M&E, Developing M&E results checklists, indicators and methodology. How to choose the good indicators of M&E? Pilot and documentation of M&E methodologies, e.g., KAP. 1-Standards and uniformity is a matter in understanding the things for example local knowledge is limited so how to bring the things in a streamline - this will be an issue all the time for the practitioners 2-Evaluating the things I am not much expert at these things through software Page 39
3 4 5 6 7
9 10
11
12
13 14
15 16 17 18 19 20
21
22
23 24 25 26 27
28 29 30 31 32 33 34
35
36
37
38 39 40 41 42
43 44 45
3-I need a software training which may do the analysis of the data quantitative and qualitative - Different books on M & E - Variety of M & E reports - Intensive training in results based M & E - Sampling - SPSS programming/data management - Online Monitoring The resources required for the job are lacking Setting indicators for government implemented projects. Particularly for Bangladesh Climate Change Trust Fund projects. How to combine environmental and socio-economic M&E. I always possess a task-list in bullet points which is reviewed by and large every night prior to a working day This is a highly dynamic area and as such there's a need for regular, constant updates of data, information, background scientific knowledge, etc. Forums and easy e-access for exchanging information and knowledge is therefore crucial. FGD report preparation. Progress Report of the project, baseline data available M&E knowledge management Need free access to data I need to Monitor all the time the project operations and for Evaluation I need to use for reflection the gaps between planning and implementation. Advance SPSS useful also, Database Development, GIS I would like to have a better understanding on how to best evaluate the climate change adaptation project in forestry sector. I would like to be able to establish an index, creating tools, to provide a robust monitoring & evaluation work in my project. My need for M & E is the knowledge and practical experience of other places related to how the design of the program was made, how the implementation, and how it impacts on beneficiaries. Also how the design and implementation of M & E is carried out in accordance with the local culture. So the M & E tool which is based on the local culture is very important to me. There is quite a bit of info on approaches that produce impact, but very little that can help inform decision making about how to bring effective programs to scale. There is a need for process indicators for this. Currently I'm learning how to conduct an SROI (social return on investment) workshop, which is intended to guide all participating stakeholders to jointly develop a numerical representation of the benefits attributed to an educational intervention. CC adaptation and mitigation indicators, impact M&E Good practice guide Designing frameworks, data collection and analysis tools. Training in Monitoring and Evaluation Define Climate Change adaptation and resilience indicators to measure progress towards urban and regional climate adaptation and/or resilience In respect of climate change, M&E knowledge needs would be nationally and globally accepted M&E framework, together with budget allocation for M&E data collection, correct method of measuring M&E indicators etc. Using log frames and internalize log frames to all teams especially the frontliners (ex. project officer in the field). - How do we have holistic in mind? - Do we care for the nature and the future sustainability? In terms of how we can ensure that our plan meets with the long term goal and fulfilling Page 40
46
47 48 49 50
51
52
53
54
55 56 57 58 59 60
61 62
63
64
achieving the immediate objective and outcome (SMART indicators is sometime hard to be achieved especially for the qualitative program approach). It is a field I would like to explore and participate in on a personal level and maybe incorporate in a professional level, particularly the resiliency and habitat restoration and adaptation aspects. Transiting from M&E to Knowledge Management Comparative data, practices, etc. May be based on an interdisciplinary approach I feel that I need to keep up to date with the various M&E debates and new trends but finding out what is relevant and what is not is difficult in an ever changing environment. Repositories where we can find out specific tools and information on an ad-hoc basis is also welcome. I would also very much like to know what doesn't work but unfortunately people are always very reluctant to document this. With the ever changing paradigms and concepts of project interventions, there is need for M&E approaches to change as well. Information on new ways of monitoring and evaluating programs to suit a certain paradigm is therefore paramount. There is therefore need to share M&E approaches for specific contexts as much as possible. I work with very rural, very impoverished, uneducated communities. I cannot be on site all the time to update their information for evaluation. Further, they cannot access the Knowledge Products in number 11 below, nor do they have the means to stay in contact on a regular basis. I need a knowledge chain that the poorest can access whenever necessary. Understanding more on the M&E theories and debates as well as practical use of M&E. E.g. are randomized controlled trials really possible in the real world setting? Can Theory of Change provide a more holistic picture than logframe approach? If so, how do I use it? What is the best way to analyze monitoring data so that the team can use it to adapt plans and strategies? The numerous resources made available by various national and international agencies, i.e., World Bank, Asian Development Bank, UNDP, SEA Change, others to M&E practitioners I believe are sufficient , if not more than, enough to fill and support our needs in the conduct of our work. - Approaches to M&E for Learning - Conceptual and methodological tools for measuring contribution I coordinate a large program of monitoring and evaluation activities - its set of models - activity>input>output>outcome>impact 1. Designing the theory of change, 2. Designing Evaluation methodologies In my programming and M&E work, i sometimes lack the information, indicators to fully monitor and evaluate the effectiveness of the work we are doing As a consultant who undertakes M+E I have requirements for many types information and data and tools and other peoples experiences in order to design an appropriate approach to evaluate a project or programme. How can i perfectly do the monitoring and evaluation of some programs we implement Mostly methodological knowledge; e.g., when a comparative case study analysis would be viable versus not. ways to track and evaluate the scaling and replication of mitigation interventions. etc. etc. Practice for developing logframe, terminology used by many institutions (often, different institutions use different terms but the same meaning, vice versa that caused confusion), practicing integrating gender for the M&E, Practice to develop good indicators in different types of field, Good or Best practices in M&E, monitoring resiliency in terms of climate change adaptation, relevant case studies. - M&E Framework / - Evaluation Design / - Assessment Report Writing Page 41
65 66
67 68
69 70
Not specific as it will vary from project to project. a) Good M&E practices of organizations and countries as applied to M&E of climate change adaptation and mitigation programs, b) Developing program logic models and indicators in CC adaptation and mitigation programs and projects, c) Designing evaluation frameworks and tools for CC adaptation and mitigation programs Harmonization of M& E framework - developing indicators - developing log-frame - systems and processes of M&E - how to write evaluation report Impact evaluation and knowledge dissemination and presentation. In addition, the interplay between policy discourse and formulation and implementation vs. M&E. - How to Monitor/Evaluate Social Protection cluster, - How to Monitor/Evaluate Health Cluster, - How to Monitor/Evaluate Education Cluster, - How to Monitor/Evaluate Child Protection Cluster, - How to Monitor/Evaluate DRR/DRM Clusters, Keeping in view all the above mentioned clusters, i want to extend my knowledge of monitoring by preparing questionnaire in the related fields, which will eventually help in collecting data/information for evaluation. Online resources, training, discussion platform n/a My M&E Knowledge Needs means opportunities for action, learning, reflection and planning within personal individual time and group membership event (sharing and discussion). I absolutely value the opportunities to learn, design, plan, implement and reflect with organizational team and other M&E professionals. How to make M&E part of the workflow of researchers or technocrats. Need to know - given context and components of evaluation - which definitional norm(s), processes and method(s) to apply the apparent dearth of substantive M&E framework & actual evaluation examples that demonstrate methodological & theoretical innovations in dealing with CC adaptationmitigation interventions, including how country-sector-project domains are dealt with, to say- analyze how Results are influenced when looking at them from segregated vantage points [e.g. sector only, or project only] or when integrated. Added to this is the scenario of the interplay of various other domains such as private sector interventions, natural resource management, DRRM, energy, etc. - learning from others by sharing good practices through discussions, - planning change and corrected measures I need more technical statistical understanding. I am good on the broad concepts and strategy M&E with specifies in Climate Change and Food security Project Guidelines, innovative approaches and procedures to M&E, country specific M&E update. How Adaptation knowledge needs coincide (or don't coincide) with development knowledge needs, in other words how would an M&E framework for adaptation be different from an M&E framework for development? Keep up to date on new knowledge products Need to better understand techniques to monitor climate change mitigation in the natural environment and how to measure both trade-offs and synergies with biodiversity conservation Familiar with the basic processes and data needs
71 72 73
74 75 76
77 78 79 80 81
82 83
84
Page 42
Question 11. Please rank the following "Knowledge Products" from 1 being the most important to 7 being the least important, in relation to your M&E Knowledge Needs. Answer Options Relevant case studies M&E frameworks and their comparison Handbooks and manuals on (climate change) M&E Background notes on (climate change) M&E Sharing of good practices through discussions Webinars on useful M&E topics E-learning materials 1 8 21 16 1 15 7 16 2 12 17 12 11 8 11 13 3 18 12 14 7 16 9 8 4 16 4 11 9 18 12 14 5 11 8 11 15 14 10 15 6 7 15 10 19 4 7 12 7 10 22 9 Rating Average 3.94 3.40 3.70 5.04 3.67 Response Count 84 84 84 84 84 84 84 84 26
23 12 4.48 6 12 3.77 answered question skipped question
Question 12. To what extent do you agree or disagree with the following statements? Answer Options Strongly disagree 1 3 Disagree Don't agree, nor disagree 14 14 Agree Strongly agree 14 7 Rating Average 3.58 3.19 Response Count 84 84
The learning aspect of M&E is often forgotten Knowledge management does not get the attention it deserves in the organization I work for Most of the M&E knowledge materials I use come from outside the organization I work for Monitoring and evaluation does not get the attention it deserves in the organization I work for
16 26
39 34
18
34
21
3.75
84
26
20
21
2.92
84
answered question skipped question
84 26
Page 43
Question 13. Do you have specific M&E Knowledge Gaps in your day-to-day work? Answer Options Yes No Response Percent 81.3% 18.8% answered question skipped question Response Count 65 15 80 30
Question 14. To what extent do you agree or disagree with the following statements? Answer Options Strongly disagree Disagree Don't agree, nor disagree 22 Agree Strongly agree Rating Average Response Count
Knowledge Gaps arise from the INABILITY TO FIND knowledge materials Knowledge Gaps arise from a LACK OF TIME to search for knowledge materials Knowledge Gaps arise from a LACK OF FREE ACCESS to knowledge materials Knowledge Gaps arise from an actual LACK OF KNOWLEDGE MATERIALS Knowledge Gaps arise from a LACK OF ACCESS to knowledge specialists
17
35
3.23
80
15
49
10
3.79
80
16
16
37
3.44
80
15
23
31
10
3.43
80
12
14
42
11
3.63
80 80 30
answered question skipped question
Page 44
Question 15. In your opinion, how can Knowledge Gaps be best addressed? Answer Options Response Percent Response Count 16 18 31 15 5 80 30
Online discussion threads and knowledge 20.0% sharing Provision of synthesis/meta-analysis on 22.5% knowledge gaps Setup technical working groups to address 38.8% needs Gap analysis type exercises 18.8% Other (please specify) answered question skipped question Question 15 Other specified:
Learning for doing Bringing the relevant civil society, local community members on board All of the above really Including online knowledge sharing and material access Group Discussion/ Forum
Question 16. Could you indicate any particular climate change M&E Knowledge Gaps in your day-to-day work? No. 1 2 3 4 5 6 7 8 9 10 Response Text How are M&E results being used in adaptation projects? - Relevant material and skills - Any platform/expert group to clarify my doubts Case studies on development of national M&E frameworks for climate change. How to measure the reduction of greenhouse gas emission Framework for data analysis How does technical M&E knowledge transfer to community people easily and understandably? Depends on the particular project(s) I'm working on at a particular time. But for right now, I am not facing an M&E knowledge gap. Confusion on up to what extent the implementation of M&E is regarding as climate change M&E? n/a Critical mass of clear well documented field based case studies that provides the evidence base for climate change adaptation in a landscape context. Most of the information is ad hoc, anecdotal and not systematic and therefore difficult to apply Frameworks for assessing policy change. Frameworks for conducting organizational and key actor landscape assessments. Frameworks for conducting and presenting evaluations of governance and resource allocation decisions. Frameworks for assessing gender and climate issues, and stakeholder/community involvement. - Computing carbon footprints of organizations - More structured insight into climate modeling (without being an expert in the field) Page 45
11
12
13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
32 33 34
35 36 37 38 39 40 41 42 43 44 45 46 47
Relationship between climate change and M&E, - Lack of information materials - Inadequate training Did that above. Training & documentation. n/a There is lack of standardized indicators for climate change adaptation. 1-How we can give a good input to our own ecology for example protecting our own species and lands and greeneries and lakes etc. Knowledge on climate change and related issues Data collection tools and analysis Combined environmental and socioeconomic M&E. To assess effectiveness of infrastructure projects Absence of active and vibrant discussions forum Actual verified linkages between climate change (extreme weather events) and infrastructure and economic damage. Questionnaire making and data analysis. The hand book and the guidance on how to conduct good monitoring and evaluation on climate change project - module or manual for measuring sustainability - standard indicators for output and outcome Not really understanding why the globe change this why I still use the old system for M&E in my daily work. n/a I couldn't find any specialist whom I could discuss and work with regarding the concepts of measuring climate change related indicators such as establishing baseline of GHG emission. I couldn't find any suitable reference that could best describe how to monitor a climate change project like I'm involved in. GIS, Database,Research bahan material sering tidak cocok dengan permasalahan lapangan yang kita hadapi There is limited work being done that links the extensive experience from sectoral M&E (food security, DRR, ag, local governance etc) to inform CCA M&E. Too much focus on doing something "new", means we are not using what we know already N/A. While I am still an evaluation practitioner, I am not currently in the climate change field. Indicators, impact M&E Data and result framework Not applicable - Lack of Time - Lack of access to knowledge specialists Nationally and Globally accepted M&E framework. The difficulties to provide sufficient information on the indicators achievement. wrong notions about the success n/a I have basic knowledge, so am unable to contribute. I do not know what i am missing Knowledge management None I really had to cope with Primarily I would be interested on the intersection of climate change with other thematic areas like gender, health, empowerment etc. In order to mainstream climate change we need to have a much better understanding of how we measure the intersection of climate change (causes and effects) on other areas of work not strictly labeled as climate change. Page 46
48 49
50 51 52 53 54 55 56 57 58 59
60 61 62 63 64
65 66
67 68 69 70
Designing databases and manipulating standard data processing software. 1. Local, especially agriculturally-based, communities in Southeast Asia face climate change on a daily basis through the damage to their land, crops, water, and livelihoods. Yet they have no means to report to local or regional policymakers and researchers. Rural people are the majority in e GMS and are bearing the worst of climate change yet they still seem to off the map in terms of policy dialogues and M & E input. Finding a balance between rigor and minimizing resource use. I have not so far met such a problem. - M&E for learning and accountability - how to do both together and well - Methodologies for evaluating contribution No real CC related knowledge gap in my day-to-day work, Lack of time to know the updated information as I am not directly involved in Climate change - Background information on climate change M&E - Designing performance measurement indicators for climate change It is sometimes difficult to track down guidelines, tools and frameworks when you are a freelancer and don't have access to organizational libraries or data. How do you monitor enhancement of resilience either to a community or ecosystem We encounter lack of free access and the actual lack of knowledge materials - Ability to extract and extrapolate from case studies - Availability of comparative case studies - Frameworks for tracking policy advancements and contributions (of the party being evaluated) to the advancements. This is particularly hard for demonstration/replication projects or scaling projects. - Counterfactual methods - Methods for influencing policy change (which is most almost always inherently opinion based) Balancing between measuring results and measuring process. Monitoring impact of climate change on various socio-ecological systems and how they respond to climate change impacts International standard for climate change a) Developing sound M&E frameworks including establishing logic models and indicators b) Formulating sound evaluation methodologies and tools - For M&E on climate change adaptation, we need a localized climate impact scenario to design our programs and decide on our intended outcome. Community people need scientific knowledge and information; vice-versa the scientists need to know the local contexts and needs and capabilities of the climate affected people to come up with a solution, for example, technology. Unavailability of well-versed technical professional to rely on, and absence of proven knowledge material on approaches and methodologies to conduct assessments. I think my climate change M&E knowledge Gaps in day-to-day work include a lack of free access to knowledge materials of M&E related to climate change as well as knowledge specialists in M&E related to Climate Change. - How to develop a questionnaire for a particular cluster or develop a universal questionnaire, which can be shaped according to ones need by few amendments. None n/a - How do you monitor and evaluate improvements in adaptive capacity wherein outcomes may only materialize beyond the lifetime of a project? - How do you monitor and evaluate social and ecological resilience when temporal factors are at play? - How do you convince donors that traditional LFA is too linear? Page 47
71
72 73 74 75 76 77 7880
- How do you monitor complexity? Some gaps exist in carrying over non-sovereign related M&E definitional norms, processes, and methods to climate change M&E. For example, despite existing definitions in private sector operations/evaluations regarding leverage, additionality, and transformation - there exists some disconnect or reluctance in carrying over such definitional norms when evaluating climate change related funds. Sample M&E on CC adaptation-mitigation No, I am directly working on climate change issues. I am learning from others through elearning programs, webinars and conferences. n/a Practical Analysis of Impact of Mitigation and Adaptation for food security water and sanitation As above Adaptation M & E vs 'development M&E n/a
Question 17. For SEA Change members only: To what extent do you agree or disagree with the following statements? Answer Options Strongly disagree Disagree Don't agree, nor disagree 12 14 Agree Strongly agree Rating Average Response Count
The SEA Change platform fits my climate change M&E knowledge needs The SEA Change platform promotes knowledge exchange between members The SEA Change platform makes new relevant knowledge readily available
3 3
2 2
42 37
14 17
3.85 3.86
73 73
12
34
23
4.00
73
answered question skipped question Question 18. For SEA Change members only: How can we improve the SEA Change platform to better suit your Knowledge Needs? No. 1 2 Response Text The webinars are great, and it is especially helpful that they are being recorded and our easily viewable though your website! Present moderator is doing BEST and justifies his role and position. Extremely relevant posts he shares. More important is now to generate discussions and evolve practices. Once in two years, a technical thematic meet is important where practitioners can share and also debate on existing knowledge. To disseminate guidance on M&E Knowledge Needs, since not everyone has internet access all the time to be able to join a webinar I don't think it's anything SEA Change can do, but I just need more time! (like everybody Page 48
73 37
3 4
5 6 7 8 9 10 11 12 13 14 15 16 17 18
19
20 21 22 23 24
25 26 27 28 29
30 31
32
else, right?) The discussion forum could probably be expanded, tied to other information and webinars on the site, as well as expanded I am still getting to know the platform as a new member, so I do not have ideas yet of how to improve the platform. More interactions among members to identify their needs and address them accordingly. By providing more reading material, updates about events, discussion forums Participating in workshop & seminars and training, e-learning & reading manuals etc I'm new to the SEA Change and will need more time to understand better how it works. No such ideas / specifics however I am not able this time to give an idea but I understand that the things are going on will refine itself with passage of time - Arranging seminar/workshops/training - Field/exposure Visit/More sharing So far so good. Facilitate interaction between members interested in similar issues. Setting up Working Group(s); E-conferencing; Just keep doing what we're doing, Always looking for improved ways to communicate and exchange knowledge. Conduct workshops or training for the members on climate change M&E since not all of the members are specialist in M&E For the two statements above that I don't agree, actually the SEA Change has provided the access but the member is not quite active so the knowledge exchange is particularly conducted in a passive way, through newsletter If it is possible SEA Change needs to organize a workshop/conference for all members to share and learn the experiences from each other and especially build up the network among themselves. Conferences and love-ins Not sure, I am a new member in this group Request for some practical training on climate change research, GIS, and Database development.... Multiply the sharing of experience and create online training for members in order to reach all You are presenting a lot of info. What would be more useful is an analysis of all this info. Like what you did on M&E frameworks. This can be facilitated through user interaction or done in-house by SEA Change. Your opinion/analysis does not need to be perfect but it will certainly help members forward in their own thinking. Organize webinars Online tools E-learning, distance group discussion with specific topics, field M&E exercises. Learn from doers, take holistic look at the things, have happiness as the index, trust in womens on the programs, short and long term success parameters. Conduct regular face to face events to share good practices among the Southeast Asia Countries and global as well, and provide fellowship for member especially for those who can't afford it. I have only been too recently involved with the organization to know what changes have to be made The work you do is awesome! I would love more webinars, more online learning and more free online recording of other seminars & learning opportunities. All those that have watched on SEA change, I have found hugely interesting, rewarding and rich of practical ideas to apply. Please see the above comments in 16. Thank you very much for a great platform and the opportunity to exchange findings and concerns. Page 49
33 34
35 36 37 38
39
40 41
42 43 44 45 46
47 48
49
50 51 52 53
Have more materials on M&E of climate mitigation Encourage members to engage in M&E practices in public and private institutions needing this kind of service through "Volunteerism." A roster of SEA Change members who have the capability and time to devote in extending volunteer service to institutions needing the service could be listed and tapped. I am too new to it to know! - Opening it up to other field of M&E - Developing Communities of Practices among members We need to have more discussions on this topic and this we need to receive travel grant so that we can able to attend. 1. Providing more relevant materials for the sub Saharan African region. For us as practitioners in the Africa, we need more specific materials on the platform. 2. More rigorous on-line and in-house training on climate change M&E in Africa especially in the sub Saharan region. I am happy with regular updates and newsletters. I sometimes dont have time to read all of the material so it would be great if the website had materials classified under subject headings (if does not already exist). You are doing it well I think the existence of SEA Change platform is very helpful, I will definitely ask for help on M&E from this platform on the first place. However, I realize that I have not maximized my M&E Knowledge Needs from this platform, mostly because I have so little time to find some materials from this platform. Recently, I have new assistant to take over some of my work, so hopefully, I can use more of SEA Change Platform to work with my M&E work for the next 4 years :) . I will certainly give feedback if I have used it more. Discussion Forum/ Rooms related to problem faced by the members More frequent interactions of SEA Change members such as through webinars and conferences designed for knowledge sharing and capability building. Introducing more and more self-managed on line trainings leading to a step by step certification and webinars. I think the SEA Change platform should have drop box and create good practice documentation among key stakeholders related to M&E and Climate Change. - Sea Change Platforms can be further improved by Seminars/ Workshops to be arranged at some place, where access to all is easy e.g. Thailand/Malaysia. As Many of countries have restriction for other country's citizen, barring them to participate in Seminars/Workshops. Create discussion groups or community of practice on specific M&E applications, e.g. outcome mapping. Commence online thread discussions with a hard statement on an M&E related topic. It's irrelevant whether SEA Change supports the statement - it should be clearly intended to spark and engage constructive professional conversation. I think the idea adopted by IDEAS, CES and now EES, is great --that is...initiating the operation of TWGs or ITIGs that focus on particular areas or topics that need concentration of efforts. These ITGs /TWGs are 'maintained' by diverse members, who themselves are also very busy, but fueled by their interest and dedication to the practice. We have to promote a biannual seminar to bring members together for face-to-face discussions and sharing experience, provide training and training materials, Funding for attending workshop in Asia Pacific Seek all participants to provide 'good-practice' case studies for all to see, in a predetermined short and concise format I think SEA Change provides an excellent platform and I have no specific recommendations for improvement. My main challenge is finding the time to review all of the interesting items that are provided by the platform! Page 50
Annex 5: Climate change M&E knowledge needs panel session results

Part 1 of the knowledge needs group work: Knowledge needs results
Page 51
Page 52
Part 2 of the knowledge needs group work: Knowledge products results
Page 53
Page 54

SEA Change CoP Evaluation Conclave and Members Meeting Report 2013

Uploaded by

Copyright:

Available Formats

SEA Change CoP Evaluation Conclave and Members Meeting Report 2013

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

SEA Change CoP Evaluation Conclave and Members Meeting Report 2013

Uploaded by

Copyright:

Available Formats

SEA Change Evaluation Conclave report

SEA Change Evaluation Conclave report

SEA Change Evaluation Conclave report

SEA Change Evaluation Conclave report

SEA Change Evaluation Conclave report

SEA Change Evaluation Conclave report

SEA Change Evaluation Conclave report

SEA Change Evaluation Conclave report

Summary of the group work process and results

SEA Change Evaluation Conclave report

SEA Change Evaluation Conclave report

First theme: M&E of climate investments

SEA Change Evaluation Conclave report

Second theme: MRV systems in REDD+ programs

SEA Change Evaluation Conclave report

SEA Change Evaluation Conclave report

Section 3. Draft knowledge generation strategy

Technical Working Group(s)

Final Knowledge Generation Strategy

Knowledge Generation Plan

Learning Formats / Knowledge materials

SEA Change Evaluation Conclave report

Section 4. Pre-Conclave Climate Change M&E Knowledge Needs survey

Profile of respondents to the M&E Knowledge Needs survey

National NGO Private / Commercial business Government CSO / CBO

SEA Change Evaluation Conclave report

Knowledge needs responses

50% 40% 30% 20% 10% 0%

SEA Change Evaluation Conclave report

Webinars on useful M&E topics

SEA Change Evaluation Conclave report

Knowledge gaps responses

SEA Change Evaluation Conclave report

Section 5. Climate Change M&E Knowledge Needs panel session

SEA Change Evaluation Conclave report

SEA Change Evaluation Conclave report

SEA Change Evaluation Conclave report

1. Continue doing what SEA Change is good at

SEA Change Evaluation Conclave report

2. Introduce new elements in line with the concept of knowledge generation

3. Include new ideas outside the scope of knowledge generation

SEA Change Evaluation Conclave report

Annex 1: The Evaluation Conclave in Kathmandu, Nepal

Climate change and the environment, a Conclave sub-theme

SEA Change Evaluation Conclave report

Annex 2: List of SEA Change participants

Indonesia Indonesia USA Thailand Thailand

Cambodia India Cambodia Philippines

Project Officer Professor, retired.

World Fish Center SEA Change CoP

SEA Change Evaluation Conclave report

Name Ram Chandra Khanal (Mr.)

Organization Community of Evaluators Nepal / GERES / CDKN

Philippines Philippines Bangladesh Cambodia India Thailand USA Canada Vietnam

Umi Hanik (Ms.)

SEA Change Evaluation Conclave report

Annex 3: Draft SEA Change knowledge generation strategy

Technical Working Group(s)

SEA Change Evaluation Conclave report