SOLSTICE M&E Plan V2
SOLSTICE M&E Plan V2
SOLSTICE M&E Plan V2
SOLSTICE
26 September 2018
i
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Contents
1 Introduction .................................................................................................................. 1
4 Indicators .................................................................................................................... 16
10.1 Storage................................................................................................................... 37
10.3 Privacy................................................................................................................... 38
11 Appendices ............................................................................................................. 39
ii
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Acronyms
SOLSTICE - WIO Sustainable Oceans, Livelihoods and food Security Through Increased Capacity in
Ecosystem research in the Western Indian Ocean, www.solstice-wio.org
iii
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
1 Introduction
SOLSTICE-WIO (www.solstice-wio.org) is a four-year collaborative project funded by the UK Global
Challenges Research Fund (GCRF) RCUK Collective Fund. Launched in October 2017, SOLSTICE-WIO
brings together local knowledge, international research expertise and state-of-the-art technologies
to address challenges facing the Western Indian Ocean (WIO) region in a cost-effective way. The
total budget for SOLSTICE is £8,000,000 covering the four year project duration which ends on 31
December 2021. The SOLSTICE team comprises researchers from the United Kingdom, South Africa,
Tanzania and Kenya with “in-kind” support from the wider Western Indian Ocean community.
• Identifying evidence needs to ensure proposed project outputs are leading to measurable
outcomes and impacts
• Identifying the stakeholders and user groups which need to be engaged to maximise project
impacts
• Strengthening program design: clear indicators of success allow Leadership Team to
optimise project activities and sharpen research questions
• Enabling clear communication: Leadership Team and project PIs are able to communicate
based on shared understanding of the different levels of outputs, outcomes and impacts.
• Maximising collaboration with existing networks and projects with shared vision and
objectives
The M&E Plan was created by the SOLSTICE M&E team (see Roles and Responsibilities) and includes
SOLSTICE Theory of Change, Logical Framework and the Key Performance Indicators. The M&E Plan
is a living and evolving document and is subject to change. All proposed changes to the M&E Plan
must be discussed with the M&E Team and approved by SOLSTICE Leadership Team. All versions of
the M&E Plan including the latest one can be found on the project website www.solstice-wio.org
1
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Theory of Change
M&E Plan v1 Created for 6 months
Log Frame
GCRF review (March
KPIs
2018)
Title SOLSTICE
Duration 31.12.2021
2
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Beneficiaries The primary beneficiaries are policy makers and resource managers tasked with
delivering sustainable management of marine living resources and climate adaptation
options.
- Commercial and artisanal fishers and their families will benefit from better yields and
greater stability of sustainably managed fisheries or from guidance on alternatives where
existing practices are unsustainable.
- Fishers, their families and the general public will benefit from a better understanding of
the marine environment and the services provided by healthy marine ecosystems.
3
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Cost £8,000,000
4
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
SOLSTICE Theory of Change present a conceptual view of the changes that SOLSTICE research and
capacity development must influence if it is to successfully contribute to development impacts for
people and ecosystems. It shows how and why SOLSTICE research is planned to help influencing
thinking, attitudes and behaviours amongst its user communities, so that long term improvements
can be brought about. SOLSTICE Theory of Change also draws attention to assumptions which are
made to allow this conceptual progression. All development initiatives are only as sound as its
assumptions which must be met for the goals to become achievable.
The Theory of Change recognises the SOLSTICE is located within a wider set of international, national
and local initiatives in the area of sustainable use of living marine resources, environmental issues
and impacts of climate change. These all offer opportunities for collaboration and co-production of
outputs.
It is important to remember that SOLSTICE Theory of Change is a conceptual model, not a literal
representation of a linear process. Its main purpose is to provide an analytical framework for
understanding the design of the planned research and the main type of changes the project must
achieve for long-term success.
The schematic of the SOLSTICE Theory of Change is presented in Figure 1. It describes a hierarchy of
changes starting at the lower level with Program Outputs achieving a number of Program Outcomes
leading to short-to-medium term changes. At this level ToC acknowledges that SOLSTICE is not the
only initiative working in the area of living marine resources and cutting edge technologies. Thus
attribution of the Outcomes and, further down the line, the Impacts to SOLSTICE outputs becomes
challenging.
It is important to remember that SOLSTICE is only a four-year project with a substantial component
dedicated to the world-leading primary research in marine environment and ecosystems. The process
of generating and publishing primary research results to act as evidence in the decision-making
process takes time comparable with the project lifespan. Thus the project Leadership Team and GCRF
as the project funding agency need to be prepared to evaluate the success of the project at the level
of the Outcomes rather than the Impacts. Furthermore, GCRF as a funding agency and NOC as a
leading organisation must find the way of carrying on Monitoring and Evaluation efforts beyond the
end of the project with the necessary budgetary implications.
5
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Following requirements of the GCRF “Grow” call, SOLSTICE focuses its effort on producing four key
outputs:
Output1: Strong and self-sustaining scientific transdisciplinary WIO-UK networks producing high
quality intensive body of knowledge on ecosystems dynamics, human dependence on them, their
future trends and human responses
Output 2: Capacity developed in WIO to conduct interdisciplinary ecosystem research that meets
the needs of EAF, policy, industry and markets. Capacity developed in UK to meet the needs of ODA
research.
Output 3: Strong body of evidence produced by the network in each Case Study addressing societal
challenges and providing strategy options co-created with stakeholders and based on ecosystem
approach
The first three outputs should be common to all projects funded by this call, while the fourth one
(technology transfer) is unique to SOLSTICE.
One important assumption will have to prove sound for the Output 1 and 2 to lead to desirable
outcomes, namely that upskilled researchers remain in employment either in their institutions or in
general area of management of the marine environment. If successful, we expect to achieve
Outcome 1: Networks grow, involve newly trained researches and stakeholder participation and
begin to attract new funding. It is important to recognise that attracting new funding is a critical
feature for a sustainable network and long lasting legacy of the project.
SOLSTICE recognises four key groups of players it must influence to realise its impacts: research
(academia and NGOs), private sector industry, policy makers and communities. Through its
engagement activities, SOLSTICE aims to overcome a wide spread resistance to employment of new
technologies and to achieve its Outcome 2: Key players become aware, receptive and show intention
to invest into new technologies and adopt ecosystem approach.
6
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
In providing a strong body of research evidence in each of the case studies and ensuring the optimal
pathways for communication between science and policy, SOLSTICE will achieve its Outcome 3: Case
studies provide strong body of evidence for action with a measurable outcome.
At this level ToC acknowledges that SOLSTICE is not the only initiative working in the area of living
marine resources and cutting edge technologies. Thus attribution of the Outcomes and, further
down the line, the Impacts to SOLSTICE outputs becomes challenging. However SOLSTICE recognises
that synergies with other activities in this area present excellent opportunities for achieving the long
term legacy and sustainability of the results.
It is important to remember that SOLSTICE is only a four-year project with a substantial component
dedicated to the world-leading primary research in marine environment and ecosystems. SOLSTICE
research underpins all four Outputs. The process of generating and publishing primary research
results to act as evidence in the decision-making process takes time comparable with the project
lifespan. Thus the project Leadership Team and GCRF as the project funding agency need to be
prepared to evaluate the success of the project at the level of the Outcomes rather than the
Impacts. Furthermore, GCRF as a funding agency and NOC as a leading organisation must find the
way of carrying on Monitoring and Evaluation efforts beyond the end of the project with the
necessary budgetary implications.
Nonetheless, the project Theory of Change and its Log Frame recognises achievable and measurable
impacts, albeit put them into the longer term category, potentially beyond the end of the project.
These are:
Impact 1: Sustainable networked centres of excellence established in South Africa and Tanzania
Here again, we need to recognise critical assumptions which are a prerequisite to the project
achieving its impact and providing the long term legacy. SOLTICE recognises three such assumptions
at the level of Impacts:
2. Case studies move rapidly along impact pathways and become flagship examples
7
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Each of these assumptions is only partially based on the success of SOLSTICE effort and is under
influence of the factors outside of SOLSTICE control. For instance, success in securing funding
requires both: scientific excellence of the applicant (SOLSTICE objective) but also availability of
private or governmental funds to bid for (mostly outside of SOLSTICE area of influence).
ToC aspires to contribute to further Impacts which are next to impossible to attribute to SOLSTICE or
any other individual program:
It must be recognised, that SOLSTICE ToC is a generic framework designed to cover simultaneously
SOSLTICE general approaches and three very different Case Studies set in three countries with
diverse economic, scientific, social and cultural conditions. As the Case Studies progress, each of
them is expected to develop their own, much more specific Theory of Change.
8
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Figure 1. SOLSTICE-WIO Theory of Change diagram, to be read from the bottom up, following flow
arrows and touch-points.
9
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
3 Logical Framework
Pre-SOLSTICE: NOC/PML leading expertise in marine technologies: Robotics, Modelling, Remote Sensing and Environmental Economics; Existing model
outputs, data sets, software tools and socio-economic frameworks; NMU-NOC Innovation Bridge, Newton Chair and new institutional building at NMU to
Inputs work as a hub for technology transfer.
SOLSTICE: Man-hours and face-to-face interactions (meetings, training, collaboration activities) as detailed in the budget; Local interest and involvement from
9 WIO partners
Stakeholder engagement workshops; collaborative research (“learning by doing”); co-supervision of students; training courses; MOOC; filed work and
Activities
associated training; exchange visits
10
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
11
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
12
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
• Number of MSc & PhD students actively using • 6 months student project reviews
Output 4: Transfer of cost technologies as part of projects Project and publication
saving technologies • N of cross-discipline papers using support of • Review of publication by M&E group reviews are carried out by
underpinning ecosystem technologies • Project report on cost-benefit analysis M&E group
research to overcome limited • Cost-benefit analysis employing technologies employing technologies
investment into research • N of technical demonstrations in Tanzania and
infrastructure in WIO. SA
13
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Outcome 2: Key players become 2. number of requests for briefs, demonstrations, Number of locally relevant
aware, receptive and show cost estimates of employing new technologies; 2. Interviews with key players; surveys applications of
intention to invest into new number of requests for training and capturing changing attitudes to technologies developed in
participation in case studies; number of technologies carried out by M&E team
Outcomes technologies and adopt collaboration with WIO
additional researchers and managers becoming
ecosystem approach involved into case studies and demonstrations experts are growing
Outcome 3: Case studies provide 3. Number of papers, conference presentations, There is sufficient capacity
strong body of evidence for reports presenting analysis of underpinning 3. Publication records, conference papers, in the region to
ecosystem dynamics and recommended desktop reviews
action with a measurable successfully use modelling,
management or policy actions.
outcome remote sensing and
economic decision
frameworks
14
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Impact 1: Sustainable networked 1. Number of new funding applications with the use 1. Regular survey of project participants and Network is successful in
centres of excellence established in of technologies and addressing aspects of food partner institutions carried out by M&E panel leveraging additional
South Africa and Tanzania security (baseline, mid-project, end-project) funding
Impact 2: Institutional uptake of 2. Number of organisations directly and indirectly 2. Interviews with partner institutions and key Key influential stakeholders
new technologies has a using data products generated by technologies (i.e. stakeholders; statistics of data use, expression of endorsed technology-based
measurable impact on measured through data portal metrics, publications, interest and requests for evidence. References approach
Impacts management of living marine from non-academic publications (reports and
funding applications, MOOC participation) and use
resources the results as evidence in decision-making process briefs)
3. Number and type of cases where case study Upskilled researchers have
outcomes were evident in management decisions support of their institutions
Impact 3: Strengthened fisheries (mention in reports and strategy documents). 3&4. Continuous collection of data on the direct and remain in employment
management Number of managers trained in MOOC and MOOC- and indirect use of project outcomes in
light. Number of UK REF impact stories built of case management and policy. Surveys of MOOC
studies. participants.
4. Number and type of interventions where case Altmetrics to capture exposure and reach of
study outcomes were considered in policy relevant project outputs.
Impact 4: Uptake of research-
activities (mention in policy documents and climate 15
based evidence in policy
adaptation plans). Number of REF impact stories
built of case studies.
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
4 Indicators
1. 30
2. 15
3. 10
4. 10
5. 10
Tool Excel
Frequency 6 monthly
Reporting Details will be submitted by institutional leads to S. Alexiou every six months for
the reporting period.
Quality Control This group of indicators is straightforward and easily verifiable as workshops,
supervision and exchange visits are organised centrally. The questionnaire needs to
recognise that peer-reviewed publications have the following stages: submitted;
under revision; resubmitted; accepted; in press; published.
16
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
1. 4
2. 10
3. 300
4. 50
5. 10
6. 10
7. 10; user stories (qualitative, interview-based)
8. 100
Data Collection 1. Reports by PIs delivering training
2. Reports by PIs delivering training
3. Statistics on MOOC attendance (web-based)
4. Report by institutional PIs on use of off-line version for local training
5. Report by institutional PI on state of PhDs and MScs
6. Questionnaires before/after training; reports by PIs
7. Interviews with users (as identified by institutional leads)
8. Web site statistics
Frequency 6 monthly
Responsible S. Alexiou
Reporting Details will be submitted by institutional leads to S. Alexiou every six months
Quality Control Quantitative indicators in this group are straightforward and easily verifiable.
Project members collecting qualitative indicators will attend M&E training including
approaches to collection of qualitative indicators.
17
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Log-frame reference Output 2: Capacity developed in UK to meet the needs of ODA research
1. Number of success stories for upgrading UK technologies to meet ODA
Indicators
requirements (conference abstracts, interviews)
2. Number of training courses attended by UK team members leading to step
change in ODA research capacity
3. Number of young career scientists applications for ODA funding as Co-I or PI
4. Number and amount of ODA-type funding applications
Baseline refers to the 5 year period prior to 1st October 2017
Baseline
1.
2.
3.
4.
Frequency 6 monthly
Responsible S. Alexiou
Reporting Details will be submitted by UK WP leads to S. Alexiou every six months for the
reporting period.
Quality Control Quantitative indicators in this group are straightforward and easily verifiable. All
data are expected to be held either at NOC or at PML and thus easily obtainable.
Project members collecting qualitative indicators will attend M&E training including
approaches to collection of qualitative indicators.
18
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Output 3: Strong body of evidence produced by the network in each Case Study
Log-frame reference
addressing societal challenges
Tool Excel
Frequency 6 monthly
Responsible S. Alexiou
Quality Control Quantitative indicators in this group are straightforward and easily verifiable.
Project members collecting qualitative indicators will attend M&E training including
approaches to collection of qualitative indicators.
19
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Indicators 1. N of MSc & PhD students actively using technologies as part of projects
2. N of cross-discipline papers using support of technologies
3. Cost-benefit analysis employing technologies (indicator: costs saved)
4. N of technical demonstrations in Tanzania and SA
5. N of stakeholders attending technical demonstrations
6. Number of presentations on use of technologies in key partner and
stakeholder institutions delivered by SOLSTICE researchers
Baseline refers to the 5 year period prior to 1st October 2017
Baseline
1.
2.
3.
4.
5.
6.
1. 10
2. 10
3. £1,000,000
4. 2
5. 50
6.30
Tool Excel
Frequency 6 monthly
Responsible S. Alexiou
Quality Control Quantitative indicators in this group are straightforward and easily verifiable.
20
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Indicators 1. Strength, extent, centricity and other key parameters of network as measured
by social network analysis
Selection of network parameters for monitoring and their targets to be identified
after baselining exercise in April 2018
Note that social network analysis is an area of science rather than a monitoring
exercise. Ongoing development of the methodology is expected.
1.
2.
3.
4.
5.
Data Collection A survey with the network profiling indicators will be completed by all project
partners, in-kind contributors and associated PhD students.
The questionnaire will cover: co-authored publications, co-supervised students, co-
organised workshops and co-developed funding applications.
Reporting A project report on the results of the network analysis will be produced by K.
Popova and S. Alexiou
Quality Control The methodology was tested in the framework of international GULLS project. If
required, a professional advice on network analysis will be requested from Dr.
Ingrid van Putten, CSIRO who performed the analysis for GULLS.
21
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Outcome 2: Key players become aware, receptive and show intention to invest
Log-frame reference
into new technologies and adopt ecosystem approach
1. 20
2. 20
3. 50
4. 100
5. 20
6. qualitative
Tool Excel
Frequency 6-monthsly
Responsible S. Alexiou
Quality Control Quantitative indicators in this group are straightforward and easily verifiable.
22
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Outcome 3: Case studies provide strong body of evidence for action with a
Log-frame reference measurable outcome (e.g. projections of the squid dynamics leads to a proposal
of pension fund)
Baseline Baseline refers to the 5 year period prior to 1st October 2017
1.
1. 10
2. 5
Tool Excel
Frequency 6-monthsly
Responsible S. Alexiou
Quality Control Quantitative indicators in this group are straightforward and easily verifiable.
23
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
1. Number and bidding amount of new funding applications with the use of
Indicators
technologies and addressing aspects of food security
2. Number of new PhD/MSc students using technologies
3. Number of developing country researchers that contribute to production of
high quality (journal IF>1) multidisciplinary research publications on the link
between ecosystem dynamics and food security (SOLSTICE network researchers
only, post-SOLSTICE Special Issue only)
1. 10
2. 10
3. 30
Tool Excel
Frequency 6 monthly
Responsible S. Alexiou
Reporting Details will be submitted by UK WP leads to S. Alexiou every six months for the
reporting period.
Quality Control Quantitative indicators in this group are straightforward and easily verifiable. The
questionnaire needs to recognise that peer-reviewed publications have the
following stages: submitted; under revision; resubmitted; accepted; in press;
published.
24
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Indicators
1. Number of organisations directly and indirectly using data products generated
by technologies
2. Number of cases where data products generated by technologies are used as
evidence in decision-making process (each accompanied by a “success story”)
1. 20
2. 10
Tool Excel
Frequency 6 monthly
Responsible S. Alexiou
Reporting Details will be submitted by UK WP leads to S. Alexiou every six months for the
reporting period.
Quality Control Quantitative indicators in this group are straightforward and easily verifiable.
25
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Indicators
1. Number and type of cases where case study outcomes were evident in
management decisions (mention in reports and strategy documents),
accompanied by success stories.
2. Number of managers trained in MOOC and MOOC-light.
3. Number of UK REF impact stories built of case studies.
Tool Excel
Frequency 6 monthly
Responsible S. Alexiou
Reporting Details will be submitted by WP leads to S. Alexiou every six months for the
reporting period.
Quality Control A professional M&E advice/bought-in service might be though at this stage for the
Indicator 1.
26
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Indicators
1. Number and type of interventions where case study outcomes were
considered in policy relevant activities (mention in policy documents and
climate adaptation plans), accompanied by success stories.
Tool Excel
Frequency 6 monthly
Responsible S. Alexiou
Reporting Details will be submitted by WP leads to S. Alexiou every six months for the
reporting period.
Quality Control A professional M&E advice/bought-in service might be though at this stage for the
Indicator 1.
27
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
• Project partnership development and co-design meetings were run in WIO before official
start of the project (four workshops, NOC investment)
• Annual Advisory & Leadership (A&LP) panel meeting (April 2018) dedicated ½ day session (2-
day meeting) to discuss equitable partnerships giving voices to both WIO and UK institutions
• An Equitable Partnership Survey (EPS) (anonymous) was developed in May and then
executed in July 2018 (Full survey can be found in Appendix 2). This will be repeated at 24
and 36 months. It is the main mechanism that SOLSTICE uses to demonstrate equity
between its partners.
• Survey indicators and interim responses from the teams on challenges and benefits of
SOLSTICE partnerships are reviewed by the A&LP on annual basis.
• The survey aims to evaluate and provide evidence for the following characteristics of
equitable partnerships (as defined by GCRF):
Pros and Cons of working in ODA partnership (overall assessment of equitable partnerships)
Although the above characteristics provide a good foundation to build equitable partnerships,
further steps are needed to ensure that equitable partnership actually works.
In ODA-style research, balancing the interests of Northern and Southern researchers, institutions,
disciplines, network members in different career stages, political agendas, and cultural differences is
not easy. And yet an equitable partnership can be defined as a partnership where each member’s
benefits outweighs the challenges such a partnership brings.
Hence key benefits and challenges were identified for WIO (by WIO partners) and for UK (by UK
partners) independent of each other during the first Annual Advisory/Leadership Panel meeting.
These benefits and challenges were then rated by each member of the partnership via the online
EPS (Appendix 2). Survey aims to assess if benefits outweigh challenges both for UK and WIO team
members.
SOLSTICE will continue to use the results from the EPS (questionnaire) as equitable partnership
indicators.
28
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
6 Evidencing Interdisciplinarity
Conducting Interdisciplinary Research (IDR) is a hotly debated topic articulated in a growing
volume of dedicated literature (including Nature). While there is varying consensus on what IDR
is, there is less clarity on (1) what constitutes quality in IDR, (2) how to evaluate and report on
IDR. Of course, once a project produces peer-reviewed publications, interdisciplinarity can be
evaluated more readily via cited bibliography. Until then, project interdisciplinarity can only be
reported using an IDR framework. Although many theoretical frameworks of IDR exist, we have
not come across any which seem appropriate in an ODA research setting.
We therefore developed our own pragmatic framework of interdisciplinarity based on the PIs
experience with similar ODA projects. This utilises recommendations in the ESPA policy brief ―
Interdisciplinary research for development impact: How can funders walk the talk?, and in Tang et
al. (2014) (The relationship between interdisciplinarity and impact, UKESRC report).
OPTIMUM (Objective-Practice-Team-Integration-Management-Uniqueness-Motivation) – A
pragmatic framework for designing, managing and evidencing interdisciplinarity in large projects.
Objectives
Most IDR aims to develop responses to social and economic problems. Narrow objectives are likely to have
solutions. Broader objectives less so with the generated knowledge only partly addressing them.
SOLSTICE mostly belongs to the first category of solution-oriented objectives driven by the case
studies, but they are set within the wider GCRF challenges.
Practice
Method of deploying human resources, i.e. deployment of a large number of part-time senior researchers vs. a
small number of full-time junior researchers. The former maximises number of disciplines in the project,
outputs and ability to expand by attracting additional funding but introduces challenges of coordinating and
integrating the network. The latter simplifies coordination, integration but minimises spread of disciplines and
potential for expansion and maximises the risk of relying on a few key individuals.
SOLSTICE is predominantly the former category, employing 33 senior scientists working part-time
and 7 full-time early career scientists. The larger number of senior scientists has been advantageous
attracting more than 20 MSc/PhD students into the SOLSTICE team.
Team
Refers to composition of team and cognitive distance between disciplines. IDR requires a combination of single
discipline experts as well as T-people (generalists). Warning: assembling single discipline experts
does not guarantee the interdisciplinarity (hence “I” – Integration).
29
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Integration
Integration of single disciplines into a multidisciplinary effort is expensive, time consuming, requires face-to-
face interactions, and occurs at expense of advancement of single disciplines (which are most likely to produce
highest ranking peer-reviewed publications; their advancement is also critical to the project).
SOLSTICE from an interdisciplinary perspective, integrates 3 key outputs: (1) Selection of case studies
(achieved), (2) “Foundation study” to identify gaps and set hypothesis (achieved first draft), and (3)
“Synthesis study” at end of the project to synthesize disciplinary efforts and interdisciplinary
connections and recommendations into a policy output.
We are also monitoring the growing number and strength of emerging interdisciplinary connections
using network profiling method (see Category 6) and graphical Social Network Analysis (M&E v2).
Management
This refers to how strongly, and by what methods, integration is driven. Management of integration is
expensive both in time and budget.
SOLSTICE drives integration strongly for a small set of clearly identified outputs from an early stage of
the project. We use a combination of a small number of actual interdisciplinary workshops with regular
virtual Webex meetings with coordinators identified both in WIO and UK. Connections between
specialised studies (projects) are constantly promoted.
Uniqueness
What are you doing to make it interdisciplinary? Uniqueness of your project is most likely to be its bestselling
feature for securing future funding. It is worth considering it as a special case worth extra effort.
SOLSTICE’s unique feature is marine robotics. An interdisciplinary study linking progress of marine
robotics in the region (ocean technology) to how marine robotics is perceived by local fishermen
communities (social science) has already emerged and is underway.
Motivation
Interdisciplinarity is often perceived to be bad for early career scientists (ECS) and takes more time to produce
output. Also, evidence suggests interdisciplinary papers are cited less than specialised. Single discipline
researchers are strongly motivated by advancement of their own disciplines.
SOLSTICE uses well defined Case Studies with strong societal significance and leveraging of funding in
similar interdisciplinary areas as key motivations
30
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Scientific networks are at the center of all international programs, especially those concentrated
around “wicked problems” such as food security. Well-organised networks harness capacity to solve
shared but complex problems by unifying scarce resources, create Capacity Development
opportunities, and hasten the development of a common knowledge and understanding. A growing
volume of literature is now dedicated to scientific network analysis and theoretical frameworks. And
yet, what constitutes a research network and how a network becomes a network remains obscure in
many cases.
SOLSTICE aspires to developing a research network which is (1) international, (2) interdisciplinary
and (3) focused around the challenge of food security and sustainable use of living marine resources.
Desirable characteristics include research excellence, sustainability (ability to secure future funding)
and potential to address complex problems (research capability). The ultimate ambition is for this to
evolving into a global network.
SOLSTICE is not starting from scratch. Rather it combines several partnerships already in existence to
form an effective, well-focused research network. These include (1) a BELMONT Forum GULLS and
ESPA program GLORIA projects between the UK, South Africa, Madagascar, India, Brazil and
Australia, (2) the well-established and connected Western Indian Ocean Marine Science Association
(WIOMSA), and (3) links between national and inter-governmental institutions. Consequently, core
institutions in the SOLSTICE Food Security UK-WIO wide network are NOC, PML, NMU, WIOMSA,
TAFIRI, KMFRI, IMS and CORDIO.
The term “Network” is highly subjective, and so is a notion of what constitutes “network strength”.
Given the opacities of the issue, we have developed a new, simple but quantitative method that
allows us to objectively assess the building of the SOLSTICE output-based network and its
effectiveness. It also allows comparisons with other networks. In our method, connection strengths
between two researchers (in a network) is identified by the following:
1. Number of co-authored peer-reviewed papers
3. Number of co-supervised PhD and M students (including connections between students and
supervisors)
5. Number of co-applicants for a funding grant (PIs, Co-Is) (includes successful and unsuccessful
applications)
31
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Baselining of the SOLSTICE network has been completed via a SOLSTICE Network Survey (Appendix
2). After baselining, development of the network will be monitored using 6-monthly project KPIs
which contain the five criteria identified above for all the network participants. If the network
expands substantially beyond its original size, we may only be able to track two out of five
parameters (peer-reviewed publications and commissioned reports).
We employ Social Network Analysis (preliminary graphical analysis of baselining is shown below)
which demonstrates the following characteristics of the network: (1) expansion of the original
(baselined) network, (2) strength (efficiency), (3) interdisciplinary and (4) international spread
(including strength of the links between UK and developing countries). Social Network Analysis is
best reported via graphics but ResearchFish doesn’t allow this. Therefore, we simply using a set
network development indicators:
• Number of individuals involved into the network (67)
• Number of countries (4)
• Number of UK-WIO connections (39)
• Degree of interdisciplinarity (see Category 5)
• Averaged strength of WIO-UK (1.9) and interdisciplinary connections (effectiveness) (4.2,
preliminary estimate)
We may adjust the above indicators and/or method of their calculations as our network theory
develops.
As it develops, the SOLSTICE network will be promoted at WIO regional and international
conferences. We intend to publish the network development method, results and practical
recommendations in a peer-reviewed journal (Popova et al in prep).
Preliminary results of the SOLSTICE network profiling using Social Network Analysis
32
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Figure 1. Preliminary results of SOLSTICE network baselining via Social Network Analysis. Dots represent
network members (individual scientists). Lines represent connections between members (measured via co-
authored publications, reports, etc. See full criteria in the text). Members are coloured by disciplines showing
that interdisciplinary connections already exist in SOLSTICE.
33
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Figure 2. Preliminary results of SOLSTICE network baselining via Social Network Analysis. Dots represent
network members (individual scientists). Lines represent connections between members (measured via co-
authored publications, reports, etc. See full criteria in the text). Members are coloured by countries showing
strong clustering within countries and weak international connections.
34
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
Interdisciplinary connections
Number of connections:
Discipline ES OT F SE GS
ES - 5 3 2 12
OT 5 - 3 2 4
F 3 3 - 7 10
SE 2 2 7 - 8
GS 12 4 10 8 -
ES – Earth Sciences
OT – Ocean Technologies
F - Fisheries
SE - Socio-economics, governance
GS – Generalists or T-shaped scientists
35
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
NOC/NMU • 6-monthly review of indicators and their suitability for evidencing project outcomes
and impacts in the area of Science into Policy and Governance
• 6-monthly review of indicators and their suitability for evidencing of the impact in
the SA and TAN network clusters
• Production of the “Success Stories”
PML • 6-monthly review of indicators and their suitability for evidencing project socio-
economic outcomes and impacts in the Case Studies
• Contribution to the baselining, mid term and final M&E reports
SOLSTICE M&E Panel will meet every 6 months (October and April) for reviews of data over the
reporting period and review of ToC, Log-Frame and formulation of the Key Performance Indicators.
M&E Panel will report annually to the Leadership Team and the Advisory Panel.
36
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
9 Data Flow
10 Data Management
10.1 Storage
Data collected during M&E process will be mainly stored in Excel spreadsheet where it can be
analysed. Some data will be collected through automated analytics, ie: number of downloads from
our website etc, which will then be exported to Excel and PDFs for analysis, interpretation and
reporting. All data will be stored on the NOC secure server, behind encryption firewalls. Files are
stored digitally on the NOC shared storage system, which is only accessible by NOC users and each
drive having its own permissions locking them down to specific users. These drives are backed up
nightly and held off site for up to a year.
37
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
10.2 Analysis
Majority of the data will be analysed in Excel. For the social network analysis Matlab tool box will be
used.
The interface between Excel network data base and Matlab will be streamlined to ensure smooth
operation of the analytical tools and clarified by October 2018.
The interface between Excel and data base usage statistics will be clarified by October 2018.
The interface between Excel and model output usage statistics will be clarified by April 2019.
The interface between Excel and MOOC usage statistics will be clarified by October 2020.
10.3 Privacy
SOLSTICE and its participant partners will abide by and comply with the EU’s General Data Protection
Regulation (GDPR), ec.europa.eu/info/law/law-topic/data-protection_en. Any personal data that
may be collected during M&E will processed lawfully, transparently, and for a specific purpose. Once
that purpose is fulfilled and the data is no longer required, it will be deleted.
38
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
11 Appendices
Appendix 1. Capacity development matrices
GCRF anticipate for the Capacity Development to happen at three levels:
Individual: involving the development of researchers and teams via training and scholarships, to design and undertake research, write up and publish
research findings, influence policy makers, etc.
Organisational: developing the capacity of research departments in universities, thinks tanks and so on, to fund, manage and sustain themselves.
Institutional: changing, over time, the 'rules of the game' and addressing the incentive structures, the political and the regulatory context and the resource
base in which research is undertaken and used by policy makers.
We use Research Capacity Assessment Matrices (RCAMs) as the main framework and operational tool to assess CD at the individual, organisational and
(partially) institutional level. The matrices follow the general structure:
39
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
No Category Please indicate the present status of your institution for each of the categories listed. Your responses should be specifically related to:
Use of autonomous marine robotic systems to monitor marine ecosystems and climate change impact
1 2 3 4 5 Simple test question for
Level 3
1 Skills to undertake No skills available to work Some basic skills available, Some data processing skills are Good understanding of Broad skillset available, competent skills If provided with raw binary file,
processing and with data obtained by but core training in data available from one or two oceanographic data processing are available in processing, analysis and multi-variable data from ocean
analysis of data robotic systems. processing and analysis is individuals but further training techniques, analysis and interpretation of new observations with gliders could you produce a
collected from required. in processing and analysis interpretation. Further training or the ability to lead on peer reviewed processed and quality
marine robotic Basic oceanographic data specific to marine robotics is assistance is required to develop publication in international journals. controlled dataset and produce
platforms interpretation skills are not required. Basic oceanographic interpretation into peer reviewed Expertise is being continuously broadened plots demonstrating changing
available. data interpretation skills are publications. and brought up to date via regular training. mixed layer depth and relative
available. Some mentorship is available location of the chlorophyll
from senior colleagues. maximum throughout the
deployment?
2 Infrastructure of No access to marine No access to robotic Able to participate in Access to marine robotic systems Direct access to Marine robotic systems If requested, could you plan the
marine robotic robotic systems or platforms and technical experiments with marine and relevant technical support either by ownership or via a national deployment of an ocean glider
systems and ability technical support. No support but limited access to robotic platforms undertaken from partners. Able to direct the facility, including access to technical to provide 1 month of repeat
to manage datasets access to data collected data collected by partner by partner institutes. Sufficient use of robotic systems to meet support. Data quality protocols follow transects between two fixed
to international from marine robotic institutes. Limited skills or skills available to manage (e.g. own science and technical internationally recognised standards. stations in the West Indian
standards. systems. infrastructure to manage provide quality assurance and objectives. Every effort is made Proven track record of reports/peer- Ocean, and deliver the resulting
observational data collected control) and securely store data to ensure high data quality reviewed literature presenting data multivariable data at a suitable
using robotic platforms. collected. following “best practice” gathered using marine robotic platforms. quality for use in a peer
guidelines. Research into future development of reviewed publication?
robotic systems is planned/ or undertaken.
3 Access to No contacts/collaborations Limited contact or Working in collaboration with Working in collaboration with Taking a leading role in international Can you host a workshop to
international with marine robotics collaboration with marine international groups to address international partners addressing groups that coordinate or develop provide training for local
expertise / networks international community or robotics international scientific or technical questions scientific and technical questions oceanographic research proposals scientists and technologists on
identified partnerships community (scientists and in the marine environment in the marine environment using associated with marine robotics. Provide the deployment and recovery of
with other institutions technologists). Attendance of using robotic systems. Have robotic systems. Able to request lead authorship of peer reviewed papers in ocean gliders using
using marine robotics. relevant personnel at developed suitable contacts assistance and training as international journals regarding operation internationally recognised
international meetings or within the marine robotics required. Representation on and application of marine robotic systems. expertise?
workshops that include international community to international groups associated Providing assistance and training to
emerging users of marine provide training and assistance with marine robotics. Co- international partners.
robotics. in operation and data analysis. authoring papers with
international partners.
4 Recognition and Use of marine robotic There is some appreciation of Use of robotics to conduct Some investment has been made The use of marine robotics to conduct Can you produce a cost-benefit
support at systems for ocean research the benefits of using robotic oceanic research is perceived into facilities and training of ocean research is highly valued by analysis of the use of marine
organisational level is not considered a systems for oceanic research as a valuable activity and personnel to help deliver local management and receives sufficient robotic systems for use in a
valuable undertaking for but it is insufficient for opportunities for local objectives with marine robotics funding to enable outputs to feed into future marine monitoring
policy and business significant investment. development have been to deliver policy needs and policy needs and to support business programme?
development. identified. business development with the developments without the need for partner
40
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
5 Recognition and The use of robotic systems The use of robotic systems to Use of robotic systems appears Use of robotic systems and the A national facility or coordination Can you identify two current
funding by National to investigate marine investigate marine ecosystem in funding calls and national impact of climate change on the programme exists that enables open access nationally funded projects in the
/African/ Global ecosystem function or function or climate change marine programmes. Use of ocean are a regular subject of to marine robotics and technical support last year, where marine robotic
Institutions, policy climate change impacts impacts does appear in robotic systems is identified as funding calls or a required that is actively encouraged to be accessed systems form a critical
making and funding does not appear in funding funding calls and national one of the priority areas in component of funding calls. through targeted funding calls. component of the research
bodies calls and national marine marine programmes, but is capacity development by Strong national and/or programme?
programmes. identified as one of the areas national and international international capacity
where capacity development African bodies. development efforts exist in this
is needed. area.
41
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
No Category Please indicate the present status of your own skills (1-3) and your institution/country (4,5) for each of the categories listed. Your responses should be specifically related to:
Ocean Model Analysis* for marine ecosystems and climate change impact
*
Please note that this area refers only to analysis of existing models rather than new model development
1 2 3 4 5 Simple test question for Level 3
1 Computing Limited or no skills or Some basic skills are available Some data processing skills are Adequate skills in data analysis and Adequate skills in data analysis and If provided with a web link to a
facilities and data available to work but core training in data available but up-to-date and visualisation are available visualisation are available and expertise is 100Mb netcdf file containing 3D
processing skills with model outputs processing and visualisation advanced training in data analysis being continuously brought up to date monthly mean fields of Chl-a, would
is required and visualisation is required you be able to plot surface field, a
crossection and extract annual cycle
in contrasting locations?
2 Access to model No guidance material Some relevant material Relevant material and access skills Relevant material and access skills Models are used effectively to guide Given a 1 month notice, can you get
data relevant to or training on access and/or training is available are available but scientific expertise are available. Models are used policy and blue economy. Quality of the an access to suitable model output
the region and to models available but not enough expertise to is insufficient to interpret and use effectively to guide policy and blue models for the given region is assessed and produce a visualisation of a
analytical skills to implement it the models effectively industry and user feedback is regularly given to surface field of primary production
interpret the data international model developers. in your area and its future
projections under a variety of
climate scenarios?
3 Access to Don’t have any Have some contacts with Work in collaboration with groups Work in close collaboration with Work in close collaboration with groups Do you have an ecosystem
international contacts/collaboratio modellers (physical and/or in other countries doing ecosystem groups in other countries doing in other countries doing model modelling component in your
model development and analysis.
expertise / ns with ocean biological) in other countries model development and analysis. Co-author papers with these development and analysis. Co-author current project even if done by
networks (physical and/or and/or institutions and aware groups and can request papers with these groups and can request another institution?
biological) modellers of their effort and progress. assistance/training in access and assistance/training in access and analysis
analysis of their model output if
in other countries of their model output if required. Write
required.
and/or institutions funding proposals in collaboration with
these groups.
4 Recognition and Modelling is not There is some appreciation of Modelling is perceiveda Modelling
as is valued by Modelling is valued by management, its Can you name three institutional
support at perceived as a potential value of modelling valuable asset and it is seen as management, its use is encouraged use is encouraged and institutional projects or project proposals in the
but funding is insufficient
organisational valuable tool for but it is insufficient for being worth the investment funding is allocated last year where ecosystem
level policy and business investment modelling was a component?
development
42
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
5 Recognition and Ecosystem modelling Ecosystem modelling and Ecosystem modelling and analysis Ecosystem modelling and climate Ecosystem modelling and climate change Can you name three current
funding by and analysis of analysis of climate change of climate change projections for change projections for marine projections are a regular subject of national programs or national
National /African/ climate change projections for marine marine environment occasionally environment appear in funding funding calls or a required component of funding calls issued in the last year
Global Institutions, projections for environment occasionally appear in funding calls. Modelling is calls. Modelling is identified as one funding calls. Strong national and/or where ecosystem modelling was a
policy making and marine environment appear in funding calls and identified as one of the areas of the priority areas in capacity international capacity development component?
funding bodies do not appear in national marine programs. where capacity development is development by national and efforts exist in this area
funding calls and needed international African bodies.
national marine
programs.
43
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
No Category Please indicate the present status of your own skills (1-3) and your institution/country (4,5) for each of the categories listed. Your responses should be specifically related to:
Satellite remote sensing* for marine ecosystems and climate change impact
*
Please consider all the main satellite technologies and data types used for marine applications (ocean colour, SST, altimetry, SAR) when assessing status
1 2 3 4 5 Simple test question for
Level 3
1 Computing facilities and Limited or no skills or available Some basic skills are available Ability to use visualise and Adequate ability to process and Adequate skills in satellite data If provided with NetCDF data of
data processing skills to work with satellite data but core training in the use of process some types of satellite visualise marine data from most analysis and visualisation are SST, chlorophyll-a, SAR, gridded
marine data form different data is available, but up-to-date satellite technologies is available for all the main SSH and SSHA, and along-track
satellite technologies is needed and advanced training in the full available, and the techniques satellite technologies, these are altimetry, would you be able to
range of relevant marine data can be applied in synergy when frequently used in synergy to process and visualise at least 3
types is needed required improve available information, of these data types to reveal
and expertise is being features of interest?
continuously brought up to date
2 Access to satellite data No access to satellite data or Some relevant data is available Data access and analysis skills Relevant material and access Earth observation is used Given a 1 month notice, can you
relevant to the region information on how to analyse not enough expertise to use are available for some satellite skills are available for all the effectively in synergy with other get access to suitable satellite
and analytical skills to and interpret different data analyse and interpret it in technologies, but scientific main satellite technologies for information to guide policy and data to undertake a study of
interpret the different types. relevant contexts expertise is insufficient to marine monitoring. Data from a blue economy. Data from some of the environmental
data types identify, access and interpret all range of sensors are used to different sensors is used in factors influencing ecosystem
the different types of data guide policy and blue industry. synergy, to increase accuracy of variability in your region
relevant to marine ecosystem Algorithms are adapted to information, and feedback from
dynamics improve accuracy of some data users guides development of
products. new or improved algorithms.
3 Access to international Don’t have any Have some contacts with marine Work in collaboration with Work in close collaboration with Work in close collaboration with Do you include satellite data in
expertise / networks contacts/collaborations with remote sensing scientists in groups in other countries doing groups in other countries doing groups in other countries your work, even if the expertise
marine remote sensing. Co-
marine remote sensing other countries and/or who are involved in marine author papers with them and working with satellite data. Co- to analyse some of the required
scientists in other countries institutions and aware of their remote sensing can request assistance/training author papers with these groups data sets used come from
and/or institutions effort and progress. in accessing, analysing and can request assistance/training another group?
interpreting data products
if required. Write funding
where local expertise is lacking
proposals in collaboration with
these groups.
44
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
4 Recognition and support Satellite remote sensing is not There is some appreciation of Remote sensing is perceived as Satellite remote sensing is Remote sensing is valued by Can you name three
at organisational level perceived as a valuable tool for potential value of using remote a valuable tool that is worth the valued by management, its use management, its use is institutional projects or project
is encouraged but funding is
research, or for policy and sensing data, but it is investment needed to access insufficient to fully take encouraged and institutional proposals in the last year where
business development insufficient for investment in and analyse relevant data in the advantage of available data funding is allocated to ensure remote sensing was a
adequate infrastructure and future that relevant facilities and component?
support for skills development expertise are available
5 Recognition and funding Satellite remote sensing for Satellite remote sensing for Satellite remote sensing for Satellite remote sensing for Analysis and interpretation of Can you name three current
by National /African/ marine applications does not marine applications occasionally marine applications occasionally marine applications appear in marine satellite data are regular national programs or national
Global Institutions, appear in funding calls and appear in funding calls and appear in funding calls, and is funding calls. Developing subject of funding calls or a funding calls issued in the last
policy making and national marine programs. national marine programs. identified as one of the areas capacity to take advantage of required component of funding year where the use of marine
funding bodies where capacity development is global data archives is identified calls. Strong national and/or satellite data was a component?
needed as a priority by national and international capacity
international African bodies. development efforts exist
45
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
No Category Please indicate the present status of your own skills (1-3) and your institution/country (4,5) for each of the categories listed. Your responses should be specifically related to:
46
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
4 Recognition and support Regular or routine sampling of There is some appreciation of Regular monitoring is perceived Routine sampling is valued by Routine sampling is valued by Have high level discussions
at organisational level coastal waters is not perceived the benefits of routine as a valuable activity and it is management and government management, it feeds into /reports produced a plan for a
but funding is insufficient
as a valuable undertaking for monitoring but it is insufficient seen as being worth the policy and institutional funding routine sampling strategy for
policy and business for investment investment is allocated national coastal waters?
development
5 Recognition and funding Biogeochemical sampling to Biogeochemical sampling to Biogeochemical sampling to Biogeochemical sampling Biogeochemical sampling and Can you name three current
by National /African/ investigate the impacts of investigate the impacts of investigate the impacts of appears in funding calls. the changing ocean are a regular national programs or national
Global Institutions, climate change or climate change or climate change or Routine sampling is identified as subject of funding calls or a funding calls issued in the last
policy making and anthropogenic activities does anthropogenic activities anthropogenic activities one of the priority areas in required component of funding year where biogeochemical
funding bodies not appear in funding calls and occasionally appear in funding occasionally appears in funding capacity development by calls. Strong national and/or sampling was involved?
national marine programs. calls and national marine calls. Routine sampling is national and international international capacity Did they involve more than
programs. identified as one of the areas African bodies. development efforts exist in this nutrients and chlorophyll
where capacity development is area measurements?
needed
47
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
No Category Please indicate the present status of your institution for each of the categories listed. Your responses should be specifically related to:
Conducting interdisciplinary projects in Ecosystem Approaches to Fisheries and sustainable use of living marine
resources
For a score of 2 and above, evidence (e.g. proposals, publications) and budget estimates must exist where appropriate
1 2 3 4 5 Simple test question for
Level 3
1 Scientific Project Does not have capacity Has sufficient expertise to Has sufficient expertise to Has sufficient expertise to design Has sufficient expertise to design an Over the last three years, have
Design (defined here for scientific project design a single discipline design an interdisciplinary an interdisciplinary project covering interdisciplinary project covering areas you submitted an
as a proposal design project addressing issues project covering three or more areas from physical oceanography from physical oceanography through interdisciplinary funding
developed for an relevant to the marine of the following areas: physical through marine ecosystems to marine ecosystems to fisheries, taking proposal covering three or
internal or external ecosystems and fisheries oceanography, marine fisheries, taking account of climate account of climate change impacts, socio- more of the following areas:
funding body) ecosystems, fisheries, climate change impacts, socio-economics economics and governance to address physical oceanography, marine
change impacts, socio- and governance to address pressing pressing societal issues ecosystems, fisheries, climate
economics, governance societal issues involving latest technologies change impacts, socio-
economics, governance
2 Funding generation Does not have evidence Has evidence of successful Has evidence of successful Has evidence of successful funding Has a capacity and expertise to generate Over the last three years, have
(defined here as a of generating funds for funding generation for a funding generation for a generation for a multi-institutional, funding for a multi-institutional, multi- you won any competitive
successfully funded a project project as described above project as described above multi-national project which national project which includes all of the funding for a proposal as
proposal by an internal includes all of the above above described above?
or external body)
3 Executing a research Has no evidence of Has evidence of successful Has evidence of successful Has evidence of successful Has evidence of successful completion of Over the last three years, have
project project completion completion of a project as completion of a project as completion of a multi-institutional, a multi-institutional, multi-national you successfully completed a
described above described above multi-national project which project which includes all of the above project as described above?
includes all of the above
4 Publishing results of Has no track record of Has a track record of peer- Has a track record of peer- Has a track record of peer- Has a track record of peer-reviewed Do you have peer-reviewed
research projects publications reviewed publications reviewed publications reviewed publications describing publications describing research findings publications describing research
describing research findings describing research findings of research findings of the above of the above findings of a project as
of the above the above described above?
5 Work with Has no experience of Can identified key non- Has an experience of effective Has an experience of involving Has an experience of involving Have you run a stakeholder
stakeholders communicating with academic stakeholders of the communication of the research stakeholders in the design of stakeholders in the design of projects and information event or similar?
48
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
non-academic project outputs results to the non-academic projects obtaining their support as cash or in-kind
stakeholders stakeholders investment
6 Informing Does not conducts Conducts research of Has evidence of producing Has a track record of the use of Has a track record of the use of scientific Have you produced a policy
policy/governance research of potential potential relevance to policy policy-relevant material some scientific results in results of interdisciplinary projects in brief or a stakeholder report?
relevance to policy policy/governance policy/governance
49
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
No Category Please indicate the present status of your institution/country (4,5) for each of the categories listed. Your responses should be specifically related to:
Capacity to carry out international development research addressing the SDGs (UK only)
1 2 3 4 5 Simple test question for
Level 3
1 Skills at individual No Some PIs have experience of Some PIs have experience of A range of PIs has previous experience A range of PIs has previous experience In the last three years, have the
level experience/skill conducting research in ODA conducting research in ODA context. of conducting research and delivering of conducting research, delivering organisation submitted a
s of working in context. No skills of M&E in the Some limited ability of M&E in the projects in ODA context. Early career projects and securing funding in ODA funding proposal for ODA-
ODA projects. organisation. organisation. researchers have opportunities to and SDG context and publishing compliant research with strong
contribute to ODA research and synthesis results in peer reviewed M&E plan, log frame, theory of
understand additional challenges of literature. Early career researchers change including partners from
ODA. have opportunities to contribute to ODA countries?
Organisation has some M&E expertise ODA research and understand
but can get access to M&E skills via additional challenges of ODA.
external contractors. Organisation has M&E experts and
stakeholder engagement experts for
ODA context.
2 Tools, data sets, No capacity or The organisation has a capacity (if Recognition that the effort is needed Organisation is working towards Large data sets are adapted to low Does the organisation have
instruments are recognition of the need arises) for adapting the towards adapting the large data sets adapting the large data sets to low broadband environment. Data examples of quick adaptation of
adapted to ODA the need to large data sets to low broadband to low broadband environment; broadband environment; developing processing and visualisation tools are research tools and methods for
conditions adapt to ODA environment and data processing developing data processing and data processing and visualisation tools developed for unskilled users. work in ODA context?
challenges tools for unskilled users and visualisation tools for unskilled users. for unskilled users. Some work Instruments and analytical capabilities
adapting instruments and analytical Recognition for the need of adapting underway for adapting instruments are adapted for small boats/shore
capabilities for small boats/shore instruments and analytical capabilities and analytical capabilities for small deployment and lack of research
deployment and lack of research or small boats/shore deployment and boats/shore deployment and lack of infrastructure.
infrastructure. lack of research infrastructure. research infrastructure.
3 Access to No North-South PIs have some contacts in ODA PIs have existing and PIs have existing trans-disciplinary
national PIs have strong existing trans- In the last three years, has the
international links exist countries international collaborations and track national and international North- disciplinary national and international organisation been able to
South networks which have some
expertise / record of publications with ODA track record of publishing ODA North-South networks which are respond to an ODA funding call
networks country researchers research in SDG context and some active in publishing ODA research and at a short notice using existing
joint North-South funding applications pursuing new funding opportunities networks with ODA countries
with projects in excess of £1M and ODA experts in UK?
50
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
4 ODA research ODA work is Organisation accepts that ODA ODA context is valued by the ODA context is highly valued by the Organisation has a strong track record Can leading PIs get involved into
capacity at discouraged. work should be a part of the organisation. PIs are recognised for organisation. PIs are recognised for of delivering large ODA projects. ODA ODA research without
their work in ODA/STG context and
organisational research portfolio their work in ODA/STG context. output expectations are adjusted context is highly valued by the damaging/slowing their career
level accordingly. Organisation has ODA organisation. PIs are recognised for progression?
experienced PMOs, coms and other their work in ODA/STG context and
science support functions.
output expectations are adjusted
accordingly. Organisation has ODA
experienced PMOs, comms and other
science support functions.
5 ODA research There are no There are some funding There are funding opportunities for Complexities of research in ODA/SDG Complexities of research in ODA/SDG Does the organisation have any
capacity at funding opportunities for leading western leading western research context are recognised by national and context are clearly recognised by ODA-compliant research funded
institutional opportunities research organisations to become organisations to become involved into international funding bodies. There national and international funding by external bodies?
(national and involved into research with ODA research with ODA countries. Policy- are funding opportunities for leading bodies. Clear evaluation frameworks
international countries. making bodies are aware of ODA western research organisations to of North-South collaborations exist.
funding and projects. become involved into research with There are funding opportunities and
regulatory bodies) ODA countries. Policy-making bodies evaluation incentives for leading
level are aware and receptive of science western research organisations to
into policy process with ODA projects. become involved into research with
ODA countries. There are appropriate
journals, conferences and reports for
dissemination of such research. Policy-
making bodies are aware and
receptive of science into policy
process and actively collaborate with
ODA projects.
51
SOLSTICE, Monitoring & Evaluation Framework 26 September 2018
52