Wavelength: Evaluation in an SDGs Era

Karen Persad CAPAM Knowledge Exchange AdvisorKaren Persad, CAPAM Knowledge Exchange Advisor

‘Wavelength’ is part of an ongoing series dedicated towards CAPAM contributors exploring ideas about issues and opportunities associated with public administration.

According to the Sustainable Development Goals (SDGs) Fund, monitoring and evaluation (M&E) as a function and practice “identifies trends, measures changes, and captures knowledge to improve programmes’ performance and increase transparency”.1  For those countries that are leveraging Agenda 2030 as an overarching framework in managing for national development results and carrying out SDGs mainstreaming in national public policy, M&E as a discipline is instrumental for tracking, measuring and assessing results related to each of the 17 SDGs. The ethos of “No one left behind” is a compelling statement that echoes principles of interdependence, solidarity, universality, accountability and inclusivity, which could be bolstered by an M&E system. 

This article will highlight elements that all Commonwealth countries should consider and/or advance throughout national, budget and policy planning and advising; budget decision-making, performance reviews and reporting processes. By no means are the following points exhaustive, but they do serve as primary, practical considerations for Commonwealth governments: indicator setting, data responsibilities and responsiveness, technological preparedness, capacity and capabilities, culture, integrated governance, and management agility. 

The targets outlined for the 17 SDGs are intended to guide the different levels of government in their implementation and planning processes. Progress towards the vision of Agenda 2030 through the 169 targets will be measured through a set of globally-harmonised indicators for monitoring performance. However, tracking and assessing progress through the sheer number of indicators, 229 proposed, within an SDGs framework for governments create challenges, given their complexity and diversity, such as geography, governance, environment, socio-economy, technological fit, and more. Therefore, the expected “local level” rollout of these SDGs will require the resources and capacity to develop a tailored subset of indicators to reflect the nuanced local and sub-national levels in different countries. The data emerging for each of those indicators will also need to be disaggregated at the different levels of government, different sectors, different departments and agencies, and so forth. Without conducting these steps, accountability relationships will be poorly defined, if at all.

In addition, SDGs progress and results will be determined through data2 in different forms: qualitative, quantitative, diagrammatic, storytelling and other. A successful outcome of the Millennium Development Goals (MDGs) is that more data is now available in countries than the pre-MDGs era. Many circles are touting technology and innovation as the solution to meeting the SDGs, acknowledging the data-intensive nature of the SDGs framework. However, this could demand an unaffordable investment for developing and transitional economy governments in the short- to medium-term, and a lot of time for agencies/departments to mature with these new systems. Institutionally, changes would be required to retrofit traditional government structures to a less manual, more digital government model, triggering additional requirements to re-orient the thinking, processes, capacities and capabilities. Further, an accurate interpretation and analysis of the data in their various formats must be conducted by and for all levels of government. If data is not transformed into timely, reliable, valid, credible and insightful information that can be processed and digested by multiple stakeholders and citizens, then timely policy decisions (national or sectoral planning, budgetary, programme management, inter alia) and results will be compromised, adversely affecting government transparency. This is not helpful in an age where dwindling public trust and citizen engagement pose real challenges in the government environment. 

The contextual, capacity, regulatory, legislative, and financial resource challenges bring into question the practicality of collecting and reporting on the large number of SDG indicators. Where there is a deficit of technological advancement for national governments, be it platforms for communications or databases for data collection and storage, agility of less technologically advanced governments to respond innovatively to Agenda 2030 will be strained. Some public institutions and statistical offices do not have updated software or sophisticated systems to process high-volume data. As well, the private sector as a key partner may have the technical expertise and resources available to assist the public sector with innovation. However, there are risks that such enterprises may not be as well-developed or know-how/ experience with coordinating public-private partnership arrangements is inadequate for both parties.  

The debate on the number of global indicators that should be set for monitoring by all countries may strongly consider a platform for designing sub-national-level indicators that are linked to the universal 2030 Agenda indicators. In that way, data collection and reporting mechanisms will be “home grown” – developed in meaningful, participatory, practical and context-specific ways. This approach will ideally complement any multi-purpose, interaction and general indicators that can inform progress on more than one goal and target concurrently in those sectors with intersecting, interrelated outcomes.

On the Agenda 2030 journey, public officers and officials are not only the drivers of government operations but specifically are the implementers of policies backing national and international strategies.  Under the SDGs framework, public institutions, their human resources and citizens are assumed to be at mature or even peak capacity and capabilities, particularly in times of natural disasters or social crises. People and systems are expected to know what data to collect, clean, process, analyse and eventually communicate via reporting mechanisms, and how. In the African Commonwealth as well as Asia Pacific contexts, language must be understandable to all subsets of the society. This will require monitoring and evaluation tools and guidelines to be tailored according to a highly varied linguistic demographic. Equally with public officials, citizens should constitute an audience group in capacity-building initiatives to develop awareness and knowledge of data collection and use. This is crucial in the security and environment sectors where installations like early warning systems and sensors produce readings for immediate response and action; a matter of saving lives and ensuring public security. 

The International Institute for Environment and Development (IIED) highlights four challenges that can reasonably present opportunities for governments to equip themselves and transition to more results-oriented entities in an SDGs era. Please refer to the following graphic:

The four key challenges to National Evaluation Capacity
The interlinked relationship between capacity, culture, M&E culture and partnership is demonstrated through the World Bank Group’s CLEAR Initiative, which establishes centres of excellence globally. The Wits School of Governance (University of Witwatersrand) in partnership with the Department of Planning, Monitoring and Evaluation (DPME) hosts the centre, located in South Africa. This long-term partnership has produced a national evaluation policy, ensures delivery of in-service training courses by local academics in context, and enables regional knowledge sharing through South Africa’s own lessons.  

Within a rapidly evolving environment of opportunities and challenges for governments, culture is the notional mindset underlying the operations, data, technology, capacities, strategies and organisational direction of any given public sector. Globalisation is placing some pressure on businesses and the vast network of government agencies and departments to deliver “smarter” services. When these demands require changes in the name of development or equality, this creates friction. Friction emerges when new ways of working and living challenge or augment, for better or worse, the societal values, perceptions, assumptions, beliefs and other ethnographic factors embedded in social and political institutions.  Hence, crafting national evaluation policies (NEPs) by governments should articulate the importance of culture in regard to competencies, awareness, responsiveness and sensitivities in designing4 and implementing evaluations.5  

As well, government departments and agencies that do not have a history of data- and results-driven6 systems and mindsets will need to develop a government-wide M&E culture. This could be operationalised rather quickly using the simple but costly solution of a shared infrastructure - restructuring silos to whole-of-government platforms that enable shared resources, shared systems and shared services. Such an investment adds value in many ways given its interactive nature.  Joint platforms are poised to not only create a seamless response of information and services to businesses and citizens but also can simplify the processes followed by public officials to collect, manage and report on data for more manual, routine tasks. The resulting time savings will allow increased focus on the strategic outcomes of national and international frameworks, including Agenda 2030. In this regard, the World Bank Group’s CLEAR Initiative in Latin America is commendable as it partners with governments (federal and sub-national), academia and civil society to host an annual “national evaluation week” that aims to increase awareness and promote knowledge sharing about the importance of evaluation.

Governments have been delivering services, protecting society and enabling economic prosperity through the traditional silo model. However, to effectively respond to the vision of Agenda 2030 for sustainable development, a more anticipatory and integrated model is critical for the public sector. Ideally, for efficient generation of data on the SDG indicators from various sectors, different governments will need to put in place formal agreements and tools for bottom-up (local to national to international) and top-down (international to national to local) reporting. In this way, duplicating efforts, stretching already scarce resources and double counting can be avoided. Analytically, the data collected can point to the SDGs that are correlated and pinpoint co-benefits more clearly, allowing governments to craft stronger and holistic policies and solutions for “wicked problems” cross-cutting or intersecting certain goals. 

The ICLEI Global7 has developed a carbon climate registry (cCR)8 that enables worldwide reporting by local and subnational governments through vertical integration of greenhouse gas inventories and other climate-related data. This platform is able to demonstrate more explicitly the actions being taken by local and subnational actors and the extent to which they are impacting more macro-level priorities – national as well as SDGs targets. A digital approach for developing Commonwealth countries means departing from standardisation to adopting simplification through innovation. This is not an easy task and therefore sets some countries at a disadvantage within the entire SDGs framework for effectively and efficiently tracking and assessing progress.

Finally but not exclusively, governments in industrialised, transitional and developing Commonwealth countries can better respond to Agenda 2030 if a paradigm shift is made to an agile results-based management approach. Agility and adaptation of the public service means that the design and delivery modes are non-linear to accommodate mid-course corrections, learning and adaptation – key elements of managing for better results and performance. This lends to the notion that failure can be constructive if public sector stakeholders have the opportunity to learn what works and what does not work in a proactive and practical way. An incremental and iterative governance style can allow for testing and piloting emergent national (and sub-national) action plans (sectoral, national, subnational) and strategies that directly or indirectly mainstream the SDGs. Further, this management approach is likely to influence a relatively greater level of interaction and engagement of the workforce. This latter byproduct of agility further promotes the reality that M&E is an integrated function requiring multi-departmental/agency collaboration. This collaboration breaks down the traditional silo model without having to invest in major reforms or re-platform existing processes and transactions to leverage crucial data and information from wide networks of data sources. 

1    Source: http://www.sdgfund.org/monitoring-and-evaluation

   The public sector information ecosystem can include sectoral data from sensors, geospatial data, social media data, statistical data, big data and web-generated data.

3    Source: Kassem El-Saddik, Dorothy Lucks, Stefano D’Errico, Thomas Schwandt and Zenda Ofir. IIED Briefing. (December, 2016).

4    Hood, S et al. (2015) Continuing the journey to reposition culture and cultural context in evaluation theory and practice. Information Age Publishing, Charlotte, NC, USA.

5    CJPE (2016) Special Edition of the Canadian Journal of Program Evaluation 30(3) doi:10.3138/cjpe.30.3.02.

6    Using data to set goals, measure performance, inform policy decisions and demonstrate transparency.

7    ICLEI originally stood for “International Council for Local Environmental Initiatives”. In 2003, the organisation dropped the full phrase and became “ICLEI-Local Governments for Sustainability” to reflect a broader focus on sustainability, not just environmental initiatives.

8    Source: http://carbonn.org