Building a sector-wide approach to measuring impact

Ecstra Foundation has created a framework to enable a common approach to measuring impact across a range of financial wellbeing initiatives.

• SVA Quarterly
• Measurement and evaluation , Philanthropy

In 2020, Ecstra worked with SVA Consulting to develop a framework that would provide a consistent approach to measuring change for the organisations and initiatives that Ecstra partners with. Given its pivotal funding role in the financial capability landscape, Ecstra’s new framework was also an opportunity to improve data quality and capture, and encourage greater sharing of insights about impact and ‘what works’ in the Australian context.

Throughout the project, SVA and Ecstra were keenly aware that developing consistent and robust measurement approaches should not also create an unnecessary administrative burden for grant partners.

The result is Ecstra’s financial wellbeing impact framework and outcomes tool, an accessible and free digital resource specifically designed to measure financial wellbeing outcomes.

SVA’s Lou Campbell sat down with Stephanie Hui, Ecstra’s Head of Policy and Engagement, to reflect on what was achieved through their work together, and Ecstra’s experience of implementing the impact framework.

Lou Campbell: What was the vision behind this impact measurement project?

Stephanie Hui: Given that Ecstra is one of the largest funders of financial wellbeing initiatives nationally, it is imperative that we understand whether and how our work has a positive impact.

“We envisioned an approach to measuring impact that would create value not only for Ecstra and our grant partners, but for the sector, people and the communities we work with.”

Supporting consumers and communities as well as addressing structural barriers to financial capability means our partners and their approaches are incredibly diverse in breadth and scope. Examples of our grants include:

• Developing financial education partnerships with Australian Business and Community Network and through Deakin University
• Community led approaches with ten Community Foundations across Australia supporting local citizens experiencing financial vulnerability
• Women’s economic security initiatives with Global Sisters, Two Good and ACOSS’ Gender and Poverty project, and
• Consumer care grants, to improve support and outcomes for consumers including Redfern Legal Centre’s state-wide Financial Abuse Service, Seniors Rights Service’s elder abuse awareness campaigns and Consumer Action Law Centre’s DemandARefund tool.

We envisioned an approach to measuring impact that would create value not only for Ecstra and our grant partners, but for the sector, people and the communities we work with. Through this project, we wanted to develop deeper engagement with our grant partners and apply learnings to help achieve long-term change.

We had several goals in mind as we designed this new impact framework, including to:

• understand and consistently measure the impact of our work and help our grant partners do the same
• synthesise insights, common themes and challenges and share these with our grant partners and the wider sector
• support grant partners to use these insights to improve program design and delivery, drive innovation, and to scale and increase cross-sector collaboration
• grow the evidence base of ‘what works’ in Australia – a more robust understanding of what works (and what doesn’t) will lead to better outcomes for people and communities, and
• help grant partners demonstrate and amplify their social impact and share their stories.

The willingness of our partners to engage and share gave us confidence that this process could be more than merely transactional. It could add genuine value for partners, Ecstra and the wider community. We strongly believe that organisations can achieve better outcomes if they work together to respond to people’s needs.

Lou Campbell: What was your starting point – what was essential in terms of designing the framework?

Stephanie Hui: Resourcing and capacity constraints are common barriers to impact measurement for many organisations. This is not a unique challenge to Ecstra or the financial capability sector; measuring social change is complex. Through our research and discussions with stakeholders, it was clear there was appetite to improve this.

“We wanted to make impact measurement accessible for more organisations at no cost, without imposing onerous reporting requirements on grant partners.”

Our aim was to assist organisations to implement a simple yet rigorous and consistent approach to impact measurement, specifically designed to measure financial wellbeing outcomes and to help us better understand the impact of our collective efforts. We wanted to make impact measurement accessible for more organisations at no cost, without imposing onerous reporting requirements on grant partners.

We identified a range of financial wellbeing impact measurement challenges to address:

• A lack of consistent terminology, measurement methodology and outcomes framing, making it difficult to undertake comparative analysis and draw out insights across different programs
• How many initiatives measured activity, not outcomes or change
• Resourcing and capacity constraints, where grant partners recognised the importance of measuring outcomes though some had limited capacity or resourcing to do so, and
• A lack of appropriate outcomes planning tools such as logic models, indicators and survey questions.

In addition to addressing these issues, it was vital that the framework be:

• Evidence based, drawing on existing research and internationally recognised survey and indicator banks relating to financial wellbeing
• Easy to use, accessible and meaningful, both internally and for our grant partners, and
• Practial and relevant, making it as simple as possible for grant partners to identify its pertinence and to apply it to their measurement practices.

Lou Campbell: How did your grant partners respond?

Stephanie Hui: Engaging with our grant partners through a consultation and development process was crucial to designing a framework that was relevant and fit-for-purpose. It also helped us identify potential implementation challenges and leverage their knowledge and insights.

Some critical considerations emerged from these discussions:

• The framework must be adaptable for a diverse range of user and program needs. Our grant partners vary in size and the work they undertake and face different impact measurement challenges. Some had existing impact frameworks, others report against multiple funder requirements, and some have limited internal capacity.
• Flexibility is important. However, this should be balanced against maintaining consistency to enhance data quality and comparability. Ecstra’s outcomes and measures are flexible enough to integrate with existing outcomes and measurement approaches that grant partners may already be using.
• Access to support and guidance is valuable, particularly opportunities to discuss the framework with Ecstra at the outset of the of the grant and at different points in the process. Our digital toolkit was created based on this feedback – ensuring grant partners were well supported with guidance materials, templates, example survey questions and a simple guided user process.
• There was overwhelming support for enhancing data collection. Many grant partners have existing data collection capabilities, however some recognised these were focused on output metrics. There was also a strong appetite for the indicator bank developed as part of the framework. Integrating the framework into Ecstra’s grants management system and being clear about the level of detail required helps partners plan and manage data collection requirements.
• A two-way feedback loop should be maintained. Partners provided feedback on all aspects throughout the development process. We will continue this feedback loop through implementation and the grant life cycle and update as needed.

The Australian Business and Community Network (ABCN) brings businesses and schools together to address educational disadvantage. Ecstra has partnered with ABCN for over two years, funding its work to support young people from low-socioeconomic status communities increase their financial capability skills and resilience as they transition from school into adulthood.

According to the ABCN, Ecstra’s new outcomes framework has added value to its work.  “It will enable us to gain a greater understanding of the impact of our initiatives, providing valuable insights to innovate our programs as required.

“The aggregated sector insights will provide an opportunity to understand the impact of our project within a community of Ecstra partners, which will support greater sector collaboration and sustainability of initiatives.

In terms of ACBN’s experience of working with the framework to date they stated, “We have been able to align the framework with our existing evaluation methodology. The framework is very clearly laid out and straight forward to use. We found particularly useful the introductory user guides, templates and examples provided. It supports us to ensure rigour around reporting outcomes in a practical, user-friendly way.”

Lou Campbell: It took time to determine the most suitable outcomes that align with Ecstra’s purpose as well as being pragmatic and useful to grant partners – what approach did you take?

Stephanie Hui: Ecstra had already identified the strategic priorities and outcomes that underpin our work. The challenge was refining the list of outcomes to identify those that best reflected Ecstra’s purpose and the diversity of our work, while also being meaningful to our grant partners.

The process that SVA undertook was invaluable in helping us refine this list, test our thinking, and identify overlapping concepts and gaps in our impact framework. This included:

• considering external and internal factors, and Ecstra’s ability to influence these
• determining causal relationships between outcomes and time horizons for each
• balancing rigor against useability and having enough breadth to reflect the diverse work of Ecstra without compromising the ability to capture meaningful insights and data, and
• ensuring alignment with best practice impact and measurement frameworks where appropriate, enabling comparisons to be made to other organisations and benchmarks.

A vital part of this process was consulting with our grant partners on the refined set of outcomes. We also tested the outcomes against our existing grants for relevance and suitability against a range of financial wellbeing initiatives.

The result was a clear set of outcomes we seek to achieve across our four impact areas, including a subset of priority outcomes. (Figure 1 shows the outcomes grouped into those four impact areas: Strong Communities; Money Lessons for Life; Consumer Care; and Increased Collaboration).

Lou Campbell: Ecstra’s impact framework provides outcomes that grant partners include in their measurement and reporting. Your new online tool enables partners to navigate the menu of outcomes and measures. Walk us through this process.

Stephanie Hui: Our new outcomes tool helps organisations create and implement impact measurement that is specific and relevant to their project. The tool delivers a rigorous and consistent approach to impact measurement, in a practical, user-friendly platform. Grant partners can select and apply outcomes, indicators and measures from our framework in three easy steps. It also contains user-guides, a theory of change template and an example survey.

Step 1: Grant partners select the outcomes that align to their project from the list of Ecstra outcomes.

Step 2: Based on their outcome selection, the outcomes tool generates a list of indicators grouped under each outcome. Grant partners review this list and select the indicators to apply.

Step 3: The tool generates a full list of selected outcomes and indicators together with suggested measurement methods (such as survey questions), and source and data collection frequency. These are then incorporated into Ecstra’s grant management system.

The tool can be used in conjunction with other existing measurement systems with the aim of not creating unnecessary work for our partners. Partners are encouraged to include other outcomes and measures that apply to their project to tell the complete story of the changes their work is seeking to make. The aim is to ensure that grant partners are tracking impact across a minimum number of common outcomes and measures. The Ecstra grants team is also on hand to help throughout the process.

We are gradually implementing the framework and outcomes tool across our multi-year grants and eligible new grants. Initial feedback from our grant partners has been positive, particularly regarding clarity of purpose, the contribution it is making to capacity building within organisations and ease of use. We will continue to track which outcomes and indicators most resonate with our partners’ work, test data collection processes and identify any measurement gaps. The framework and tool will be regularly reviewed and refined as part of this ongoing process.

Lou Campbell: With impact measurement now in place, what changes do you expect to see over the coming years?

Stephanie Hui: This is the most rewarding part of the project, sparking deeper engagement with our grant partners.

We believe there is great potential for the framework to create collective value and help achieve longer term improvements in approaches and outcomes for Ecstra, our partners, and the people and communities we work with.

Early positive signs indicate that more grant partners are focused on outcomes planning and measurement, helping to put the needs of beneficiaries at the centre of design and delivery.

As we capture more measurement information:

• insights and data will be aggregated and analysed and shared widely to help the sector better understand and identify issues, trends, gaps, opportunities and what works in building financial wellbeing
• a key focus will be on facilitating learning by supporting partners to respond to these insights to improve program design and delivery, drive innovation and increase collaboration. Importantly, we’ll also share and leverage these insights to support and influence systems change and policy settings
• greater consistency in capturing data and insights will help inform Ecstra’s future funding decisions, optimise program delivery and improve outcomes. The framework and tool will help Ecstra better assess our organisational contribution and grant making effectiveness.

We hope the framework also helps partners share stories of the difference their work is making to people’s lives, and the lasting positive impact this has.

“For all sectors, a deeper understanding of what doesn’t work is as vital as what does.”

Importantly, the framework and tool will also help us understand what doesn’t work.  For all sectors, a deeper understanding of what doesn’t work is as vital as what does. Innovation and testing new approaches are needed to drive change and address stubborn problems. When an initiative does not achieve the impact nor outcomes desired, this information is still valuable and can inform future initiatives.

We are continuously collecting feedback from our grant partners. The framework and tool will be updated and refined over time as we apply the feedback and insights, and to reflect developments from ongoing research and best practice in Australia and internationally.

To read more about Ecstra’s impact framework visit: Impact measurement — Ecstra Foundation

This work was runnerup in the SIMNA Awards 2021 in the category of Leading Funder in Social Impact Measurement.

Author: Lou Campbell
Contributor: Stephanie Hui, Head of Policy and Engagement, Ecstra Foundation