How can evaluation better recognise Indigenous self-determination?

The right of Indigenous peoples to self-determination has significant implications for evaluating policies and programs that affect Aboriginal and Torres Strait Islander peoples.

• SVA Quarterly
• Measurement and evaluation

This is the second in a series of articles in which we explore the role of data and evidence of what works in delivering improved outcomes for and with Aboriginal and Torres Strait Islander peoples. The first article explored the importance of data sovereignty – Indigenous peoples and organisations having control over how data is defined, collected and used – for improving outcomes.

We share some of the perspectives of Aboriginal and Torres Strait Islander leaders, evaluators and civil servants…

In this article we take a closer look at the evaluation of policies and programs affecting Aboriginal and Torres Strait Islander peoples.

We share some of the perspectives of Aboriginal and Torres Strait Islander leaders, evaluators and civil servants that we’ve worked with and/or interviewed for this article, on how evaluation practice needs to change, consistent with the right of Indigenous self-determination.

The article is structured with reference to five areas for improvement:

1. Evaluation commissioners should invest in more and better evaluations to build a stronger evidence base
2. Evaluation commissioners and evaluators should ensure Aboriginal and Torres Strait Islander people take the lead in defining what ‘successful’ policies and programs look like
3. Evaluators should use genuinely participatory and culturally appropriate methods for gathering data and consulting community (and evaluation commissioners need to pay for these methods)
4. Evaluation commissioners should invest in building the capacity of Aboriginal and Torres Strait Islander organisations in evaluation, learning and improvement – not just funding external evaluations for compliance and funding acquittals
5. Evaluation commissioners and evaluators should ensure evaluations collect information about strengths, opportunities and existing resources.

By evaluation commissioners, we mean those who commission and pay for evaluations, whether government, private funders or service delivery agencies. By evaluators, we mean organisations who conduct evaluations in response to commissioning organisations’ requirements e.g. SVA.

These five areas aren’t comprehensive, but they provide a starting point for those of us working in the field, whether government, service provider or evaluator as we seek to lift the standard of evaluation and – ultimately – improve policies and programs affecting Aboriginal and Torres Strait Islander peoples.

Evaluation’s poor track record in recognising Indigenous self-determination

The UN Declaration on the Rights of Indigenous Peoples (UNDRIP), as cited by the Productivity Commission, defines self-determination as, “the right Indigenous peoples have to ‘freely determine their political status and freely pursue their economic, social and cultural development’ (article 3). In exercising this right they have the right to ‘autonomy or self-government’ (article 4).”¹

While the UNDRIP definition is helpful, it is important to acknowledge that the principle of self-determination has different meanings for different Indigenous peoples. Broadly we understand it to encompass notions of choice, control, respect, freedom and empowerment.

Those who are most invested and most impacted… must be the designers, the architects, the builders and even the evaluators for impact and change.

Evaluation is today used as a key mechanism for making and justifying decisions about what policies are needed, and how money should be spent on programs and services. So, the quality of evaluation, and how evaluations are conducted, play a critical role in upholding the right of Indigenous self-determination.

Romlie Mokak, a Commissioner with the Productivity Commission, says “My learnings over these years are fairly simple, really: that those who are most invested and most impacted must not be assigned to simply be policy render. They must be the designers, the architects, the builders and even the evaluators for impact and change.”²

Unfortunately, evaluation as a practice does not have a good track record of recognising Indigenous self-determination in Australia.

In June this year the Australian National Audit Office (ANAO) reported that the evaluation framework for the Australian Government’s $5bn Indigenous Advancement Strategy (IAS) had no reliable method for measuring long-term outcomes, a full five years after the IAS was established.³

The IAS was set up in 2014 under the Abbott Government. It brought all Commonwealth programs and policy-making for Aboriginal and Torres Strait Islander communities – with the exception of health – into the Department of Prime Minister and Cabinet. While the IAS has an evaluation framework, the ANAO found that the agency was still only in its ‘early stages’ of implementation.

… mainstream approaches to evaluation don’t necessarily have the right approaches or methodologies to capture the work of ACCOs.

The case of the IAS is emblematic of how mainstream institutions and federal and state governments have historically approached the evaluation of Indigenous programs and policy: not doing enough evaluation, and not doing evaluation in a way that is consistent with Aboriginal and Torres Strait Islander peoples’ right to self-determination.

Gabrielle Johnson, Senior Project Officer for Victorian Aboriginal Child Care Agency’s (VACCA’s) Outcomes and Evaluation Team, states that “mainstream approaches to evaluation don’t necessarily have the right approaches or methodologies to capture the work of ACCOs (Aboriginal Community Controlled Organisations).

… there are no data systems in place to capture that work with Aboriginal communities and the knowledge of the Aboriginal organisations.

“ACCOS have been working with their communities for decades – they know their communities and they know what works for them and how to look after them, but there are no data systems in place to capture that work with Aboriginal communities and the knowledge of the Aboriginal organisations.”

Below we share some of what Aboriginal and Torres Strait Islander leaders, evaluators and civil servants are saying is needed to build better evidence in a manner that is consistent with the right of Indigenous self-determination.

1. Evaluation commissioners should invest in more and better evaluations to build a stronger evidence base

A fundamental issue is that there simply aren’t enough quality evaluations to build a documented evidence base on what works. This isn’t just a problem associated with the IAS. The Productivity Commission’s Indigenous Evaluation Strategy issues paper cites a 2016 analysis that estimated that less than 10% of programs for Aboriginal and Torres Strait Islander peoples are evaluated.⁴

To do this with rigour, VACCA is drawing on research evidence, practice wisdom and client preference in designing its programs.

Conducting better evaluations means defining good practice and standards of rigour in evaluations. Evaluation approaches need to identify culturally appropriate ways of working that capture Indigenous knowledge and ways of working.

To do this with rigour, VACCA is drawing on research evidence, practice wisdom and client preference in designing its programs. This includes an emphasis on privileging the often undocumented practice evidence of its last four decades of work with children and families. VACCA’s emerging approach to evaluation incorporates five elements:

• involving VACCA’s service users and community members in defining program logics, outcomes and indicators;
• documenting evaluation plans, including methodology;
• taking a mixed-methods approach where appropriate and relevant that includes using pre and post outcomes measurement tools;
• following an ethics process;
• and identifying negative and unintended consequences through the evaluation.

Another key consideration for evaluation commissioners is to co-design evaluation methodologies and approaches with ACCOs. This should include what data will be collected, how to minimise the burden of data collection, and how to maximise the usefulness of the evaluation for the organisation.

2. Evaluation commissioners and evaluators should ensure Aboriginal and Torres Strait Islander people take the lead in defining what ‘successful’ policies and programs look like

Because evaluation is about values, it matters exactly who defines what is ‘valuable’, ‘good’ or ‘successful’. And if evaluation is going to support better outcomes for Aboriginal and Torres Strait Islander peoples, then they need to take the lead role in defining these terms.

Many Aboriginal and Torres Strait Islander organisations and communities are already defining what successful policies and programs look like in their eyes. But they need governments and other funders to support this work.

… ACCOs could determine what is working and what’s not and communicate that impact before governments make policy decisions for us and without us.

As Karly Warner, now CEO of Aboriginal Legal Service (NSW/ACT) Limited, noted in our previous article on data sovereignty: “If we had a greater role in defining, collecting, analysing and controlling our own data with true data sovereignty, then ACCOs could determine what is working and what’s not and communicate that impact before governments make policy decisions for us and without us.”

At the moment, many Aboriginal and Torres Strait Islander organisations and service providers have to navigate a tension between what community members and staff believe is important, and what funding bodies want to see.

… and sometimes there is pressure to try and achieve things that may not necessarily be as important as other outcomes.

Melanie Ashman, Evidence and Research Officer at VACCA, gives an example: “Staff and community members in a program logic workshop often know what outcomes they would like to see for a program, and sometimes there is pressure to try and achieve things that may not necessarily be as important as other outcomes.”

Governments and private funders make it difficult for Aboriginal and Torres Strait Islander organisations by not giving them the space to define success in their own terms.

3. Evaluators should use genuinely participatory and culturally appropriate methods for gathering data and consulting community (and evaluation commissioners need to pay for these methods)

It also matters how evaluators go about collecting data from Aboriginal and Torres Strait Islander communities. The data collection methods used – how people are consulted, how much time is spent visiting communities and when, the languages and methods used and who is consulted – are all factors that influence the quality of data, and the potential for evaluation to support community capacity and self-determination.

… conventional approaches to evaluation… just don’t work, particularly in remote communities.

Jason Elsegood is Director of Cross Cultural Consultants (CCC), a consulting, community engagement and training firm based in Darwin. He says that conventional approaches to evaluation – with fly-in-fly-out consultants, visiting on tight timeframes, and using highly structured data collection tools that focus on answering specific questions – just don’t work, particularly in remote communities.

“There’s a constant flow of outsiders coming to community to capture information of all sorts… if you’re looking at people who don’t know the community, they don’t know who to speak to, where to speak with people to get engagement, what country is appropriate or not, what community events might affect participation, or how to engage in language,” says Elsegood.

… if you’re engaging community in English, they can’t engage from a position of power in their own language, so already we’re setting up the process to be dominated from that one cultural point of view.

“Up here in the Northern Territory we’ve got over 50 Aboriginal languages, and there are over 100 dialects within that. So, if you’re engaging community in English, they can’t engage from a position of power in their own language, so already we’re setting up the process to be dominated from that one cultural point of view.”

CCC’s community engagement method aims to get around these issues by selecting community members, called Aboriginal Community Engagement Specialists (ACESs), to conduct local consultations. The ACESs are briefed on the evaluation purpose, scope and key questions, provided training and supported throughout consultations. Rather than conducting formal surveys, ACES collect data through informal discussions through families, kinship networks, and community gatherings.

… because they don’t see the change attached. They just think, ‘what was it all about?’ and there’s a lot of fatigue.

Another central part of the ACES model is feeding back to the community. “We call it closing the loop,” says Elsegood. “It’s really important that the ACES go back and discuss with the community what was heard, and what the next steps will be, or where their information has gone.⁵

“Because otherwise it can impact on the ACES’ relationship in the community, and also the community’s motivation because they don’t see the change attached. They just think, ‘what was it all about?’ and there’s a lot of fatigue.”

You can’t get direct and critical answers to evaluation questions unless people have a relationship with you.

While participatory methods like this are no doubt more expensive, Elsegood believes that you have to invest these resources for a rigorous evaluation.

“You can’t get direct and critical answers to evaluation questions unless people have a relationship with you. If you try to plan for two to three weeks to build those relationships in community, it’s quite often seen as non-productive or too expensive, when in reality it’s the critical part,” he says.

Unless consultation is properly carried out, the information collected will not be useful and evaluation will not produce valuable learnings for policy-makers and funders.

4. Evaluation commissioners should invest in building the capacity of Aboriginal and Torres Strait Islander organisations in evaluation, learning and improvement – not just funding external evaluations for compliance and funding acquittals

Evaluation, data and evidence should help us learn about what works and what we can do better, so we can improve policy and programs, not just prove whether they are working or not.

This reflection and learning could be a shared enterprise between Aboriginal and Torres Strait Islander community-controlled organisations and the government agencies that support them – where both parties sit down to look at data to understand what’s happening, why and what can be done. Currently these opportunities are being missed.

We don’t know there’s a problem until the Department says there’s a problem.

This can come down to how data collection databases are designed and used. Karly Warner from Aboriginal Legal Service (NSW/ACT) Limited, described: “The database that we use generates a file report of raw data and we are obliged to send this to the Attorney-General’s Department. The data is then fed into another ‘data system’ that reads and arranges our data through a process designed and controlled by the department. We don’t know there’s a problem until the Department says there’s a problem.”

Harnessing the potential for evaluation and data to enable improvement rests on funding bodies investing in Aboriginal and Torres Strait Islander organisations to build their capacity and to develop internal measurement and evaluation systems, building trusting and respectful relationships, and working with them collaboratively to interpret results and identify solutions.

5. Evaluation commissioners and evaluators should ensure evaluations collect information about strengths, opportunities and existing resources

Evaluation also has greater potential to build knowledge about solutions if it shifts to focus on existing strengths, opportunities and resources that policies and programs can build on.

Warner explains, “we have a lot of qualitative data about our clients’ unique strengths and abilities through our service delivery. If we were properly resourced to collect, analyse and interpretthis data we would have a better evidence base of how family and culture is a critical protective factor to reducing contact with the justice system.”

We do need to consider just a fundamental shift in the way we do it, because ultimately if we don’t, we’ll be in the same position in 20 or 30 years.

Elsegood notes that CCC’s community engagement tries to focus on “strengths, needs, opportunities, and challenges, because it’s what we need.”

“We’re thinking about how we can make change, what can we do, as opposed to looking at the risks and threats to doing something differently. That’s probably a challenge for bureaucracy. We do need to consider just a fundamental shift in the way we do it, because ultimately if we don’t, we’ll be in the same position in 20 or 30 years,” says Elsegood.

Hope for the future

Collectively these changes mean fundamentally overhauling the historical approach to evaluating Aboriginal and Torres Strait Islander policies and programs.

And there are many valuable lessons in the above that could be applied to the evaluation of policies and programs for non-Indigenous people, too: involving end users and community members in designing and evaluating policies and programs; shifting from a strict focus on accountability, to learning and improvement; focusing on peoples’ strengths and opportunities, rather than seeing them as a problem (see for example SVA’s Mental Health Perspective Paper). We are seeing these trends through emerging place-based initiatives and an increased emphasis on the use of co-design in policy development and program design.

There are good reasons to be optimistic that Australian governments will improve the way that they evaluate policies and programs for Aboriginal and Torres Strait Islander peoples.

… the notion that we just bolt on evaluation at the back end, or close to the back end of a program, I’m hoping that those days might be passed.

First, the Productivity Commission under the leadership of Romlie Mokak has begun developing a new whole-of-government evaluation strategy for the Australian Public Service and any programs or policies affecting Aboriginal and Torres Strait Islander Communities.⁶

Mokak, who was previously CEO of the Lowitja Institute, has expressed an intent to improve the Australian Government’s track record in evaluating Indigenous policies and programs, saying, “the notion that we just bolt on evaluation at the back end, or close to the back end of a program, I’m hoping that those days might be passed.”⁷

He has also reinforced a crucial point made by several of the practitioners above – that the views and knowledge of Aboriginal and Torres Strait Islander people and organisations should be incorporated into policy development and program design as well as evaluation.⁸

Second, there are signs that the Morrison Government aims to improve on the IAS’s poor evaluation track record. It dissolved the IAS on 1 July 2019 and began transferring its functions to a new agency, the National Indigenous Australians Agency (NIAA) under the leadership of Ken Wyatt MP.⁹ The NIAA has been given orders to improve evaluation of Aboriginal and Torres Strait Islander policies and programs.¹⁰

Ian Anderson, Deputy CEO of the NIAA, identified two broad agendas for government evaluation in this area: ‘repositioning the relationship between Indigenous Australians and the public policy process’, and the ‘professionalisation of public policy’ – building the skills and mindsets within the public service to enable evaluation.¹¹

… they provide a starting point for reflection and action for all of us working in this area.

At SVA we’ll be watching closely how the Productivity Commission and NIAA follow through on these renewed commitments to improve the evaluation of Australian Government spending in Indigenous affairs.

It’s critical that evaluation commissioners and evaluators are thinking about the implications of self-determination for the design, delivery and evaluation of Aboriginal and Torres Strait Islander policies and programs. The five areas for improvement identified above are not comprehensive, but they provide a starting point for reflection and action for all of us working in this area.

We’re certainly at the beginning of our journey at SVA.

Author: Jonathan Finighan
Contributor: Brendan Ferguson