Evaluation

Addressing evaluation issues specific to women

Women’s services are generally against the grain of mainstream services, and decision-makers often regard them as not as important as generic services or services where anyone can attend. In order to make the case for our specialist women’s groups and women-only services, we need to have a strong evidence base that says why women like them, what they do well and what impact they have on women.

The topic of evaluation needs to be introduced gently with care and compassion, transparently and purposefully. It is important to explain to your women’s peer support group what you are doing, how you will do it and why it matters. Clarity about what will and won’t happen to their information is really important and what the boundaries are around confidentiality. The beauty of peer support is the informal space: without some of the structures of more formal service provision, what can feel typical in a direct service feels clunky in peer support space.

It’s useful to think about what data you already collect and how that can be used without putting additional burden on women. For example, can you use registers/sign in sheets to produce detail about numbers of women using the groups, numbers of regular attenders, or community reach through an analysis of protected characteristics? Make use of the experience of your facilitators by asking them to reflect on the impact that they feel the group has had on them personally, and use these reflections to produce anonymous or composite case studies.

When you do need to talk to participants about what impact the group has on them and why they attend, it’s important to approach the topic sensitively. Try to underline why it is important to collect this information for evaluation purposes: that is, what is the benefit for the participants of doing the evaluation (e.g. increased funding, sharing their stories, measuring the impact of the program, ensuring that the service is safe and accountable to the group). If you have examples from other projects, you can show how you will present data, who it will go to and what purpose it will have.

Women, especially those who have lived with coercive, controlling partners or who have experience of being monitored and stalked, may have a deep fear of being judged and watched. They may worry about the impact on their migration status, their children or their benefit entitlements. As evaluators, you need to be able to answer questions on what will be done with the information and who will be able to access it.

Framing evaluation positively as a key part of group development and keeping the evaluation ‘external’ is key. Externalising is a technique from narrative therapy which locates the issue being discussed outside of individuals; it’s mainly a linguistic technique that gives distance and safety so that women know they are not being judged. The purpose of evaluation is not to know how the women performed in the group but to justify the group’s purpose and impact. If you can lift it into this realm, it can make talking about experiences safer for participants.

Give plenty of reassurance about confidentiality and explain where their information will go and how it will be stored. Tell women that they don’t have to use their real name. Try to give different ways for women to share their evaluation feedback: short forms filled out with no names and placed in a box or one-to-one conversations with facilitators if they aren’t confident with literacy. Post-it notes as part of a group discussion can work well as evaluation tools also.

Negative and constructive feedback is just as useful for service development as positive feedback. Women in a safe and positive environment can be encouraged to see that their opinion is valid, and to view complaints or constructive feedback as a tool of empowerment and finding an authentic voice.

How you create a space where improvements and concerns can be shared and how you receive that feedback will demonstrate the levels of safety in the group. Be aware that that women can find sharing negative feedback in groups difficult, and the same is true in one-on-one conversations because of women not speaking openly anyway, or not wishing to complain. Reframing this to what worked well and what didn’t work well as a way of getting insight to how projects can be improved might help you to get more of this feedback through anonymous forms.

Evaluation work can be a temperature check on how well trust and safety is being created in the group. For this reason, it is worth doing it regularly, rather than at the end of a funding cycle or when a monitoring report is due. A safe space is one where women can say what they really feel and know they will not be judged for sharing their truth. Having regular spaces where facilitators listen, and make improvements and changes to the service based on what women tell them, is an important part of trust- building.

Trauma-informed environments encourage reflection and working at a pace that is appropriate for the women creating the space. Evaluation can be a key part of understanding group dynamics, understanding how safe the group feels and giving group members and facilitators a way to demonstrate their trusting and empowering respect to the women in the group.

The resources above in the leadership section on reflection and experimentation are key tools in peer evaluation and self-evaluation. Reflection sheets, gratitude logs and journal prompts can all be used to develop self-evaluation skills. Women can be encouraged to reflect individually after each group, in a way that suits them, and you can build personal reflection experiences into your group exercises semi-regularly – maybe every three to four months. Women can be encouraged to reflect in creative ways or verbally, which ever way suits the participants best.

We’ve given some journal prompts here that you can tailor to the nature of your group. Most are informed by solution-focused approaches to coaching questions. (It might be useful for facilitators to do some background reading on solution-focused approaches to coaching to help with evaluation discussions.)

What had the biggest impact on you today?
Did anything make you think or feel differently?
If I ran into you on the street in a couple of months’ time and asked you to tell me about being in the group, what would you tell me about it?
If you could change three things about the group, what would they be?
How do you think the group could be made more inclusive?

Always be transparent about how reflection tools will be used and shared, and it’s fine to ask women to do them in a variety of ways in order to build their mental well-being and skills. Evaluation is an essential life skill: it helps us to develop positive inner voices, confidence and an authentic voice. It is important that evaluation isn’t seen as an add-on to the project but as an intrinsic part of developing trust, rapport and safety in the group.

Women Side by Side evaluation learnings and addressing barriers

According to the McPin Evaluation report on the Women Side by Side programme, some projects engaged very well with the evaluation, and developed new skills, but many projects did not build their capacity to self-evaluate which is crucial for sustainability. It’s important that the attitudes of the leaders and facilitators of the group model curiosity about impact and safety. Evaluation is a key tool in keeping group experiences safe and transparent and a key way to demonstrate your commitment to partnership: putting women in control (ownership) of the process so that it is something we are doing with them, not to them.

Here we list the most common barriers and suggest ways to overcome them through trauma-informed approaches.

Fear is a common emotion when we feel something might be out of control, many women, especially those who have had negative experiences with statutory services, fear telling their stories and the impact this might have.

Demonstrate your trustworthiness by showing examples, reversing the power dynamics, having women in the group carrying out interviews with each other, making anonymised feedback an option and using externalising language so that they don’t feel personally under scrutiny.

If you think about the work of the group as reconnecting women to their voice and their power, then evaluation is a key mechanism for achieving the group’s aims.

Divide tasks into small, meaningful projects that can be allocated across the group to help share the work. The Women Side by Side project noted that where there were ‘project coordinators’ to support completing evaluation, and colleagues who were regularly involved in the monitoring of grant forms, they produced better reports than organisations with no help with evaluation.

Try to have three or four women in the group who are aware of what funders are requiring of the group so that they can share responsibility for organising evaluation, and create activities that will help you meet your outcomes.

A formal evaluation questionnaire that is long and very verbal can be off-putting for those with low literacy and neurodiversity. A very formal-looking form can be triggering for women with poor experiences of statutory services.

Can you get the data you need in creative and inclusive ways, during the session so that women have the support they need to feed back to you and get support if they find the process difficult?

Women with ADHD, learning difficulties or communication differences often need a shorter and more accessible alternative to long forms, e.g. visual record and in community languages. Everyone should always be given choice and control over how they give their feedback. Make good use of translators and think about what community/first language you think is most appropriate for your group members.

Although we advocate for regular inclusive and ongoing evaluation processes that are part of the regular work of the group, we have also seen examples of good practice where groups have gone outside of their normal group setting so that evaluation can be completed together in a different space.

A coffee shop or community space, for example, could provide an element of distance from the group to encourage reflection and evaluation. If you do experiment with trying evaluation in different settings, emotional safety and confidentiality need to be taken into account.

McPin peer researchers were able to attend many of the peer support groups and explain the process of evaluation to the women. The benefits from having a peer researcher attend were really positive to many groups, who felt that she became part of the peer support group and looked forward to her attending. Many peer leaders spoke about finding the peer researcher as a form of peer supporter for them.

Challenges

Sometimes the questions were triggering, meaning that the women did not feel safe completing them. It was left to peer leaders and facilitators to deal with the fall-out of distress, which was difficult for some. Then there were practical issues: one woman had dyslexia and did not want an intermediary person to complete her form. Another project would have preferred to have the forms in Welsh.

Summary

Be aware of how you frame evaluation so that it doesn’t sound intimidating. Make it part of your everyday work in building trust, respect and accountability.
Know that trust-building takes time. If you wait to evaluate until the participants trust you and each other, you will get a fuller picture.
Do you have space to negotiate with funders about the methods of evaluation and the timescales? Demonstrate your commitment to quality and trauma-informed practice by creating a regular staged system or building blocks towards evaluation data, rather than overwhelming participants with detail at the end.
When your group fluctuates in membership, regular and ongoing evaluation will catch more voices and be more responsive to the group than storing up evaluation activities until the end of the project.
Evaluation can be built into other exercises, such as trust-building and leadership activities.
Share resources with other groups and get the support from other staff and group members, so that evaluation is not one person’s job but the responsibility of the group.
Be open to different methods, including visual for people who are non-literate and bitesize for people with short attention spans. Try experimenting with creative methods to get feedback and evaluation data.
Accessibility: consider how you will provide additional support for participants with disabilities and differences, such as dyslexia, where necessary.
Emphasise the bigger picture: we are doing this evaluation to help shape services, inform best practice and measure the impact of the funding and program.
Be clear that it is the project, not the person, being evaluated, despite the fact that personal views and experiences are being asked for. Use externalising language to help reinforce this.
Be strengths based: start with your group’s individual experiences, and what they already know and like.
Don’t think you have to do the same for everyone. When you have new members or a woman with additional needs, they might need a more tailored approach.

The Evaluation Society is an excellent resource for best practice.