Evaluation
Evaluating co-design
Five main principles are established to guide the evaluation of the co-design process:
- equality of participation
- quality of participation
- effectiveness of procedures
- stakeholder satisfaction
- value persistence.
Equality of participation
This relates to the opportunity for any stakeholder to have a voice, take action, or to influence the outcome of the co-design process. This is evaluated at two levels:
- stakeholder representation
- stakeholder engagement.
Stakeholder representation refers to the extent to which stakeholders who engage with the co-design process represent all possible stakeholders. To assess this representativeness stakeholder mapping should be undertaken for various partners. Community mapping can occur at Stage 1 of the co-design process, characterising community demographics, including gender, ethnicity, household size, employment, and other context specific factors.
A decision should be made on expected representation in relation to feasibility (e.g. 10%).
Stakeholder engagement refers to the extent to which stakeholders are actively involved in co-design activities. Engagement can be recorded for attendance at workshops and activity during the workshops.
Attendance at workshops should be available to any stakeholder, and absence of stakeholders from workshop should be assessed. Systematic exclusion of stakeholders may occur due to recruitment bias, practical arrangements (e.g. workshop timing, other commitments), or interest and motivation. Evaluation should assess which stakeholders are attending and which expected stakeholders are not present.
During co-design activities, active engagement can occur individually, across a number of groups and group sizes. Participants should have the opportunity to participate at each level and stage. While it is not expected that each participant will want to or feel the need to participate equally with others, the evaluation of equality of participation balances opportunity with activity. During co-design activities, fieldnotes are used to capture the extent to which each participant is provided with and takes the opportunity to participate. This equality should be established within and across co-design workshops.
Quality of participation
Co-design activities are expected to produce discussion and other materials, such writing, sketches, and voting sheets. Ideally discussion should be open, free flowing, and productive. The production of materials should be assessed as a process and with an outcome.
Measures of quality of discussion can relate to the levels of stakeholder engagement in terms of production of new information, disclosure of personal experience, and creation or representation of novel or diverse ideas or experiences. This should not preclude diverging perspectives, contradicting experiences, or disagreement.
However, discussion should be free flowing, require minimal prompting while following the structure of activities. These can be evaluated through measures of frequency of facilitator intervention, divergence from topic, and turn taking. Evaluation should consider the extent to which these features help progress towards a desired outcome or outcomes.
The processes of production of co-design materials should be assessed against the outcome. Observation of the production should take note of the ease of participation, at the start and while undertaking the tasks, the challenges for completion, and the extent to which participants complete the activity.
Production should also represent equality of participation, whereby each participant has opportunity to produce or effect the production of outcomes.
Following the co-design activities, the produced material should be collated and recorded where possible, for instance, through photographing materials. The produced materials and discussion should be evaluated against the specified aims for the activity, their utility to refine the design space, their diversity, the representativeness of the community values, and their overall relation to the design space.
Effectiveness of Procedures
Across the stages of co-design, the overall aims and objectives are achieved through activities. Each activity includes sub-goals, data collection requirements, and outputs. For each co-design workshop a procedure is designed, which includes activities and expected outcomes and timings. Following the delivery of each workshop, the effectiveness of the procedures for structuring participation should be evaluated. This can include assessment of keeping to time, use of physical space, suitability of materials, and should consider the equality and quality of participation as they relate to procedural design.
Stakeholder Satisfaction
Stakeholder satisfaction should be assessed at three levels: the overall process, the co-design workshops, and the co-design activities. Satisfaction should take account of the extent to which community partners and stakeholders feel their needs are being met, that activities and procedures support them in moving towards need fulfilment, and whether they are achieving satisfactory equality and quality in participation.
Stakeholder satisfaction can be assessed for the overall process through explicit feedback (such as anonymous questionnaires) and through implicit feedback such as participant retention rates, wider community engagement and participant diversification, and observations regarding the quality of participation. Informal feedback from participants can also be included in the assessment.
Value Persistence
Value elicitation is a prominent feature of the co-design process during characterising the community and requirement capture. All activities of co-design should move these values forward, and it is therefore necessary to evaluate whether these values persist, whether needs related to those values are met, and whether the process of co-design has raised or altered these values. Values should always be considered in terms of their representativeness of the stakeholders.
During detailed design, values are explicitly referenced and evaluated with stakeholders against the design proposal. Open discussion can support extensive discussion on the values, and the satisfaction questionnaire can be used to explore changes in the values.
During each phase of the co-design process, researchers collect and share reflections on the co-design process. These include: fieldnotes, community descriptions, observations and video recording, voting sheets and satisfaction questionnaires.
Co-design activities also elicit and produce materials designed to respond to specific aims and sub-goals, such as value lists, vote sheets, and visual media. All materials should be collected and recorded for evaluation according to the following recommendations.
Leadership
The aims and objectives can be communicated to stakeholders where appropriate. This can happen at various stages of the co-design process. This can support refining aims, managing and setting expectations, and creating trust and openness for evaluation. It is expected that aims will change with respect to the expected level and nature of outcome as the process advances, and these refinements should also be made clear.