- Luc Moens, Price Waterhouse, UK
Luc Moens, presenting the methodology, results and the lessons learnt from Price Waterhouse's monitoring and evaluation of the first fifteen months (January 1996 to March 1997) of CSV's Citizens' Service Pilot Schemes in the UK, sought to share good practice on the monitoring and evaluation of NYS programmes.
In January 1996, CSV launched pilot schemes in Cardiff, the London Borough of Southwark, and Sunderland for a national Citizens' Service, the main objectives of which were to recruit annually 150 young people aged 18-25 for up to a year; to provide placements which are of community benefit; and to supervise the placements and to support the volunteers as necessary.
Key findings of the monitoring and evaluation were: (i) trends in recruitment of volunteers are encouraging, and pilots are successful in finding places for volunteers (available places outnumber volunteers recruited); (ii) about one third of volunteers work in the education sector, 26% in community care and health services, 18% in environmental projects; (iii) the pilots are attracting a wide range of volunteers aged 18-25, including university graduates, some young offenders and school leavers; (iv) the initial cost per volunteer varied according to location, and was slightly in excess of the initial target range; (v) assuming the value of the work done by volunteers to be £10 per hour in Southwark and £7 per hour in Cardiff and Sunderland, the return per pound invested is, on aggregate, 1.5; and (vi) under a less favourable assumption, say, £5 per hour, it is 0.9; once target numbers are reached, at £5 per hour the return becomes 2.8.
The ultimate objectives of the evaluation were to assist formulation of policy in the future, and to inform practitioners on the best ways of handling Citizens' Service schemes. More specifically: the approach used was designed to collect and analyze information about schemes in order to enable the evaluators to: (i) identify successful approaches that might be adopted more widely; (ii) make suggestions for improvements in project management and monitoring; (iii) identify potential obstacles and barriers to success, together with mitigating or corrective actions; and (iv) draw conclusions on the relative cost effectiveness of different programme components. Key issues addressed included: critical success factor of pilot schemes, achievement of objectives, and the impact and effects ie what has happened as a result of the programme. Specific issues included: objectives and targets of the pilot schemes; best methods to identify what is working and what is not; how to optimize the recruitment process; how to maximize the individual satisfaction of volunteers and the retention rate; how much supervision and support is needed and wanted by volunteers; how much induction and training is needed and wanted by volunteers; what needs to be in place before an organization can properly use volunteers; job substitution - how to spot it and guard against it; how to maximize the positive impact of volunteers in their project; and how volunteers can most effectively lengthen and strengthen services.
The evaluation was an `evaluation for management': what was being evaluated should change and improve as a result of the findings during the course of the study. Guiding principles included: independence, top-down and bottom-up reviews, qualitative work combined with quantitative techniques, linkages with monitoring, the need to take account of local specific characteristics, pragmatism (recognizing that the resources of the evaluation are limited), and creating incentives for stakeholders to participate.
Monitoring and Evaluation Design: the variety of analytical and evaluation methods used included: (i) ; the systematic collection of monitoring information. Data collected covered: the three pilot schemes, every volunteer and every project, and were regularly logged in spreadsheets for easy retrieval and analysis (ii) a longitudinal analysis of all volunteers, at different stages of their involvement in the scheme (entry, exit; start and completion of each programme); (iii) a programme of interviews with beneficiaries; (iv) discussions with volunteers, face-to-face interviews and focus groups; (v) case studies (two per scheme) to illustrate the key learning points; (vi) site visits - each pilot scheme was visited twice; (vii) monthly monitoring meeting to share lessons; and (viii) longitudinal analysis to track where volunteers have moved on to (still in progress).
Responsibilities in the monitoring and evaluation process were as follows: Price Waterhouse was responsible for the preparation and management of all forms used for the evaluation; for the editing, coding and tabulating of results; for the analysis and recommendations, and for reporting. Citizens' Service managers were responsible for: completing the monthly monitoring form; completing the quarterly cost schedule form: completing the logsheets for individual volunteers and projects; and making sure that volunteers and projects completed the assessment forms.
Some of the key indicators used were: performance targets and indicators - efficiency of the recruitment process (number of enquiries, interviews, starts), analysis of drop-out rates, average length of stay, assessment by volunteers (degree of satisfaction), assessment by placement supervisors (degree of satisfaction); profile of volunteers; and profile of placements.
The monitoring and evaluation produced learning points for both Citizens' Service and for Monitoring and Evaluation itself. The main learning points on Citizens' Service were: (i) Citizens' Service needs to establish a clear identity, distinct from other job-creation `welfare-to-work' or training schemes; (ii) flexibility in terms of commitment, a range of opportunities etc is valued; (iii) the scheme requires both a placement service and facility for support and mentoring of volunteers; (iv) volunteers put value on the voluntary nature of the scheme; (v) the scheme should be open to all and operate a principle of non-rejection. It has to be recognized, however, that this has cost implications; (vi) Citizens' Service should accommodate both individual and team placements; (vii) the evaluators recommend a two-step process: an `orientation' period, followed by a long-term commitment; (viii) Citizens' Service must take into account how volunteering fits into young people's lives; and (ix) Citizens' Service should be delivered locally, probably within a clear national framework with independent audit and quality assurance.
With regard to the monitoring and evaluation itself, there are many lessons: in terms of the contribution to learning and other effects on the organization - the line between commenting upon matters relating to efficiency and effectiveness, and the policies which underpin a department's operation can be a fine line; ex-post evaluation is often carried out too late to be a strategic input in shaping the policy/programmes; it is important to be `customer-focused' eg by collecting feedback from beneficiaries and feeding findings into the decision-making process; being at the cutting edge of evaluation enhances rigour and discipline, fosters an `evaluation culture' (eg better procedures, create short learning loops, think more operationally, moving towards a learning organization); and being aware that the first draft of findings can be a `nasty shock' for some stakeholders (stakeholders not familiar with evaluation, or unhappy with style ie `rough facts'; emotional investment by programme managers, and reluctance to have `own failures, mistakes' in the public domain).
Lessons on mutual trust and respect include: the need to clearly define roles and responsibilities at the outset; the need to educate stakeholders re. objectives of the evaluation and the nature of the outputs; the need to build mutual trust and respect; and the need to know and respect the organization you are evaluating (even if a major culture change is needed!). It is also necessary to know where to draw the `thin blue line' between the results of the evaluation being `useful' or `used' - evaluation reports will be used by stakeholders, they can become ammunition for opponents, but they can also lift the profile of a programme/policy eg by giving it respectability (`gravitas'). There are also lessons for the conduct and attitude of the evaluator: keep lines of communication open throughout and after the evaluation; be open-minded - let stakeholders put across their viewpoints; stick to the facts; stakeholders value a professional attitude; and `honesty is critical and advocacy is not an option'. Lessons for the role of the Advisory Group include: the role of the Advisory Group is very important to support the evaluator (tying up/confirming questions and methods, providing the opportunity to validate findings, addressing sensitive policy areas); but the role of the Group is also important in voicing the interests/views of external stakeholders. There is also always the need to keep in mind the nature of the audience the evaluation report is addressing.
There is wide agreement on the essential qualities of a good evaluator and a good evaluation report: for a good evaluator - professionalism, honesty and integrity, critical distance, respect for the facts `the ring of truthfulness', and the need to keep facts separate from comments; and for the evaluation report - meeting needs, having a relevant scope, and a defensible design etc.
There is a need to care as much about the fairness of the process as about the outcome of the evaluation - `Fair process profoundly influences attitudes and behaviours that are critical to high performance. It builds trust and unlocks ideas'. Three bedrock elements of fair process are: engagement - means involving individuals in the decisions that affect them by asking for their input and allowing them to refute the merits of one another's ideas and assumptions; explanation - means that everyone that is involved and affected should understand why final decisions are made as they are; expectation clarity - requires that once a decision is made, managers state clearly the new rules of the game.
In conclusion, the main messages about monitoring and evaluation are: multiplicity must be recognized; absolutism is misguided; keep communication lines open; honesty and integrity are critical; but so is the fairness of the process; quality of staff is key; and there is a need for minimum standards.
4th
Global conference on national youth service (1998) 
