Measuring Outcomes

Print

The Department's success will be determined by the extent to which the departmental outcomes of 'protecting the public' and 'reducing re-offending' are achieved. This success in turn contributes to the justice sector end outcomes of 'safer communities' and 'a fairer, more credible and more effective justice system'; and, through this, to the Government's priorities of 'Economic transformation', 'National identity' and, particularly, 'Families - young and old'.

Attribution and Accountability

Measurement of progress can be intrinsically difficult because outcomes are often influenced by factors beyond an individual agency's control. The nature of the measures developed for each outcome means that some measures are better suited to providing information on the impact of the Department's interventions, as opposed to those external factors.

The Department's interventions seek to change the behaviour of the persons who come under its care and control. The likelihood of individual offenders re-offending is assumed to be based, in part, on relatively stable (though potentially changeable) personal and social characteristics. Personal characteristics include violence propensity, substance dependence, attitudes and values supportive of criminality, as well as deflicits in educational and occupational skills. Social characteristics include a lack of social support, homelessness and lack of employment. The Department therefore offers a range of services that are intended to:

  • reduce the severity of antisocial traits
  • build socially adaptive skills and abilities
  • resolve or mitigate environmental obstacles faced by offenders.

In addition to attempting to encourage personal change, and providing support and assistance to offenders, the Department also provides monitoring of, and direction to, those offenders subject to community sentences and orders. In doing so, the Department endeavours to prevent offenders from relapsing into undesirable patterns of behaviour, as well as enabling preemptive actions to be taken when risks become elevated.

It is recognised, however, that the Department cannot completely control the outcomes experienced by offenders. Key limits to control include:

  • for many offenders, the sentence imposed is too brief in duration to allow meaningful engagement with services
  • many offenders are unmotivated to address the personal and social issues that underlie their offending risk (and there are seldom reasonable or effective ways to compel them to do so)
  • when in the community, the location and behaviour of offenders cannot be continuously monitored, except in certain circumstances such as the first 12 months of an extended supervision order
  • events and circumstances well beyond the control of the Department, which may dramatically and abruptly alter an offender's risk of re-offending, can (and frequently do) occur.

Fundamentally, criminality is understood to be a relatively robust and enduring personal characteristic in adult offenders. Even when exposed to the most sophisticated and intensive rehabilitative programmes and reintegrative services, a substantial proportion of offenders will nevertheless go on to re-offend. The Department endeavours, through its suite of interventions, to reduce the proportion of those offenders who do so.

Further, in relation to the goal of safer communities generally, the Department's contribution is in reducing the risk associated with offenders who have already come under its care and control. The Department can exert no influence that might prevent offending by those who have not previously been under its control, or to reduce the number of young offenders entering the criminal justice system for the first time.

Linking Outcomes to Outputs

The delivery of the Department's outputs directly contributes to the achievement of its outcomes. These linkages are demonstrated in the diagram on pages 36 and 37, which highlights how the Department's range of outputs contribute to its two outcomes of protecting the public and reducing (re-offending).

Departmental Contribution

To demonstrate success, processes must be in place to measure achievement. Progress made towards achieving objectives and outcomes outlined in this Statement of Intent will be reported in the Department's 2006/07 Annual Report. The annual report will detail the work the Department has done with respect to the key priorities agreed with the Minister of Corrections, the initiatives associated with these priorities and the four themes of its Strategic Business Plan 2003-2008. It will also report on the performance measures and standards associated with the output classes.

The Department has developed a set of measures, based on data collected through its offender management processes and similar information obtained from corrections agencies in other countries, which enable comparisons to be made. These measures provide a good starting point for assessing the Department's performance, and will be enhanced over time in conjunction with the review of outcome measurement that is underway within the justice sector. The measures are grouped under the departmental outcomes. Progress in protecting the public is assessed through measures that demonstrate the quality and efficiency of the Department's custodial functions. Progress in reducing re-offending is assessed by speci'c measures developed by the Department - the Recidivism Index and the Rehabilitation Quotient. These measures are discussed in more detail on page 38.

The Department also monitors the application of specific offender management processes to ensure they are properly and consistently applied. Individual processes will also be reviewed periodically to ensure that they remain effective, and, where necessary, will be amended. This activity is intended to support a process of continuous improvement while ensuring the integrity of the information used by the Department for assessing its contribution to outcomes.

International Benchmarking

The Department has developed close working relationships, and participates in the exchange of information with a number of overseas jurisdictions, such as Australia, Canada, England and Wales, and Scotland. Other international relationships are facilitated through the Department's membership of the International Roundtable for Correctional Excellence, the International Corrections and Prisons Association and the Australasian Corrective Services Administrators Council. The information exchanged has been used to develop benchmarks for comparing performance across a range of indicators, including prisoner-to-staff ratios, costs of securing prisoners and occupancy statistics. This data is important because the Department is not able to compare performance in its core offender management functions against other New Zealand organisations. The measures are reported against in the Department's annual reports.

Protecting the Public

The departmental outcome of protecting the public will primarily be attained through the strategies and initiatives outlined in Theme 1 of the Strategic Business Plan 2003-2008. Progress towards the achievement of this outcome is measured through analysis of data collected across a range of quality and effectiveness measures designed to ensure offenders are being managed in a safe, secure and humane manner. These include measuring and monitoring:

  • compliance with, and the administration of, sentences and orders
  • the number of escapes that occur from prisons and while under escort or supervision
  • the number and severity of incidents, including deaths in custody.

Linking Outcomes to Outputs

Linking Outcomes to Outputs.

View larger image of Outcomes to Outputs image (jpg: 493KB)

Reducing Re-offending

Progress towards the achievement of the departmental outcome of reducing re-offending is assessed through the use of two measures, the Recidivism Index (RI) and the Rehabilitation Quotient (RQ). The RI measures the Department's performance in reducing re-offending as a whole, whereas the RQ measures the ability of a specific intervention to reduce re-offending.

Recidivism Index (RI)

The Recidivism Index quantifies the rate of re-offending of a specified group of offenders over defined follow-up periods after release from a custodial sentence or commencement of a community-based sentence. The RI analysis gives re-conviction rates over 12 and 24 months for those released from prison, or commencing community-based sentences. RIs are reported for all offenders in each of the above categories, as well as for sub-groups within categories. The RI analysis uses re-conviction information obtained from the Ministry of Justice's Case Management System (CMS) database.

RI figures are produced for two potential outcomes: re-conviction leading to any sentence administered by the Department, and reconviction leading to imprisonment. Imprisonment figures are perhaps the more critical of the two measures, because this outcome is associated with more serious offences and higher costs of sentence administration. However, the majority of re-convictions lead to non-custodial sentences, and re-conviction figures generally provide a more complete measure of recidivism.

In developing the RI measure, along with other measures derived from its offender management processes, the Department has enabled comparisons to be made with correctional services in other countries. The current indicators provide a starting point for assessing the Department's performance, and will continue to be enhanced over time, in conjunction with other justice sector agencies. It should be noted, however, that the use of re-conviction data as indicators of departmental performance is not straightforward: reoffending, re-conviction and sentencing are subject to a range of influences, many of which are outside the Department's direct control.

Rehabilitation Quotient (RQ)

The Department delivers a range of rehabilitative and reintegrative interventions designed to enable offenders to lead law-abiding lives. These programmes have been designed and implemented in ways consistent with internationally developed best practice principles.

Research on outcomes from rehabilitative programmes has shown that significant reductions in reconviction and re-imprisonment can be achieved when this form of intervention is effectively delivered to appropriately selected offenders.

The purpose of the Rehabilitation Quotient is to gauge the extent to which re-offending is reduced amongst those who received a rehabilitative intervention. The RQ statistic signifies the percentage-point difference in reconviction/re-imprisonment rates between offenders who completed individual programmes and matched 'untreated' offenders. In order to accurately measure the impact of the intervention, each 'completer' is matched with one or more 'controls' who are similar on key variables such as age, gender, assessed risk of reoffending, type of sentence (prison versus community), sentence length and, in some cases, offence type. The length of the follow-up period (for example, after release from prison, and during which reconviction is possible) for both groups is equivalent. Any differences in reconviction rates can then be interpreted as an indication of the programme's impact.

The Department is able to present RQ data on intervention effectiveness for only a proportion of the programmes delivered. A number of programmes cannot be included in the analysis because of inadequate sample sizes - low sample size can be a consequence of low starter rates or high non-completion rates.

Interpretation of RQ results should be treated cautiously because offenders managed by the Department have access to a wide range of services, including educational, employment and general living skills training, as well as various supports to assist with reintegration to the community. These services are available in varying degrees to offenders irrespective of whether they have also participated in the core rehabilitative interventions (the impact of which the RQ attempts to measure). At this stage, the RQ methodology does not ascertain the nature and quality of these services as accessed by individuals, and does not take into account the relative contribution that each of these services may be making to final recidivism outcomes. However, it is widely accepted that the provision of education, training and reintegrative support services can have a significant impact on reducing re-offending.

Alternative Outputs Analysis

The mix of outputs delivered by the Department is not subject to radical change on a year-by-year basis. In particular, the core custodial outputs do not change greatly in their nature, comprising as they do the central functions of providing information to courts, and managing custodial and community-based sentences.

Rehabilitative programmes may be subject to greater change. Programmes administered by the Department have been developed against a background of international research on what works with offenders. Programmes may change over time as new research, including specific evaluation work conducted by the Department in the New Zealand context, emerges.

Analysis of the RQ results that were presented in the 2004/05 Annual Report, together with the evolving knowledge about what works best with offenders, has led to decisions to change the mix of rehabilitative programmes. The essence of the changes will be to target mediumintensity programmes to mediumrisk offenders and high-intensity programmes to high-risk offenders. These programmes will be supported by a comprehensive relapse prevention programme.

In addition, a Short Motivational Programme will be delivered to prisoners serving sentences less than two years in length. The effectiveness of other motivational programmes, such as Tikanga Maori programmes, Maori Focus Units and the Faith-based Unit, will also be assessed.

The Department will also be involved in the continuing work to examine options other than imprisonment for the rising offender population. The Government has made it clear that there will not be any lessening in penalties for serious repeat offenders and hardened criminals, but that for some other offenders the use of non-custodial sanctions may be more productive, and a smarter, more effective use of taxpayers' money.

Evaluation, Research and Monitoring

The Department is committed to the development of evidence-based interventions that can be implemented successfully to assist with the achievement of its outcomes. The Department's own research and evaluation programme contributes to the development of new, or the modification of existing, interventions. This is especially so in the area of applying international work in the New Zealand context. The programme encompasses research on new interventions, assessment of the relevance of overseas approaches in the New Zealand context, and evaluations of the performance of specific interventions. This work provides a critical foundation for the selection of new interventions, or the modification of existing approaches, that support the Department's work in this area.

The Department reviews its research and evaluation programme each year, to ensure that the limited research funding available is targeted to the most appropriate projects. Projects planned for 2006/07, including some multi-year projects, include:

  • evaluation of the Violence Prevention Unit
  • investigation of the characteristics and needs of sex offenders against adults
  • investigation of the characteristics and needs of very high-risk offenders
  • evaluation of the operating philosophy at new prisons
  • evaluation of Corrections Inmate Employment activity
  • effectiveness evaluations of culturally based services, including:

- specialist Maori Cultural Assessment
- Te Ihi Tu (community residential centre)
- Tikanga Maori programmes
- Maori culture-related needs assessment (MaCRNs).

The evaluations will include assessments of the cost effectiveness of these services.

Cost Effectiveness of the Department's Overall Interventions

In addition to the assessments of the cost effectiveness of the above evaluations and of rehabilitative and reintegrative interventions through the Rehabilitation Quotient, the Department is committed to demonstrating the cost effectiveness of all interventions it delivers or administers. Ministers agreed that a joint review of the effectiveness of the Department's funding would be undertaken once the Department received the full 2005/06 Output Pricing Review funding.