Evaluation of the Working Time Directive 2009 pilot projects

Calling Time article by Professor Peter Spurgeon

Background

In 2006 an evaluation team, led by Professor Peter Spurgeon, postgraduate dean, West Midlands SHA, in conjunction with the school of Education at Birmingham University, was commissioned by Skills for Health - Workforce Projects Team to undertake a national evaluation of all the Workforce Projects Team WTD 2009 pilot sites. An obvious overarching objective of the evaluation was to assess the achievement of WTD compliance in the various sites.

A more subtle understanding of the detail of the rotas used and the changes required to implement the different approaches was also part of the evaluation as a means of providing insight and recommendations as to how the NHS more generally might tackle WTD in 2009.

During the first phase of the evaluation, the team has sought to understand better the particular approaches of each site as well as devising a system of data collection, both in terms of quantifiable metrics of outcomes and more qualitative descriptions of the processes involved, for use in the second phase of the evaluation. On the basis of this initial phase an interim (internal) report was submitted to the Workforce Projects Team and this has identified a number of preliminary themes.

Types of projects

15 sites have formed the current evaluation group - new sites recently agreed are not yet included in the dataset. From the current dataset, it appears that such types or approaches can be identified:

• Separation of emergency from elective care
• Re-modelling patient flow
• Creation of specific enhanced acute care teams
• Speciality specific projects
• Community based provision as an alternative to hospital care
• Resource allocation match to equate staff with service need on a flexible basis.

Implementation issues

Whilst all sites have a common goal of achieving compliance, it is apparent that approaches are very different and so too are the issues raised in managing the processes of implementation and change. In the longer term, this considerable diversity suggests that a simple causal link between system change and achievement of compliance is unlikely to be the case.

For example, the clinical response to the initiatives are extremely variable from concerns about lack of training time from junior doctors to consultant perceptions of increased workloads and concerns about patient safety. Some juniors have of course recognised that the re-banding achieved in many sites has adversely affected their pay.

Nevertheless despite these concerns other sites have experienced great enthusiasm and willingness to engage with the changes from all clinical staff. As ever in the NHS, the specific relationships and dynamics in particular settings may challenge the degree to which initiatives can be replicated elsewhere in a straight forward and predictable manner.

The picture with regard to resource implications is again variable and appears to be dependent upon the baseline situation with respect to staff numbers and grades. Clear savings are reported from some sites as a result of re-banding whilst others insist that compliance cannot be achieved without additional resources. As all pilot sites move towards completion it is hoped that a clearer picture will emerge to support more concrete recommendations.

Metrics

In order to provide a robust and measurable evaluation a set of metrics has been devised covering a range of areas that might be affected by the implementation of new rotas or working models. Ideally this quantifiable data would be collated before, during and after changes are implemented.

Some sites have found this more difficult than others due to their start dates and also the nature of baseline information readily available. In discussion with the Workforce Projects Team these metrics have recently been revised in an attempt to simplify whilst still enabling sound statistical analysis to be undertaken.

Evaluation Metrics

• Compliance with WTD 2009
• Proportion of patients presenting who are admitted
• A&E breeches
• Clinical incidents within the project area
• Impact on junior doctor training in context of the pilot
• Re-admissions in the context of the pilot
• Patient complaints in the context of the pilot
• Staff costs per spell within the context of the pilot
• Site specific data.

Next Steps

Over the remaining year of the project, a number of activities are planned to complete the evaluation. Seminar discussions with groups of the test sites to explore particular issues will be held. Site visits will take place to gather more detailed accounts of the implementation issues and the views of a range of stakeholders. Finally the metric data from each site will be analysed and in conjunction with the qualitative data, be used to make recommendations as to how the NHS as a whole might achieve WTD compliance in 2009.

The evaluation will also enable us to develop case studies and best practice resources that can be distributed across the service.

Click here to view the evaluation documents >>>