Lean Evaluation

Complete Report

  • Complete report can be found here.
  • Appendices for the report can be found here.

Executive Summary

The Saskatchewan Ministry of Health (MOH) has committed to a multi-million dollar investment in the implementation of Lean (Saskatchewan’s Lean Management System) across the province.

The goal is a transformation of its entire health system to produce ‘better health, better value, better care, and better teams’. Given the scope and scale of the ambition and the commitment of resources, the Provincial Leadership Team (PLT) of the health system saw the need for a rigorous evaluation of the Lean initiative.  The Saskatchewan Health Quality Council was commissioned to put together an independent research team to design an evaluation plan and to gather baseline information, conceived as the first phase of the evaluation. The University of Saskatchewan was contracted to undertake the first phase.

A multi-year evaluation is anticipated to follow as phase two.

Activities in the first phase have included:

  • Consultation with stakeholders through a Steering Committee, a core group of expert advisors and a stakeholder workshop;
  • Three rounds of interviews: an initial round with nine key decision- and policy-makers to identify priority issues for the evaluation; a second round with decision-makers and KPO representatives in ten regions; and a third round with 27 leaders, service providers and patients in four regions;
  • A systematic review of literature of the use of Lean in health systems internationally;
  • An implementation mapping study examining RPIWs and 5S activities undertaken in four regions;
  • An iterative process of theory development to guide the longer term evaluation;
  • The first stage of an economic evaluation, examining the costs of Lean implementation. In the multi-year evaluation this will be extended to a cost-benefit evaluation;
  • Endorsement of the focus areas for the multi-year evaluation by the Steering Committee;
  • The selection of baseline data for key outcomes indicators for the multi-year evaluation and design of the multi-year evaluation. 

 

Theory in the Lean evaluation

The HQC commissioned a realist evaluation because it is appropriate to the complexity of the intervention and it offers a depth of understanding to inform future policy and practice. A realist evaluation asks for whom, how, in what contexts, in what respects and to what extent interventions achieve their intended outcomes, and for whom not, where not, and why not.  Realist evaluations are theory based. 

The research team has developed a traditional program logic model, which aligns inputs, activities, and outputs with their anticipated outcomes.  This is supported by a set of program theory diagrams which fit ‘behind’ the program logic model, explaining how and why particular inputs and activities are expected to generate particular outcomes.

The impact of Lean on patient safety and efficiency have been identified, and approved by the HQC, as the priority outcomes of interest for the multi-year evaluation.  An overarching hypothesis to be tested in the multi-year evaluation has been established, this being that: Empowerment of staff and changes to leadership practices are concurrently intermediate outcomes of Lean and primary mechanisms by which Lean generates improvements in patient safety and efficiency.  A set of more detailed preliminary context and mechanism hypotheses relating to this overarching hypothesis have been developed.

Formal theories have also been selected:  Argyris’ theories of learning organizations; Reinertsen’s theory for transformational leadership for health care; Kanter’s Structural Empowerment Theory and complex adaptive systems theory. These have been used to inform the draft program theory diagrams and to assist in interpreting some interview data, and will also assist in the interpretation of findings from the multi-year evaluation.   

 

Literature Review

We have reviewed the evidence regarding the impact of complex health service interventions, including Lean, in healthcare. We worked with an information scientist to develop a focused keyword search strategy resulting in 499 search-hits. Thirty studies met the inclusion criteria for relevance and method. 

There was significant variation in Lean definitions and concepts reported in the literature, with only one study referring to the use of a formal Lean program theory. The majority of the included studies employed a pre-post comparison design, and only one study reported the use of interrupted time series design. Over 40% of studies reported on the implementation of only one component of the Lean process.

Value stream mapping was the most frequent activity reported; others included 5S, Rapid process improvement workshops (RPIWs) and Lean basics workshops. Lean was most commonly examined in emergency departments and hospital laboratories. Only two studies reported the application of Lean in more than one setting.      

Almost three-quarters of the studies reported on system improvement outcomes including referral rates, admission and re-admission rates, length of stay, various indicators of time taken for particular functions, surgical error rates, and the number of safety reports. Professional outcomes were reported by nine studies including employee satisfaction, care provider productivity, number of steps saved, nursing time spent with patients and staff overtime. Patient outcomes were employed by six studies using indicators for mortality, complications, patient satisfaction, patient safety and quality metric indicators, out of hospital rates and re-admission rates.

There was variation in the effects reported.  The majority of positive impacts related to improvement of health systems, though only a small proportion of these results were statistically significant. Very few studies reported professional and patient outcomes. Twenty percent of studies reported negative or null effects.  

Little has been documented about the failed attempts or barriers to implementation, adoption and sustainability of Lean principles in healthcare.

 

Implementation mapping

Attributing outcomes to an intervention depends in part on being able to demonstrate that there is, at least, a logical relationship between the activities implemented and the outcomes achieved.  Implementation mapping supports this function and will also help to understand whether the focus of implementation changes over time.  This data will also inform the economic calculations, for example the total costs of the Lean implementation, at baseline and for the multi-year evaluation.

We investigated implementation in four health regions during the baseline period: Saskatoon, Prince Albert Parkland, Five Hills, and Mamawetan Churchill River. We catalogued and analyzed rapid process improvement workshops (RPIWs) and 5S events because they were the most frequently used activities. The results were analysed to capture the main aims and foci of implementation activities, and number of events (in effect, the ‘program dose’).

Implementation commenced in the Saskatoon and Five Hills Health Regions in 2012 and 38 RPIWs and 15 5S events were held in that year. In 2013, 63 RPIWs and 60 5S events were conducted in the four health regions, giving a total of 161 RPIWs and 5S events over the two years.  The foci varied across regions and hospitals, but reduction of lead times and reduction of waste and errors were common.

 

Interviews

Realist evaluation uses qualitative data to gain a deeper understanding of how and why interventions achieve, or do not achieve, their outcomes. In the baseline study, interviews have been used both to collect early data on experiences of implementation and to develop hypotheses for further investigation in the multi-year evaluation. Twenty-six face-to-face interviews and one telephone interview were conducted across four regions, with leaders, allied health clinicians, registered nurses, physicians, clerical/administrative staff and patients.  Audiotaped interviews were conducted by a trained research assistant and verbatim transcripts were analysed by members of the research team.

Many participants suggested that Lean is the ‘best hope yet’ for true system transformation.  Some identified early outcomes for patients including faster access to mental health care, reduced time during admission, reduced patient falls, reduced errors in medications, and more patient-friendly information.  Others reported improved efficiency as a result of better organized workplaces. Some respondents, however, reported that changed processes were not monitored or sustained.  Most patients could not identify any change in their own care.

There was early evidence that Lean transforms some participants ‘mental maps’ in regard to how to act in situations. This was more apparent with leaders than with other staff, suggesting that intensity of training and/or Lean implementation experience is necessary.  It appears that Lean generally serves to democratize the system by giving a voice to groups that have been typically disenfranchised, such as patients, family members and front-line staff.  Both patient participation and leadership presence ‘on the shop floor’ were valued by respondents.

The highly structured nature of Lean provides leaders with a set of interdependent tools which in turn instils confidence that they can enact system change.  The structure, however, can be rigid, prescriptive and stifling. Some leaders reported discrepancies between the implementation processes they experienced and the espoused values of Lean.  Certain aspects of the training process were significant sources of discomfort, anxiety and even physical symptoms for some leaders. 

Lean activities such as RPIWs directly empower those who directly participate – staff, leaders and patients – when the processes are well run.  ‘Permission to fail’ and ‘permission to improve innovations in future’ reduced the pressure to get things right the first time and encouraged experimentation with new ways of operating. There was as yet no evidence that staff empowerment extends beyond direct participation in Lean activities, although this may be a function of the questions asked. Interviews were conducted before the focus on empowerment had been decided.

 

Economic evaluation

In the baseline study, a methodology for the estimation of cost for the Lean implementation in Saskatchewan has been developed. Direct and indirect costs are estimated. The indirect cost is primarily due to the opportunity cost of personnel time used in Lean events, which has been estimated using market wage rates.  Direct costs include consulting fees paid to the John Black and Associates, physician remuneration and honorariums paid to the patient/family representatives in Lean activities; travel, accommodation, and other incidental expenses; backfilling and overtime payments and the incremental cost of the Kaizen Promotion Offices (KPOs). 

Using the general approach described above, we estimated a preliminary cost of an RPIW, which ranges from $31,000 to $38,000. However, these results do not include costs associated with the JBA consultants’ time, travel, and other incidental expenses, or overtime payments due to backfilling because data is not yet available for these costs.  An updated and more comprehensive costs report will be provided once data is available.

The multi-year evaluation will provide a cost-benefit analysis and may include analyses for Emergency Departments and for specific health regions.

 

Methodological development for the multi-year evaluation

Consistent with the Phase 1 evaluation of Lean, the evaluation design we propose for the multi-year evaluation combines a realist evaluation with a cost-benefit economic evaluation. 

With the specific endorsement of the Saskatchewan Health Quality Council, we will focus on improved patient safety and efficiency as the primary outcomes of interest and on changed leadership and staff empowerment as the main mechanisms of interest. We will examine both acute care and primary care, but with some greater focus on acute care and with a nested case study examining Emergency Departments. 

Outcome indicators have been selected and are aligned with our program logic and selected aspects of our patient safety, leadership and staff empowerment program theories. Semi-structured interviews and focus groups will be used to refine information about mechanisms of change and to develop understanding about how aspects of context affect whether and how those mechanisms operate.  Additionally, an ethnographic study will be conducted as part of the in-depth study of Emergency Departments.

In the second year, we will develop purpose designed surveys, based on outcomes from the first year’s qualitative data, to assess the commonality, or otherwise, of relevant mechanisms and their relationship to relevant aspects of context.

Implementation mapping data will enable us to relate aspects of implementation to mechanisms and outcome patterns. The economic evaluation will enable examination of improvements in efficiency and costs and benefits.

A funding application to support this methodology has been developed.