Appl Health Econ Health Policy (2016) 14:559–568 DOI 10.1007/s40258-016-0250-5

ORIGINAL RESEARCH ARTICLE

A Qualitative Evaluation of Program Budgeting and Marginal Analysis in a Canadian Pediatric Tertiary Care Institution Neale Smith1 • Craig Mitton1,2 • Mary-Ann Hiltz3 • Matthew Campbell3 Laura Dowling4 • J. Fergall Magee5 • Shashi Ashok Gujar3,6

•

Published online: 11 June 2016  Springer International Publishing Switzerland 2016

Abstract Background Hospitals in Canada are being asked by governments to improve efficiency and do more with fewer resources. Healthcare decision makers are thus driven to find better ways to manage budgets and deliver on their mission. Formal processes of priority setting and resource allocation (PSRA) are one means to this end. Objective This paper reports an evaluation of one such approach, Program Budgeting and Marginal Analysis (PBMA), as applied at a children and women’s tertiary care facility in Nova Scotia, Canada. A brief evaluation conducted immediately after the conclusion of the PBMA

Electronic supplementary material The online version of this article (doi:10.1007/s40258-016-0250-5) contains supplementary material, which is available to authorized users. & Neale Smith [email protected]

process was supplemented with a larger retrospective evaluation. Methods The retrospective evaluation included 26 face-toface individual interviews with senior and middle managers who took part in PBMA. Interview transcripts were analyzed against a template consisting of 19 elements of structure, process, attitudes, and outcomes associated with high performance in PSRA. Results Respondents had a good experience with the implementation of PBMA, and considered it an improvement over past practice. Success was attributed to effective leadership, and substantial efforts to engage staff members. Understanding of economic and ethical principles of decision making was reportedly increased. Areas for improvement included ensuring that everyone participated in good faith, better communication of final results, and 2

School of Population and Public Health, University of British Columbia, Vancouver, BC V5Z1M9, Canada

Craig Mitton [email protected]

3

Quality and System Performance, IWK Health Centre, Halifax, NS B3K6R8, Canada

Mary-Ann Hiltz [email protected]

4

Nova Scotia Health Authority, Halifax, NS B3H1V7, Canada

5

Pathology and Laboratory Medicine, University of Saskatchewan and Saskatoon Health Region, 103 Hospital Dr, Saskatoon, Sk SK S7N0W8, Canada

6

Faculty of Medicine, Dalhousie University, Halifax, NS B3H1X5, Canada

Matthew Campbell [email protected] Laura Dowling [email protected] J. Fergall Magee [email protected] Shashi Ashok Gujar [email protected] 1

Centre for Clinical Epidemiology and Evaluation, Vancouver Coastal Health Research Institute, University of British Columbia, 7th floor, 828 W 10th Avenue, Vancouver, BC V5Z1M9, Canada

560

stronger follow-through to determine if anticipated changes and benefits in fact occurred. Conclusion The evaluation framework employed here proved useful in assessing the quality of this resource allocation exercise. The results are directly useful to local decision makers, and the identified strengths and weaknesses are broadly consistent with those reported in studies of other organizations.

Key Points for Decision Makers This study provides evidence for the usefulness of PBMA in one pediatric setting, which is an area where very little formal priority setting and resource allocation has been reported. It reports evaluation results from two points in time which better enables decision makers to observe the longer term impacts of their efforts to allocate resources. It is based upon a comprehensive framework of elements which identifies structures, processes, and behaviors that should lead to the achievement of successful priority setting outcomes.

1 Introduction Health service delivery organizations in Canada are being asked by governments to improve efficiency and, in many cases, to do more with fewer resources. Under these conditions, healthcare decision makers are driven to find better ways to manage budgets and deliver on their missions. We call these efforts priority setting and resource allocation (PSRA): there is now a fairly substantial literature describing how various organizations, in Canada and elsewhere, have attempted to transform their practices in this area. The literature articulates—normatively and empirically—the case for formalized processes [12, 16]. There is much said about features of good process, and facilitators and barriers to the implementation of these [13, 19]. However, explicit evaluation of formal PSRA processes is rare [18]. Such retrospection is needed to ensure that practice changes achieve intended benefits such as making resource allocation decisions more transparent, more defensible, better aligned with an organization’s strategic priorities, and a better use of dollars to achieve valued health outcomes. In this paper, we assess the experience of one formal priority setting process (Program Budgeting and Marginal Analysis, or PBMA) at the IWK (Izaak Walton Killam)

N. Smith et al.

Health Centre, a children and women’s hospital in Nova Scotia, Canada. Our primary aim is to share practical insights and success factors for leading the transition from a historical process to a formal, explicit approach. PBMA was used at the IWK to allocate funds for pediatric services in fiscal year 2012/2013. An initial evaluation was conducted immediately post-implementation. A more extensive evaluation was subsequently carried out—this was intended to obtain a longer term and more reflective perspective, once organizational members had a chance to see whether the choices made in the process were in fact carried out and what consequences ensued.

2 Background 2.1 Context The IWK Health Centre is a regional hospital providing primary, secondary, and tertiary healthcare services to Canada’s Maritime Provinces. The IWK is structured around a Program-Based Care Model, delivering services to patients and families through three programs: Children’s Health, Mental Health and Addictions, and Women’s and Newborn Health. Traveling clinics, particularly in the areas of pediatric neurology, orthopaedics, cardiology, and respiratory medicine, also bring the IWK’s specialist expertise directly to Maritime communities. The IWK has more than 3600 employees and over 900 volunteers. From April 2014 to March 2015, there were 29,015 visits to the Children’s Emergency Department and 237,384 ambulatory care visits. The IWK is also a major research and training facility, affiliated with Dalhousie University, serving as a primary clinical resource for pediatric and obstetric teaching in a broad range of health professions including medicine, nursing, other allied health services, and child life. For the 2011/2012 fiscal period, healthcare transfers from the Nova Scotia provincial government to the IWK were frozen. In practical terms, this meant that across the organization the IWK had to find the dollars to off-set inflation pressures equivalent to 7 % of its 2010/2011 operating budget. Usual business planning means were used to reduce the organization’s spending in consequence. The resulting ‘‘spreadsheet’’ exercise focused on individual units rather than on an integrated, systems view of resource allocation. The organization heard from physicians, allied health professionals, support staff, and all levels of management that the strategies identified to address the budget shortfalls were not the strategies they would have identified if asked. For the 2012/2013 fiscal period, the IWK faced a further budget shortfall that equaled a 3 % real reduction in health transfers from the provincial government. Rather than repeating the same cutback exercise, the IWK

A Qualitative Evaluation of Program Budgeting and Marginal Analysis

Executive Leadership Team (ELT) took this shortfall as a strategic opportunity that would allow for the introduction of a formal, explicit resource allocation framework to facilitate and support organization-wide decision-making. PBMA was identified and implemented to assist the organization in managing these financial pressures by providing a methodology for identifying both disinvestment and investment options. The introduction of PBMA was intended to complement existing business planning processes. The basic stages of a PBMA process have previously been described in the literature [14]. 2.2 The PBMA Implementation Process at the IWK The PBMA initiative had a three-part structure (See Supplemental File 1: IWK PBMA Project Structure). A Project Management Committee (N = 6) was charged with overseeing the rollout of the process, communication and education efforts, and initial vetting of proposals. A larger Working Group (N = 18) consisted of representatives from a cross-section of the organization. Membership included physicians, clinical and operations leaders, managers, a patient and family representative, a bioethicist, and a single executive member. The Working Group developed explicit criteria for assessing investment and disinvestment proposals, and took the lead in reviewing and recommending options. An Advisory Panel (N = 18) was tasked with approving all the Working Group products. In addition, uncounted other managers and front-line workers throughout the organization committed directly to the development of business case proposals put forward through their Departments. Criteria were intended to be clearly defined, mutually exclusive, and operational enough to compare the funding options. These characteristics were desirable in order to improve consistency and increase public defensibility of decisions. Fourteen criteria, in three overarching

561

categories—Strategic Alignment, Health Impact, Organizational and Health System Impact—were employed. The criteria were assigned weights totaling 100 points, with individual criteria ranging from 4 to 12 points. The preeminent criteria on this basis, then, were alignment to the IWK mandate (12 points) and impact on health status (11 points). The full set of weighted criteria is provided in Supplemental File 2: IWK Evaluation Criteria. Note that PBMA emphasizes the importance of locally determined and contextually appropriate criteria [14], so we are not suggesting in any way that these criteria should be adopted uncritically by other organizations. Over 72 distinct criteria have been used by organizations which have implemented a PBMA or other multi-criteria decision analysis approach [3]. PBMA at the IWK was an ‘‘All In’’ process. All programs and services used standardized templates to submit proposals to a central pool; both investment and resource release (disinvestment) ideas were required. See Table 1 for some examples of the sorts of proposals received. Proposals were solicited from across the organization through an extensive education campaign; project co-leads attended department meetings and met with interested persons in small groups or one-to-one. Every staff member was allowed to put ideas forward for consideration, anonymously if desired, without the need for prior vetting from management. In total, 440 short-form proposals were submitted from across the organization; 346 moved forward to present the long form submissions/business case rationale. Of these, 223 were deemed feasible to implement for the next budget year and so given full review. Eighty-one were found to be efficiency proposals (i.e., proposals that resulted in a savings without any structural change or patient impact) and were approved for immediate implementation. Decisions upon the remaining proposals—71 calling for new investment and 71 for disinvestment—constituted PBMA’s major recommendations for resource re-allocation. Given the fiscal context described above, the greatest pressure on

Table 1 Illustrative examples of Program Budgeting and Marginal Analysis (PBMA) proposals

Care Programs (e.g., Primary Health, Children’s Health, Mental Health and Addictions)

Proposed disinvestments

Proposed investments

Nova Scotia Car Seat Initiative—cut to car seat technician training

Obesity management program for children and families

Discontinue supplying diapers for families

Implement home surveillance programs for high risk infants DBT pilot program

Discontinue ante-partum home care service Service Department Programs (e.g., Engineering and Environmental Services, Information Management and Technology, Laboratory Services)

Decrease bone densitometry services Reduce funding for internal staff appreciation and awards Reduce cleaning in non-clinical area offices

Energy monitoring equipment Expansion of parenteral unit dose service to pediatric intensive care patients Hand hygiene auditing system

562

N. Smith et al.

Table 2 Summary of disinvestment proposals by department No. of proposals

One time start up costs ($CAN)

FTE impact

2012/13 NET savings ($CAN)

$229,494.00

Proposals from care programs Children’s health

16

$80,200.00

-2.09

Women’s and newborn health

11

$357,952.00

-6.0

$95,222.00

Primary health

10

$0.00

0

$76,300.00

4

$1,007,000.00

-33.11

$909,222.87

Laboratory operations Engineering and environmental services

9 6

$6000.00 $0.00

-3.6 -2.2

$208,799.00 $399,526.70

Finance and corporate services

4

$0.00

-1.8

$201,482.00

Information management and technology programs and services

3

$84,145.00

-11.6

$34,726.00

Mental health and addictions Proposals from service departments

Diagnostic imaging

2

$0.00

-1.4

$97,766.00

Pharmacy

2

$0.00

-1.5

$146,807.00

FTE full-time employee

the IWK was to achieve cost reductions through disinvestment. Table 2 shows the distribution of approved disinvestment proposals from across Departments; approximately 60 % of these came from the care program areas, and the remainder from support service areas. The Working Group reviewed all of the submitted proposals and used the criteria to score each proposal to obtain a summative measure of overall benefit. Each criterion was scored on a scale from -3 to ?3. A negative number shows that a proposal, on that criterion, has overall negative effects; a positive number shows overall benefit. For instance, on the criterion of flow/integration, a -3 indicates a proposal that would create significant bottlenecks if implemented, while a ?3 would be assigned to a proposal whose net effect would be to remove significant bottlenecks (see Supplemental File 2). Proposal options were ranked in terms of assessed benefit and recommendations were made to shift resources to optimize overall benefit. Across all the disinvestment options, assigned scores ranged from ?159 to -160. In the end, proposals with rankings of -90 or greater were approved to proceed; those with scores below were not. Note that approving proposals with net negatives reflects a loss in value for the organization, but these are still worthwhile if the saved resources can be used in ways that return greater overall benefit when assessed against the same criteria [4].

3 Methods The initial evaluation was conducted within the 2 months immediately following the submission of full PBMA proposals to senior management (i.e., during March–April,

2012), by a researcher (CM) who had provided education and support to the process. As this was considered part of the implementation and a quality improvement exercise, no formal ethics approval was required or sought at this time. Sixteen participants (two clinical leaders, three physicians, four managers, six directors, and the CEO) were invited and participated in the evaluation. There was a wide representation in terms of departments, roles, and of involvement in the PBMA process. Brief interviews were conducted by telephone. Researchers made note of key themes which recurred across interviews; the evaluation report was submitted to the IWK senior management in April 2012. The second evaluation was funded by an external research grant; it received formal ethics approval at both UBC and the IWK. For this effort, researchers began by familiarizing themselves with the PBMA process through reviewing project documentation. Subsequently, 26 faceto-face interviews of approximately 30–60 min in length were conducted during fall 2013 (31 persons in total were invited). Respondents were initially selected by a member of the research team (MC), based upon their known degree of involvement with the PBMA process. Both advocates and those known to be more skeptical were included. Advocates (n = 19) were defined as those who had promoted the PBMA model or had expressed the view that it was a success; skeptics (n = 7) were defined as those who had expressed doubt about the process beforehand or whose comments emphasized the difficulties and challenges. Since there is no information available about the views held by everyone who was affected by the PBMA process, it is not possible to say the extent to which these numbers are representative; nonetheless by deliberately

A Qualitative Evaluation of Program Budgeting and Marginal Analysis

soliciting both perspectives the authors felt it most likely to capture the range of strengths and difficulties which might have been observed. Each of the selected potential respondents received a letter from the IWK Chief Executive Officer, inviting their participation and demonstrating that the research had organizational support. Each interviewee was also invited to nominate others for participation, but no new names were obtained through this. Interviewees mainly represented PBMA leadership; we spoke with six members of the Project Management Committee (out of six), 12 out of the 18 members of the Working Group, and ten of the 18 members of the Advisory Panel. (Since some individuals played a role in multiple groups, this does not equal the total number of interviews.) Interviewees included five members of the senior leadership group, three directors, six managers, three senior physicians, five department heads, three allied health workers, and the community member representative of the PBMA working group. All of the care programs were represented, as well as approximately twothirds of the support service areas. Interviews were conducted by a research associate hired by the IWK for this project. She had no previous involvement with the IWK or the PBMA process; this may reduce any biases in interview response due to familiarity. All interviews were recorded and transcribed. Data were analyzed against the high performance template [7]; that is, comments about strengths and weaknesses of the process were coded into one of the 19 elements, whichever in the view of the coder (NS) was the best fit. Preliminary results were reviewed by two coauthors (LD, SG) and in summary form by the research team as a whole. Only three people took part in both sets of evaluation interviews. Supplemental File 3 provides the specific evaluation questions asked at each point in time. The high performance framework employed in this paper to evaluate the PBMA process at the IWK was developed in a previous 3-year project comprising literature review, online survey, and case studies of high performing organizations in Canada. The framework consists of four domains—structure, process, attitude/behaviors, outcomes—and 19 elements; these collectively reflect the wide range of factors enabling success in healthcare priority setting and resource allocation (see Table 3). The intent behind developing such a framework ultimately was that organizations can then ‘‘assess their current practices and determine areas for improvement based on evidence for best practice’’ [11, p. 99]. Researchers partnered with decision makers at the IWK in order that this objective could be pursued in this study. An assessment using the framework elements has previously been conducted successfully in two health organizations [5]; thus, we have reason to suppose it

563

provides decision makers with a useful guide for where their processes may be aligned with good practice, and where they may be weaker and candidates for improvement. We used the framework’s categories to analyze our data; this provides a broad lens with which to view the successes and challenges of PBMA as implemented in this setting.

4 Results Most (11 out of 16) respondents in the initial process evaluation felt that the recommendations (i.e., the approved investments and disinvestments) reached at the end were the right ones for the future of the organization (three had some disagreement, and two felt unable to comment). All stated that PBMA should continue at the IWK. The process was generally considered fair; the only concern expressed was around some departments not meeting their disinvestment proposal threshold. In terms of strengths, respondents stated that project co-leads and the project management team were very effective and efficient. It was felt that wide engagement was achieved—the possibility of anonymous suggestions was part of this in that truly everyone in the organization did have an opportunity to input. However, this also led to the submission of ‘‘too many’’ short forms, of highly variable quality. This required much effort on the part of proposal reviewers. Furthermore, respondents argued that while anonymous submissions facilitated wider engagement, there should have been an opportunity for affected directors to review and comment on these submissions, even to register as part of the review any disagreements or dissent they might have. Finally, an identified weakness of the process was clarity regarding the final decisions. A more detailed communication plan for the part of the process that comes after the final ratings and ranking would have been welcomed by respondents. The second evaluation re-iterates many of the early findings, which now can be put in the more comprehensive context of the framework of elements. Our aim here is to show which of the elements can be identified as present or absent in the data, rather than the extent to which informants judged them to be so. ‘‘In general, qualitative research does not seek to quantify data’’ [15, p. 114]. A point or idea that occurs infrequently, perhaps only once, within a set of interviews is not necessarily a less important insight than what might come from more commonly shared or expressed viewpoints. Sandelowski [17] argues that qualitative research must concern itself with showing that any identified themes or patterns exist in a coherent relationship with one another, rather than how common they might be; the latter aim is more appropriate for

564

N. Smith et al.

Table 3 Elements of high performance in priority setting and resource allocation, with summary of findings from the IWK (Izaak Walton Killam) Health Centre evaluation, 2013 Structures

Processes

Attitudes/behaviors

Outcomes

S1: Ability and authority

P1: Process

O1: Actual reallocation

One department had much of its funds in protected envelopes set by the provincial Ministry of Health. Otherwise, decisions makers were free to move resources among program areas in the IWK

Criteria were prima facie applicable to the IWK context. It was unclear if criteria were applied consistently by those charged with rating investment and disinvestment proposals Some Departments attempted to ‘game’ the process by submitting only unrealistic disinvestment proposals

A1: Respectful working relationships

a

Program documents and interviews detail the substantial efforts that were made to ensure staff at all levels had input into the resource allocation decisions resulting from PBMA

All respondents offered compliments in regard to the ability of the executive members to work together in delivering on the PBMA process

Available data suggest that money was saved, and some disinvestments and resource reallocations occurred. However, no respondent suggested that the success of the process should be judged by whether or not this occurred

P2: Communication

A2: Culture of improvement

O2: Stakeholder endorsement

While information about the PBMA process was widely disseminated to start, the rationale for final decisions reached via the process was one aspect of communication about which dissatisfaction was expressed

The IWK efforts to address resource allocation and improve practice via PBMA reflects a culture of improvement

Internal stakeholders supported the implementation of PBMA, though the provincial government offered signals of caution. However, more than one respondent expressed the view that the process was simply another way of orchestrating cutbacks, rather than forming the basis for a new way of doing business

S3: Coordination

P3: Skill development Though not a major theme, some respondents suggested that the IWK would have benefitted from further staff training around PBMA and PSRA

A3: Long-term strategic alignmenta

O3: Greater understanding

The IWK uses strategic planning to guide its priority setting and resource allocation efforts

Some, but not all, proposals accounted for their impacts upon other services within the organization

S4: Stability

P4: Follow-through and change management

A4: Fit with social and community values

It was unclear if the decisions made in PBMA were fully implemented or if their actual impacts were assessed

The IWK’s mandate is to deliver children’s health services. The role of community partners, and direct public input, were not significant components of this round of PBMA

S5: Time and resources

P5: Project coordinator

A5: Strong leadership

Considerable time and effort was required from participants in the process

There was not a dedicated office for the PBMA process. Literature suggests that this may be useful as a locus of accountability for follow-up

The Executive Leadership Team’s management of the process was widely praised

S2: Engagement

Respondents were not explicitly asked to comment on the continuity of membership in the management ranks; it was not raised by any informant as an issue

Respondents reported that they learned more about different parts of the organization and the role of the services offered there. Improved economic literacy is one example of broader learning reported in interview data O4: Improved health is achieved No informant could provide much by way of concrete details about long-term impacts observable at this relatively early stage

PBMA Program Budgeting and Marginal Analysis, PSRA priority setting and resource allocation These two elements are additionally discussed further in a separate manuscript [20]

a

quantitatively-based studies. Table 3 provides a general summary of the analysis; the text which follows illustrates in more detail how the data can help us understand why the PBMA roll-out at the IWK was received as it was. We present a range of quotes from our informants, to allow readers to see what interviewees said in their own words,

and so judge how we have synthesized original data into the themes shown. Putting the data itself in this central role is, we believe, key to the credibility of qualitative research findings [2]. Six elements are featured: engagement, time and resources, fair process, communication, followthrough, and long-term alignment.

A Qualitative Evaluation of Program Budgeting and Marginal Analysis

4.1 Engagement (Element S2) and Time/Resources (Element S5) The tension between encouraging wide engagement and managing time requirements again stands out in this second evaluation: ‘‘It was finding that balance of engagement and managing the process that I believe in the future we would do differently’’ (#10: Manager). Almost universally, respondents reported that the process had been burdensome in terms of time and effort: it was ‘‘labour intensive’’ (#16: Manager), ‘‘resource intensive’’ (#1: ELT), ‘‘time consuming’’ (#4: ELT), ‘‘a remarkably large amount of effort’’ (#17: Physician), and ‘‘a ton of work, a ton, a ton, a ton of work’’ (#29: Staff/public, WG member). ‘‘The workload was huge’’ (#19: Physician) and ‘‘the amount of information was overwhelming’’ (#20: Manager). However, it was also reported that broad engagement had been achieved. The open submission process was attributed a role in this; since anyone could give ideas without following ‘‘official channels,’’ the ‘‘breadth and depth’’ of participation was increased (#13: Manager, WG member). Many of these suggestions were ‘‘thoughtful’’ (#21: Manager, WG member); but negative aspects were also described. Some members contributed ‘mean-spirited’ suggestions (#26: Staff/public, WG member), or ideas about areas with which they were not familiar and so these had to be taken up and developed, or rebutted, by others. 4.2 Fair Process (Element P1) Overall, respondents assessed the process favorably and rated it an improvement over past practice. It was seen to better align with the organization’s Strategic Plan and longterm objectives. In terms of enforcement of and adherence to the process, the perceived failure of some departments to ‘‘play ball’’ was an issue about which there were still complaints, almost 2 years later. Some participants felt penalized (losing resources in disinvestments and not getting any back in reinvestment) because they took part in good faith. One informant noted a lack of sanction experienced by those who did not fulfill expectations of their participation—‘‘If you didn’t play, there were no real consequences for that’’ (#20: Manager). A number of respondents expressed the view that others tried to ‘‘game’’ the process by submitting unrealistic proposals: ‘‘Most stepped up and were realistic but a few just sent in foolish things that they knew would never be approved’’ (#4: ELT). One member of the Working Group emphasized that this issue was one of the huge challenges facing the process and had an interesting insight into how it was resolved: ‘‘The issue was brought up a few times. And when it was discussed, one of the conclusions was that, well, if we do this every year or every two years going forward, it would eventually even out’’ (#29: Staff/public, WG member).

565

4.3 Communication (Element P2) The second evaluation revealed a positive response to the initial efforts to disseminate information about and explain PBMA. The most common recurring concern was lack of explanation of the reasons for which proposals were accepted or not, expressed as the lack of a ‘‘feedback loop’’ (#27: Staff/public). Note how this is consistent with initial evaluation findings reported above. One of the key PBMA proponents presents this as ‘‘misunderstanding’’ of what the Executive intended by ‘‘transparency’’ (#10: Manager), which had not been intended to go to that level of detail. 4.4 Follow-Through (Element P4) A prominent theme in the second evaluation was the follow-up to and implementation of the PBMA recommendations. Unlike in the earlier evaluation, which occurred immediately following the completion of the process, participants now had the opportunity to assess what had or had not occurred over the months subsequent. Several respondents suggested a need for evaluation of the changes that were made to determine if they really worked out: ‘‘There needed to be greater focus on what an implementation of a proposal looked like…. And then evaluation mechanisms built in to that implementation to say, you know, did we get it right? There was some evaluation that was built in but from my perspective, it wasn’t rigorous enough’’ (#10: Manager). Not all of the changes appear to have gone through. ‘‘One of the applications that was approved in my portfolio was approved but it never happened’’ (#25: Staff/public, WG member). It was recognized that implementation might be an iterative process: ‘‘What did we cut that we ended up saying that was a mistake, and let’s learn from that. Because there are a number of things we took money out of and then discovered six months later, yeah, we couldn’t do that’’ (#18: Physician, WG member). 4.5 Long-term Alignment (Element A3) One of the benefits ascribed to PBMA is that it brought together ideas from different parts of the organization to allow ‘‘cross-pollination’’—proposals took a systems view (#12: Manager). When there is such interaction, one respondent suggested, it is harder for proposal proponents to advocate only for themselves, when they have been made aware of others’ needs (#3: ELT). PBMA proposals were problematic when they failed to identify all of the ways in which their implementation would have an effect upon other services within the organization. ‘‘[One] definite suggestion would be identifying a more effective mechanism for crossprogram or cross-team collaboration and impact analysis during the proposal development process’’ (#10: Manager).

566

This weakness could be one factor leading to failure of some approved PBMAs in implementation.

5 Discussion and Conclusions We may summarize the results of this evaluation as follows: Take this as the researchers’ overall impression of the strengths and weaknesses of PBMA as implemented at the IWK Health Centre. Respondents were generally positive about the PBMA process, and considered it an improvement over past practice. Key stakeholders internally and externally were considered to be supportive. Success in particular was attributed to effective leadership, and substantial efforts to engage staff members at all levels of the organization. Overall understanding of economic and ethical principles of decision making was reportedly increased. Communication at the front-end was seen to have been done well, and the resource allocation choices were perceived to align well with the organization’s strategic priorities. Some areas for improvement included insuring that everyone participated in good faith, better communication of final results, accounting for the impacts of proposals upon other organization services, and stronger follow-through to determine if the anticipated changes and benefits in fact occurred. The process was perceived to have been too demanding in terms of time and effort, and opportunities could be incorporated for partner and public input. Using PBMA made the resource allocation process more transparent, and gave senior leaders stronger grounds on which to defend or justify their decisions. Explicit criteria helped to ensure that resources were allocated to key strategic priorities. Participants trusted that the process was leading to better use of finite resources, but it may have been too early to reach any definitive conclusions on this point. If evaluation of PSRA is conducted at all, it tends to follow on the immediate conclusion of the process; there are few longitudinal studies which consider impacts again after a greater time has elapsed, so this is a useful contribution to the literature. The two evaluations carried out at the IWK report generally consistent views about the process and its outcomes. Time-pressed decision makers may take some comfort that an immediate brief follow-up evaluation (if well designed, and guided by a clear framework as to what constitutes success) should meet most of their quality improvement needs. We find the second evaluation perhaps contains a bit more negative comment; this should not be unexpected, since the passage of time gives greater perspective on whether or not initial hopes have come to fruition and expected benefits have manifested. Thus, the importance of follow-through (Element P4) emerges clearly in the second evaluation where it could hardly have done so at the time of the first evaluation.

N. Smith et al.

Historically, formal evaluations of healthcare PSRA efforts have been scarce [18]. Institutions with pediatric care mandates are poorly represented among these, with few exceptions [1, 21]. Of those evaluations which exist, many are ad hoc and assess success based on open-ended probing of participants about what worked and what did not. While valuable, this only captures what is on informants’ minds at the time, may miss relevant features, and is less likely to allow for systematic comparison across organizations. More structured frameworks have been proposed (e.g., [6, 10, 19]) but these have rarely been applied beyond their initial appearance in the literature. Several studies have used the ethical theory of Accountability for Reasonableness to organize evaluation (e.g., [8, 9], but this addresses only the idea of fair process. This research adds to the literature by providing a further test of a structure for evaluation, i.e., the high performance framework. We used the framework to categorize findings from these evaluation interviews. Across all interviews, most elements were spoken to at least once. We would not likely expect that every informant would speak to every one of the elements over the course of a relatively brief and semi-structured conversation. We now do have a more structured tool—a Resource Allocation Performance Assessment Tool—which allows for systematic collection of decision-maker views about each element, which provides an additional alternative means for evaluation [5]. Applying an established framework based on detailed knowledge of PSRA across a range of healthcare organizations should help us to better understand strengths, weaknesses, and gaps than a one-off evaluation, because it ensures we reflect upon the range of factors that have been shown to affect organizational performance. In addition to follow-through, as above, the high performance framework directs evaluative attention to such longer term and ongoing concerns as the coordination between priority setting and other management processes (Element S3), the breaking down of silos between departments (Element A3), and decision-maker skills and knowledge to make ethical and economically sound choices (Element O3). It also reminds us to consider the ultimate outcomes—improving quality of care, health, and well-being—to which healthcare organizations aspire (Element O4). The findings here also allow us to reconsider the way in which certain components of the framework were initially presented. For instance, this case study showed clearly how expectations about transparency in communication of decisions and rationales run up against important privacy considerations, where identifiable jobs might be at risk from a chosen disinvestment. Recommendations about communicating the rationale for all resource allocation decisions need to be tempered with this in mind. Such

A Qualitative Evaluation of Program Budgeting and Marginal Analysis

refinements to the high performance framework build upon the initial pilot test efforts [5]. 5.1 Limitations Both evaluations rely upon respondent self-reports. Participants may have had reason to emphasize the positive aspects of their experience, though they were asked at both points to explicitly comment upon what could be done better. Both evaluations, the second to a greater extent, focused upon members of the IWK who had leadership roles in implementing PBMA; we cannot be sure if their views are shared by middle managers and front-line staff who did the work to identify and build proposals—this would be an important perspective for assessing the prospects for PBMA’s sustainability in this context. The two evaluations had limited overlap in respondents. Those who participated in both described themselves in the second evaluation as moderate to strong advocates of the process. We cannot for confidentiality reasons directly compare their responses at the two times, to see if their views had changed. The limited degree of overlap should strengthen the conclusion that the perceived strengths and weaknesses are accurate, but we cannot say if personnel within the IWK have been talking and agreeing upon the ‘PBMA narrative’ in the time since its implementation. Ideally, future evaluators will be able to incorporate more objective measures in assessing the quality of priority setting and resource allocation processes. The magnitude of cutbacks required in the year of PBMA’s implementation (3 %) was less than in the previous non-PBMA year (7 %); it is possible that this—rather than the benefits of using a formal process per se—contributed to some feelings of satisfaction expressed by interviewees. The interviews did request participants to consider the process rather than the general fiscal climate, and the findings are consistent with those reported in other applications of PBMA which occur in a range of both disinvestment and investment contexts. The PBMA process at the IWK involved only the pediatric side of the Health Centre’s operations; the women’s health side was not included. Thus we do not know how well child and youth health programs would have fared if decision makers had been forced to compare and trade them off against maternal health programs. It is known that some departments have better evidence, greater public profile, or other power resources that can be deployed to obtain resources. This is addressed in PBMA through the use of explicit criteria and business cases which are standardized across all departments. Power differentials were not explicitly addressed by the evaluations, though participants were able to raise these issues if they perceived them to have significant impact upon the process. They did not emerge as a theme in this study. Due to the

567

length of time it takes to make changes to entrenched healthcare programs, even this later evaluation was perhaps too early to really grapple with the outcomes, rather than solely the process, of implementing PBMA. The high performance framework explicitly directs attention to outcomes from priority setting and resource allocation efforts, yet it can be difficult, even after the passage of 2 years, for respondents to point to particular tangible impacts of changes to management practice. 5.2 Conclusion A conceptual model, the high performance framework, proved to be useful for evaluation purposes at the IWK Health Centre. This case study strengthens our knowledge base with better grounded findings about practices which were reported here as conducive to effective priority setting and resource allocation. These findings should be of use to a broad range of health system decision makers who confront challenges which may be similar to those observed in this case. Acknowledgments The IWK Health Centre is an organization willing to invest in improvement, as shown by its implementation of PBMA as well as efforts to undertake two evaluations of its priority setting. We are grateful to them as their experiences and willingness to share these contribute to build our shared body of knowledge. This research was funded by the Canadian Institutes of Health Research, Evidence-informed Healthcare Renewal funding stream. The authors thank William Hall for helpful comments on an early draft of this manuscript. Compliance with Ethical Standards This study was funded by the Canadian Institutes of Health Research, as part of the Evidence Informed Health Care Renewal grant program. The authors—Neale Smith, Craig Mitton, Mary-Ann Hiltz, Matthew Campbell, Laura Dowling, J. Fergall Magee, and Shashi Ashok Gujar—each attest that no conflict of interest exists in relation to this paper. Author roles: MAH, SAG, MC, and FM, with the assistance of CM, participated in the IWK Health Centre PBMA project. The research study to evaluate this process was conceived by CM, MAH, MC and SAG. MAH and CM served as Co-Principal Investigators. LD conducted the interviews. NS, with the assistance of LD and SAG, conducted the data analysis. All team members reviewed and approved the findings. NS prepared the initial draft of the manuscript. All authors reviewed the manuscript, provided feedback and approved the final version of the manuscript. The opinions expressed in this paper are solely those of the authors and should not necessarily be considered the view of the IWK Health Centre.

References 1. Astley J, Wake-Dyster W. Evidence-based priority setting. Aust Health Rev. 2001;24(2):32–9. 2. Chenail RJ. Presenting qualitative data. The Qualitative Report. 1995;2(3). http://www.nova.edu/ssss/QR/QR2-3/presenting.html. 3. Cromwell I, Peacock SJ, Mitton C. ‘Real-world’ health care priority setting using explicit decision criteria: a systematic

568

4. 5.

6.

7.

8.

9.

10.

11.

12.

N. Smith et al. review of the literature. BMC Health Serv Res. 2015;15(164). doi:10.1186/s12913-015-0814-3. Donaldson C, Bate A, Mitton C, Dionne F, Ruta D. Rational disinvestment. QJMed. 2010;. doi:10.1093/qjmed/hcq086. Hall W, Smith N, Mitton C, Gibson JL, Bryan S. An evaluation tool for assessing performance in priority setting and resource allocation: multi-site application to identify strengths and weaknesses. J Health Serv Res Policy. 2016;21(1):15–23. Kapiriri L, Martin DK. Successful priority setting in low and middle income countries: a framework for evaluation. Health Care Anal. 2009;18:129–47. King N. Using templates in the thematic analysis of text. In: Cassell C, Symon G, editors. Essential guide to qualitative methods in organizational research. London: Sage; 2004. p. 256–70. Maluka S, Kamuzora P, SanSebastian M, Byskov J, Ndawi B, Olsen OE, Hurtig A-K. Implementing accountability for reasonableness framework at district level in Tanzania: a realist evaluation. Implement Sci. 2011;6(11). doi:10.1186/1748-5908-6-11. Martin DK, Giacomini M, Singer PA. Fairness, accountability for reasonableness, and the views of priority setting decision-makers. Health Policy. 2002;61(3):279–90. McCafferty S, Williams I, Hunter D, Robinson S, Donaldson C, Bate A. Implementing world class commissioning competencies. J Health Serv Res Policy. 2012;17(Suppl 1):40–8. Mitton C, Dionne F, Donaldson C. Managing healthcare budgets in time of austerity: the role of Program Budgeting and Marginal Analysis. Appl Health Econ Health Policy. 2014;12:95–102. Mitton C, Donaldson C. Priority setting toolkit: a guide to the use of economics in healthcare decision making. London: BMJ Publishing Group; 2004.

13. Mitton C, Donaldson C. Setting priorities and allocating resources in health regions: lessons from a project evaluating program budgeting and marginal analysis. Health Policy. 2003;64:335–48. 14. Peacock S, Mitton C, Ruta D, Donaldson C, Bate A, Hedden L. Priority setting in healthcare: towards guidelines for the program budgeting and marginal analysis framework. Expert Rev Pharmacoecon Outcomes Res. 2010;10(5):539–52. 15. Pope C, Ziebland S, Mays N. Qualitative research in health care: analyzing qualitative data. BMJ. 2000;320:114–6. 16. Sabik LM, Lie RK. Priority-setting in health care: lessons from the experiences of eight countries. Int J Equity Health. 2008;7(4). doi:10.1186/1475-9276-7-4. 17. Sandelowski M. Real qualitative researchers do not count: the use of numbers in qualitative research. Res Nurs Health. 2001;24:230–40. 18. Sibbald SL, Gibson JL, Singer PA, Upshur R, Martin DK. Evaluating priority setting success in healthcare: a pilot study. BMC Health Serv Res. 2010;10(131). doi:10.1186/1472-6963/10/ 131. 19. Sibbald SL, Singer PA, Upshur R, Martin DK. Priority setting: what constitutes success? A conceptual framework for successful priority setting. BMC Health Serv Res. 2009;9(43). doi:10.1186/ 1472-6963-9-43. 20. Smith N, Mitton C, Dowling L, Hiltz M-A, Campbell M, Gujar SA. Introducing new priority setting and resource allocation processes in a Canadian healthcare organization: a case study analysis informed by multiple streams theory. Int J Health Policy Manag. 2016;5(1):23–31. 21. Teng F, Mitton C, MacKenzie J. Priority setting in the provincial health services authority: Survey of key decision makers. BMC Health Serv Res. 2007;7(84). doi:10.1186/1472-6963-7-84.