Keywords

Pharmacy admissions; GPA; PCAT; Multiple miniinterview (MMI); Professional development ratings; Success predictors

Introduction

It is vital to society to have pharmacists who excel in their practice. Also, the cost and effort required to train Pharm.D. students have increased substantially as the role of the profession in medication therapy management has become more widely appreciated. As with other health professions, Pharm.D. programs have moved more to holistic admissions evaluations in an attempt to capture both cognitive and non-cognitive skills necessary for professional success. Elsewhere, we have reviewed the manner in which health care professions including pharmacy have attempted to meet the goal of admitting students who will be successful practitioners [1].

It is well recognized that traditional academic markers such as GPA and achievement test scores can predict success in the classroom, but not in experiential training or clinical practice [2-4]. In contrast, systematic ratings of professional development and innovative interview methods such as the “admissions OSCE” (Objective Structured Clinical Exams)- Multiple Mini-Interview (MMI) do seem well suited to predict performance in experiential training [5]. However, there is a marked disparity in the difficulty in obtaining and evaluating cognitive measures such as GPA and PCAT–readily available from the application and the MMI, which must be implemented on site to individual applicants. Thus, 2-step admissions processes, consisting of an initial screening using information gleaned from the application (step 1) followed by a more in depth, on-site interview process (step 2) [1].

At our institution, we have utilized holistic admissions criteria for a number of years, using such a 2-step process. Initially, applicants are screened for GPA and PCAT scores and rated on professional development using a 9-item scale (RATE). Subsequently, those applicants considered to be a potentially good fit were historically invited for standard interviews or, most recently, for multiple-mini-interviews (MMI). The present retrospective analysis determined the contribution of cognitive and non-cognitive measures to the eventual decision to admit the student to the Pharm.D. program.

Methods

As highlighted above, the admissions committee at our institution has used a 2-step admissions procedure for several years as part of a holistic review. Initially, pre-requisite GPA and PCAT scores are used for the cognitive measures. Non-cognitive information is gleaned from the full application, including letters of recommendation and summarized (RATE) via a 9-item scale with equal ratings for experience in pharmacy practice, teaching, research, student organizations, community service, overcoming hardship, essay quality, plus information from two letters of recommendation (Table 1). GPA, composite PCAT score and RATE are converted to Z-scores and the arithmetic mean of GPA, PCAT and RATE calculated-thus weighting cognitive measures twice as heavily as non-cognitive measures. This metric provides a guide to reviewers as they determine from examination of the complete application those individuals who will be invited for interviews. Following the interview process, the MMI score is determined and its Z-score calculated. Then, the average for GPA, PCAT, RATE and MMI is determined-weighing cognitive and non-cognitive measures equally and presented to the admissions committee for a holistic evaluation.

Table 1: Professional development rating (Rate)

Following final review by the admissions committee, offers in 2015 were extended to 150 applicants, a position on the wait list to 36 applicants, and deny to 49 applicants. Offers in 2016 were extended to 129 applicants, a position on the wait list to 28 applicants, and deny to 50 applicants.

Since 2015, the College has used for the interview process the MMI (8 stations with scenarios). The scenarios represent attributes selected by the College faculty members as being important to success in a Pharm.D. program and in pharmacy practice. The attributes are: 1) problem solving/critical thinking/ intellectual capacity; 2) compassion/commitment to care; 3) respect towards others; 4) integrity/ethical reasoning; 5) communication skills; 6) organizational efficiency; 7) relationship-building skills; 8) motivation to become a pharmacist; and 9) leadership skills.

Each scenario was scored for 3 or more attributes and all attributes were represented in the set of scenarios, some several times. Attributes within scenarios were scored from 1–10 via a rating scale with a grading rubric and labels for scores (1-2, poor; 3-4, fair; 5-6, good; 7-8, very good; 9-10, excellent). Three of the scenarios were interactive between an actor and the applicant or between two applicants. Although there were eight active scenarios, stations 7 and 8, involving model toy construction, were each scored by two evaluators, yielding a total of 10 evaluator scores. In the 2014-2015 cycle, the number of attributes assessed yielded a total MMI score of 300. In the 2015-2016 admissions cycle the number of attributes for the new set of station queries yielded a total MMI score of 330.

Scenarios were initially obtained from a set developed by ProFitHR and then adapted for the US health care system and pharmacy by a scenario implementation team that was part of the MMI task force. This task force met periodically for approximately three years to assist the College faculty members in evaluating Pharm D attributes and then assessing the background of the MMI literature, logistics, scenarios, resources, and data analysis. Scenarios used in 2015 were: 1) heart attack, 2) difficult patient, 3) beneficial activities for pharmacy students, 4) accidental death, x) rest station (not scored), 5) medical management team, 6) accidental vaccination (with actor), 7) model toy, construct according to instructions from another applicant, and 8) model toy, provide instructions for another applicant to make the toy. Stations 7 and 8 were each scored by two evaluators. Scenarios information for 2016 included: 1) reiki (healing touch), 2) delusions and treatment compliance, 3) pharmacy school advice, 4) conflict resolution, 5) toy received, 6) toy sender, 7) brain damage, 8) test preparation, x) rest station (not scored). Data were analyzed by logistic regression.

Results

Descriptive Information: All admissions committees have the difficult task of making the optimal match between a set of applicants and the academic program to which these individuals apply. For competitive programs, there is an additional level of difficulty for the admissions committee, namely that the majority of the applicants are quite strong so that the matching process becomes, perhaps, even more of a challenge. Below, we summarize the information on cognitive (GPA and PCAT) plus non-cognitive (RATE and MMI) for the applicants in the 2015 and 2016 admissions cycles. Overall, these represented strong applicant pools with considerable overlap between those students who were given offers and those who did not receive an offer.

Figure 1 shows data on science GPA, PCAT composite percentile, professional development rating (RATE), and MMI score for the 2014-2015 and 2015-2016 admissions cycles. Tables 2 and 3 show admissions variable data for individual applicants in the 2014-2015 and 2015-2016 admission cycles respectively.

Table 2: Admissions variables for 2014-2015 admissions cycle.

Table 3: Admissions variables for the 2015-2016 admissions cycle.

Figure 1A and 1B: Panels A-B show the applicant data for the 2014-2015 v 2015-2016 admissions cycles as upper and lower portions of the panel, respectively. (A) Science GPA (B) PCAT composite percentile.

Figure 1C and 1D: Panels C-D show the applicant data for the 2014-2015 v 2015-2016 admissions cycles as upper and lower portions of the panel, respectively. (C) Professional Development Rating scored from 1-5 with 5 maximum, (D) MMI score with a maximum of 300 for 2014-2015 cycle and 330 for 2015-2016 cycle.

It is evident that overall the applicant pools for both cycles were strong academically:

Science GPA was roughly 3.5 for applicants given offers and 3.2 for those not given offers;

PCAT composite percentile was approximately 85% for applicants given offers and 73% for those not given offers.

The tables also show that there was more separation between applicants given offers and those not given offers on the noncognitive measures:

Professional development rating (RATE) was 3.2 (out of 5) for applicants given offers and 2.6 for those not given offers.

MMI scores were higher in applicants given offers (210 out of 300 for 2014-2015 and 266 out of 330 for 2015-2016) versus applicants not given offers (171 out of 300 for 2014-2015 and 231 out of 330 for 2015-2016).

Logistic regression

The major focus of the present paper was to determine the extent to which adding a non-cognitive admissions measure (MMI) as a second step in a 2-step admissions process improved the prediction of admissions decisions. Here, step-1 consisted of a combination of 2 cognitive measures (science GPA and the PCAT composite percentile score) plus 1 non-cognitive measure (RATE, representing an arithmetic average of scores in 9-areas of professional development).

Tables 4 and 5 present the results of the logistic regression analyses for the 2014-2015 and 2015-2016 admission cycles, respectively. Crucial to the interpretation of logistic regression is the value of the odds ratio, which is the exponential of B (exp(B)). This is an indicator of the change in odds resulting from a unit change in the predictor [6]. If the value is greater than 1, then it indicates that as the predictor increases, the odds of the outcome occurring increase. Conversely, a value less than 1 indicates that as the predictor increases, the odds of the outcome occurring decrease.

Table 4: 2014-2015 Admissions Cycle. Logistic Regression Analysis*.

Table 5: 2015-2016 Admissions Cycle. Logistic Regression Analysis*.

For applicants in the 2014-2015 admissions cycle, unit increases in professional development rating (RATE) yielded a 5.99-fold increase in the likelihood of being given an offer and for MMI score, a 13.26-fold increase. For applicants in the 2015-2016 admissions cycle, unit increases in professional development rating (RATE) yielded a 1.61-fold increase (not significant) in the likelihood of being given an offer and for MMI score, a 13.23-fold increase. Thus, both cognitive and noncognitive measures contributed significantly to the likelihood of an applicant being given an offer. However, it should be noted that all predictors were significant in the 2014-2015 admissions cycle, but science GPA and professional development ratings were not significant in the 2015-2016 admissions cycle.

Discussion

From its inception in 2004 to the present, the MMI has been recognized as a tool with great potential for evaluating non-cognitive attributes in health professions [7,8]. This metric has been used to predict performance of applicants in a variety of professional programs [9-28]. The MMI has been shown to be cost-effective [29], flexible in terms of format [16-18,28-35] and independent of personality and emotional intelligence measures [36-39]. The present results have extended this work to a systematic evaluation of the relationship among cognitive (GPA and PCAT) and non-cognitive (professional development ratings, RATE and MMI) to predict admission and to discern the interrelationships among the predictor variables for a 2-step admissions process.

For the two admissions cycles studied, we found that professional development rating (RATE) and MMI score contributed significantly to the likelihood of the applicant receiving an offer. For the 2014-2015 cycle, science GPA and PCAT composite percentile score also contributed meaningfully to holistically derived offers. We speculate that this may be due to the fact that the 2014-2015 applicant pool was particularly strong in both GPA (3.54 for offer) and PCAT (87.4% for offer), whereas the 2015-2016 applicant pool was slightly less strong in these cognitive measures (3.41 GPA and 84.5% for offers, respectively). In any case, the substantial overlap between sets of applicants receiving and not receiving offers but with invitations to interview, would tend to minimize the statistical significance of these two variables.

The incorporation of the MMI into a 2-step admissions process (initial screen via GPA, PCAT and RATE) followed by interview (MMI) was highly effective for both admission cycles studied. These findings are consistent with and extend the current literature [9-12]. Furthermore, the MMI was well behaved in terms of consistency across evaluators and stations (scenarios), suggesting that the MMI was a valid measure in this study. These results are also consonant with findings in the field [11-40]. Our results are consistent with, and extend the current literature on the use of the MMI as a predictor of admission into health professions programs [5,7-9,12-14,20-43]. Significantly, our results demonstrate the potential value of professional development ratings (RATE) as important “pre-screening” tools for interview invitations using the MMI. As expected, based on the strong applicant ratings of professional development (RATE) in step 1 of the admissions process, most of the applicants performed well on the MMI. In fact, the vast majority of the applicants performed consistently on the MMI with a very few performing in an especially robust manner and a few performing poorly.

Our analyses demonstrated that there are significant differences in each of the admissions variables of GPA, PCAT, RATE and MMI between the offer-status groups. The fact that all four admission-predictor variables (GPA, PCAT, RATE and MMI) contributed significantly to an offer of admission is consistent with the hypothesis that academic and professional development measurements are useful in informing the admissions committee. In particular, our findings with both RATE and MMI extend the published literature via the explicit incorporation of professional development ratings and the demonstration of the relationship between RATE and MMI score [5-7,14]. Our results illustrate the importance of the possible independent predictive value of professional development assessments and their value as a screening tool for invitations to the MMI. Thus, our results appear consistent with the literature that the MMI assesses a different dimension than do the didactic admission measures but extend the literature in showing the overlap of professional development ratings with other measures [24,25,33,44].

Acknowledgement

The authors gratefully acknowledge the extraordinary efforts of the Pharm.D. Student Interview Team Chairs: Angelica Cuenco, Isaac Perales, Lillian Tran, & Victoria Vaughn in making the MMI possible.

References

1. Wilcox RE, Lawson KA (2018) Predicting performance in health professions education programs from admissions information–Comparisons of other health professions with pharmacy. Currents in Pharmacy Teaching and Learning.
2. Kuncel NR, Hezlett SA, Ones DS (2001) A comprehensive meta-analysis of the predictive validity of the graduate record examinations: implications for graduate student selection and performance. Psychological bulletin 127: 162.
3. Kuncel NR, Credé M, Thomas LL, Klieger DM, Seiler SN, et al. (2005) A meta-analysis of the validity of the Pharmacy College Admission Test (PCAT) and grade predictors of pharmacy student performance. Am J Pharm Educ 69: 51.
4. Kuncel N, Hezlett SA (2007) Standardized tests predict graduate students' success. Science.
5. Eva KW, Rosenfeld J, Reiter HI, Norman GR (2004) An admissions OSCE: the multiple mini-interview. Med Educ 38: 314-326.
6. Field A (2013) Discovering statistics using IBM SPSS statistics. sage.
7. Eva KW, Reiter HI, Rosenfeld J, Norman GR (2004) The ability of the multiple mini-interview to predict preclerkship performance in medical school. Acad Med 79: S40-S42.
8. Eva KW, Reiter HI, Rosenfeld J, Norman GR (2004) The relationship between interviewers’ characteristics and ratings assigned during a multiple mini-interview. Acad Med 79: 602-609.
9. Barbour ME and, Sandy JR (2014) Multiple mini interviews for selection of dental students: influence of gender and starting station. J Dent Educ 78: 589-596.
10. Cameron AJ, MacKeigan LD (2012) Development and pilot testing of a multiple mini-interview for admission to a pharmacy degree program. Am J Pharm Educ76: 10.
11. Cameron AJ, MacKeigan LD (2012) Development and pilot testing of a multiple mini-interview for admission to a pharmacy degree program. Am J Pharm Educ 76: 10.
12. Dowell J, Lynch B, Till H, Kumwenda B, Husbands A (2012) The multiple mini-interview in the UK context: 3 years of experience at Dundee. Med Teach 34: 297-304.
13. Eva KW, Reiter HI, Rosenfeld J, Trinh K, Wood TJ, et al. (2012) Association between a medical school admission process using the multiple mini-interview and national licensing examination scores. Jama 308: 2233-2240.
14. Eva KW, Reiter HI, Trinh K, Wasi P, Rosenfeld J, et al. (2009) Predictive validity of the multiple mini-interview for selecting medical trainees. Med Educ 43: 767-775.
15. Finlayson HC, Townson AF (2011) Resident selection for a physical medicine and rehabilitation program: feasibility and reliability of the multiple mini-interview. Am J  Phys Med Rehabil 90: 330-335.
16. Harris S, Owen C (2007) Discerning quality: using the multiple mini‐interview in student selection for the Australian National University Medical School. Med Educ 41: 234-241.
17. Hecker K, Donnon T, Fuentealba C, Hall D, Illanes O, et al. (2009) Assessment of applicants to the veterinary curriculum using a multiple mini-interview method. J Vet Med Educ 36: 166-173.
18. Hecker K, Violato C (2011) A generalizability analysis of a veterinary school multiple mini interview: effect of number of interviewers, type of interviewers, and number of stations. Teach Learn Med 23: 331-336.
19. Humphrey S, Dowson S, Wall D, Diwakar V, Goodyear HM (2008) Multiple mini-interviews: opinions of candidates and interviewers. Med Educ 42: 207-213.
20. Jones PE, Forister JG (2011) A comparison of behavioral and multiple mini-interview formats in physician assistant program admissions. J Physician Assist Educ 22: 36-40
21. Lemay JF, Lockyer JM, Collin VT, Brownell AKW (2007) Assessment of non-cognitive traits through the admissions multiple mini-interview. Med Educ 41: 573-579.
22. McAndrew R, Ellis J (2012) An evaluation of the multiple mini-interview as a selection tool for dental students. Br Dent J 212: 331.
23. O’Brien A, Harvey J, Shannon M, Lewis K, Valencia O (2011) A comparison of multiple mini-interviews and structured interviews in a UK setting. Med Teach 33: 397-402.
24. Perkins A, Burton L, Dray B, Elcock K (2013) Evaluation of a multiple-mini-interview protocol used as a selection tool for entry to an undergraduate nursing programme. Nurse Educ Today 33: 465-469.
25. Razack S, Faremo S, Drolet F, Snell L, Wiseman J, et al. (2009) Multiple mini‐interviews versus traditional interviews: stakeholder acceptability comparison. Med Educ 43: 993-1000.
26. Reiter HI, Eva KW (2005) Applied research: reflecting the relative values of community, faculty, and students in the admissions tools of medical school. Teach  Learn Med 17: 4-8.
27. Reiter HI, Eva KW, Rosenfeld J, Norman GR (2007) Multiple mini‐interviews predict clerkship and licensing examination performance. Med Educ 41: 378-384.
28. Till H, Myford C, Dowell J (2013) Improving student selection using multiple mini-interviews with multifaceted Rasch modeling. Acad Med 88: 216-223.
29. Rosenfeld JM, Reiter HI, Trinh K, Eva KW (2008) A cost efficiency comparison between the multiple mini-interview and traditional admissions interviews. Adv Health Sci Educ 13: 43-58.
30. Dodson M, Crotty B, Prideaux D, Carne R, Ward A, et al. (2009) The multiple mini‐interview: how long is long enough?. Med Educ 43: 168-174.
31. Kumar K, Roberts C, Rothnie I, Du Fresne C, Walton M (2009) Experiences of the multiple mini‐interview: a qualitative analysis. Med Educ 43: 360-367.
32. Roberts C, Rothnie I, Zoanetti N, Crossley J (2010) Should candidate scores be adjusted for interviewer stringency or leniency in the multiple mini-interview?. Medical Educ 44: 690-698.
33. Roberts C, Walton M, Rothnie I, Crossley J, Lyon P, et al. (2008) Factors affecting the utility of the multiple mini-interview in selecting candidates for graduate-entry medical school. Med Educ 42: 396-404.
34. Sebok SS, Luu K, Klinger DA (2014) Psychometric properties of the multiple mini-interview used for medical admissions: findings from generalizability and Rasch analyses. Adv Health Sci Educ 19: 71-84.
35. Uijtdehaage S, Parker N (2011) Enhancing the reliability of the multiple mini-interview for selecting prospective health care leaders. Acad Med 86: 1032-1039.
36. Griffin B, Harding DW, Wilson IG, Yeomans ND (2008) Does practice make perfect? The effect of coaching and retesting on selection tests used for admission to an Australian medical school. Med J Aust 189: 270-3.
37. Jerant A, Griffin E, Rainwater J, Henderson M, Sousa F, et al. (2012) Does applicant personality influence multiple mini-interview performance and medical school acceptance offers?. Acad Med 87: 1250-1259.
38. Kulasegaram K, Reiter HI, Wiesner W, Hackett RD, Norman GR (2010) Non-association between Neo-5 personality tests and multiple mini-interview. Adv Health Sci Educ 15: 415-423.
39. Yen W, Hovey R, Hodwitz K, Zhang S (2011) An exploration of the relationship between emotional intelligence (EI) and the Multiple Mini-Interview (MMI). Adv Health Sci Educ 16: 59-67.
40. Dore KL, Hanson M, Reiter H I, Blanchard M, Deeth K, et al. (2006) Medical school admissions: enhancing the reliability and validity of an autobiographical screening tool. Acad Med 81: S70-S73.
41. Brownell K, Lockyer J, Collin T, Lemay JF (2007) Introduction of the multiple mini interview into the admissions process at the University of Calgary: acceptability and feasibility. Medi Teach 29: 394-396.
42. Hofmeister M, Lockyer J, Crutcher R (2008) The acceptability of the multiple mini interview for resident selection. Fam Med 40: 734-740.
43. Tiller D, O'mara D, Rothnie I, Dunn S, Lee L, et al. (2013) Internet‐based multiple mini‐interviews for candidate selection for graduate entry programmes. Medi Educ 47: 801-810.
44. Roberts C, Clark T, Burgess A, Frommer M, Grant M, et al. (2014) The validity of a behavioural multiple-mini-interview within an assessment centre for selection into specialty training. BMC Med Educ 14: 169.