Reporting From the 2006 AAMC Annual Meeting:
“Pursuing Excellence, Creating Value”
October 27-November 1, 2006 Seattle WA

Hot Topics:
CME Reform, Achieving Diversity in Faculty and Students; Physician Workforce, Dual Degrees, Academies, Fellowships and Fostering Educational Scholarship

See full program and presenter contact information at the AAMC Website:
http://www.aamc.org/meetings/annual/2006/start.htm

Reports from the following sessions:

1. AAMC President's Address : In Search of the Public Good
2. Keynote: Jim Collins, "Good To Great"
3. MCAT Update
4. Physician Workforce Expansion and Diversity: How Do We Make This Work?
5. The Future of Allopathic Medicine
6. LCME Hearing on proposed New Standards
7. Workforce session on Physicians
8. Dual Degrees: Opportunities for Students, Challenges for Administrators
9. NBME/USMLE/FSMB Update
10. Expanding Our Horizons: Future Trends in Alumni Affairs
11. Workshop: Developing High-Quality Multiple Choice Tests to Assess Application of Basic Science Knowledge Using Patient Vignettes
12. Assessment Across the Continuum: Focus on Self-Assessment
13. Implementing a Vision for Medical Education in the United States: GEA Report for AAMC and IIME
14. Moving Educational Activities into Scholarship: Results and Recommendation from the AAMC-GEA Consensus Conference on Educational Scholarship
15. Organizational Infrastructure to Support Scholarship in Education
16. Planning for Class Size Increases
17. The Under-performing Medical Student: How to Identify and Address Learning and Emotional Difficulties in the Preclinical and Clinical Years
18. Formative and Summative Computerized Assessments in Medical Education
19. Academies Collaborative Annual Meeting
20. First Annual Meeting of Directors of Medical Education Fellowships
21. Searching for Diversity: Conducting Successful Searches
22. Reforming CME: Whose Responsibility is It?
23. Research Paper Presentations: Measuring Clinical Skills
24. Addressing Medical Student Professionalism

Thanks to the following for contributing reports:

• Linda Heun, PhD
• John R. Gimpel, DO, MEd
• Steve Shannon, DO, MPH
• Barbara M. Kriz, PhD
• Glenn Davis, MS

1. AAMC President's Address: In Search of the Public Good
Darrell G. Kirch, M.D. President, AAMC
October 29, 2006

Kirch challenged the profession to recapture the focus on the public good. In a moving and purposeful message he suggested that academic medicine was at the ‘epicenter of higher education, research and health care'. He asked if we were ready to have a conversation about priorities, to put the public good above our institutions and our own individual good. He suggested that:

• On the societal level, we need to take responsibility for the legacy we're leaving
• On the political level, we need a rhetoric-free zone to focus on the public good
• On the medical education level, we need to focus on earmarks that will advance the public good, including coming to grips with students' high tuition and debt and advancing research that focuses on patient benefits and equity
• On the personal level, we need to find our own role in the advancement of the public good

He called for a radical change in perspective and quoted General Eric Shinseki as follows: “If you don't like change, you're really going to dislike becoming irrelevant.”

Reported by:
Linda Heun, Ph.D. <meded@aacom.org>
Vice President for Medical Education
American Association of Colleges of Osteopathic Medicine

back to top

2. Keynote Address: Good to Great

Jim Collins, author of "Built to Last" and "Good to Great,"
October 29, 2006

Framing himself and his research colleagues as ‘data geeks', Jim Collins indicated that his latest efforts were directed toward understanding how good organizations became and stayed ‘great' in the context of turbulence and disruption over which they had no control. It is his overall conclusion is that identifying the turbulence and disruption doesn't predict anything or cause anything; rather it amplifies and exposures strengths and weaknesses in organizations. Therefore, it's not what an organization does in the face of disruption but what it had in place before the disruption.

While acknowledging that he was not an expert in medical education, he firmly indicated that using his research findings was not a matter of becoming more like a business. He urges the creation of a “Culture of Discipline” wherein people engage in disciplined thought and take disciplined action. He then reiterated the basic principles developed in his book, Good to Great. Key ideas as applied to medical education were:

• Begin with the ‘who' not the ‘what'. Medical education systems create the ‘who' for the practice of medicine. He suggested that the choices can no longer allow the ‘who' to be the best of the wealthy.
• Move to level five leadership, which is not charismatic, but rather humility and the stoicism to do whatever it takes. Level five deans don't manage faculty.
• Practice disciplined thinking which faces the brutal facts, keeping faith that you will succeed while facing those facts. ‘You don't sell a vision, you use an Excel spreadsheet'. In business money is both the input and the output/measure of success. In areas of social good money is the input but NOT an output - we must define relevant outputs which are measures of success. In business the key concept that drives the organization is the intersection of what you're passionate about, best at, and drives the economic engine. In the area of social good the key driver is the reputation of the organization – when reputation is lost, the flywheel stops.
• Avoid the disease of mediocrity – ‘the fall from Great to Good is self inflicted' and not because an organization isn't willing to change but because it changes the wrong things or is erratic/chronically inconsistent. Greatness is based on a discipline of values wherein no one has a ‘job', each has ‘responsibilities'. The data indicates that it takes seven years to show breakthrough; the challenge is to avoid erratic behavior when the average tenure of a medical school dean is 2 ½ years.

He suggested that the earmarks of great organizations are their outputs are related to the mission, and a distinctive impact on the community they serve. The earmarks of the ‘right people' were:

1. an apriori fit with organizational values,
2. don't need to be tightly managed,
3. understand they don't have a job, they have responsibilities,
4. have 100% track record of doing what they say they will do,
5. they ‘window' success (attribute it to others) and ‘mirror' failure (look to themselves) and
6. they get a kick out of the cause you're involved in.

Key answers in Q&A session

• If you put something on a to do list, must take something off
• Rank your priority list
• A “once in a lifetime opportunity” is a fact, not a mandate
• Set up a council of wise elders who use a high questions to statements ratio
• It's the choices that no one can see that count

See www.jimcollins.com for further thoughts and updates.

Reported by:
Linda Heun, Ph.D. <meded@aacom.org>
Vice President for Medical Education
American Association of Colleges of Osteopathic Medicine

back to top

3. MCAT Update

Speaker: Ellen R. Julian, Ph.D., AAMC

Students are taking the MCAT in increasing numbers; almost twice the number of takers annually since the mid 1980s. And there are larger numbers of students who now score higher in the MCAT than ever before, including generally increased scores in the physical sciences section, the biological sciences section, and in the verbal reasoning section (though the latter may have been affected by some internal changes with the examination). Some of the increase in the number of examinees may have been the final rush to take the paper-and-pencil MCAT before it was fully transitioned to the computer-based testing (CBT) format.

MCAT notes numerous advantages to their new CBT format, including quicker score reporting (30 days now, may be even shorter in future), more test administration dates (from 2 annually to 19 now with CBT), nicer testing environments, a shorter testing day (5.5 hours now), and biometric identification. The CBT format features 33% fewer questions, 2 essay questions, 30% less time overall (breaks are optional), and equivalent content coverage to the paper-and-pencil MCAT. One of the essays is hand-graded, and the second is computer graded, with a third human rater used if there is a wide variation.

Testing dates are clustered in April and May, with additional dates in January, July and August, and will be adjusted annually based on presumed need. Retakers can take the computer-based MCAT three times annually, with no lifetime limit. Tests are delivered at Prometric Centers internationally (same as NBOME COMLEX-USA and USMLE exams), and MCAT is looking to deliver examinations at future Prometric Centers based on university campuses. Registration and scheduling are done via a new dynamic online system. Students can also take practice MCATs online, and schools can even work with MCAT to study performance patterns of their own students.

The MCAT reports that the final judgment in Turner et al v. AAMC is currently pending.

In this case regarding ADA accommodations for testing, the judge did state that “flagging” (annotating on the official score report and transcript that an examinee took an examination with special accommodations) is acceptable, and even recommended, but that he would not rule on whether the definition of “disabled” was defined as a substantial limitation with respect to his/her peer reference group or as compared to the general population. The judge in this case also stated that California State law regarding disability must be followed in cases arising in California. AAMC has found that students who receive accommodations on the MCAT do improve their scores, but that they also “frequently underperform in medical school”.

MCAT is field testing a new component to the examination that uses video vignettes of patient scenarios to test the communication and interpersonal skills of examinees. Further information on the MCAT can be found at the website.

Reported by:
John R. Gimpel, D.O., M.Ed.
Vice President for Clinical Skills Testing
National Board of Osteopathic Medical Examiners

back to top

4. Physician Workforce Expansion and Diversity: How Do We Make This Work?

Moderator: Cynthia E. Boyd, MD, MBA
Speakers: Roberto Gomez, MD • Luann Wilkerson, EdD • Michelle Ko, MD • Joan Y. Reed, MD, MPH, MS • Gabrielle Garcia, MD

G. Garcia, M.D., Stanford:

There has been a call to increase the physician workforce by 30% by 2015, and with this, a desire/obligation to increase diversity.

While underrepresented minorities (URM) are over 17% of population, they are only 10% of medical students. Problem is getting bachelor's degree and that's where most of the focus needs to be.

Institutions need to decide what is the best student for the school and then have support from the top, and a commitment from the admissions committee: “We shall admit a diverse group of students.”

The admissions process currently favors numerical data and moves away from qualitative data – aptitude, attitude. Need relationship with and feedback from pre-health advisors to identify candidates. Need to assess the “distance traveled” by the applicant. Consider the educational context – who they are, support they had, did they work, family circumstances, quality of curriculum, activities, service. Educate admissions committee members to understand what “distance traveled” is, what it means.

J. Reede, M.D., Harvard:

Pipeline is a huge problem. It is a long haul and has to start with K-12. Need some interim successes – find something small that works.

Collaboration is key, support within university and from community leaders. Examples of programs:

• bring middle schoolers in to shadow faculty in clinics and labs; “explorations” program;
• after-school and summer camps. Use arts to talk about health of individual and community;
• posters in schools; billboards in subway; newspaper, web ads;
• AP biology scholars; supplement their programs; bring teachers to campus to train them in labs; student programs as well;
• Biomedical science careers program (BSCP); funded by community, use volunteers; provide skills workshops; industry provides scholarships, internships; no public funds;
• New England science symposium – 56% are URM
• Visiting clerkship program
• Fellowships leading to MPH or MBA

L. Wilkerson, Ed.D., UCLA:

Is diversity a compelling interest from a legal point of view?

Hopwood vs. Texas. Does building diversity produce benefits to all the students and the community? Evidence is building in support. With exposure to diverse student body, students increase their ability to think in complex ways, have increased satisfaction with their education, changes the way they conduct themselves socially – but these are studies in higher education. What about in professional education? Call to continue the research and collect the data…

One study suggests students support affirmative action. UCLA did survey to study value of diverse student body to the students. They said that this was very positive, both through classroom experiences and through informal associations. They had increased chances and got more involved in volunteer activities.

Used graduate survey to study self-perceived cultural competency, attitudes toward social justice, and looked at their plans to practice in underserved areas. Data obtained from AAMC graduate questionnaire (GQ) and selected 20 relevant questions; 120 med schools, 19,000 surveys. Didn't include historical black colleges.

18% over 30, 47% female, 12% URM, 21% other minority.

22% reported inadequate curriculum in cultural competency, concentration on healthcare issues for underserved populations, cultural differences, culturally appropriate care for a diverse population.

Avg. number of diversity-related (elective) activities: 1.69 per student. Not much time for it.

Students agree, but not strongly, that opportunity for interaction is available and encouraged. Same for whether access is a major problem. They report high levels of cultural competency, but only 21% plan to practice in an underserved area.

There were better outcomes in schools with demographic diversity and more opportunity to interact. The curriculum appears to be less important that the opportunity to work in interactive groups.

M. Ko, M.D., UCLA:

a graduate of and reporting on Drew/UCLA Medical Education Program.

Admissions: separate application; demonstrated commitment to work in underserved area required; secondary interview.

Years 1 and 2: some additional basic science instruction.
Years 3 and 4: required clinical rotation in Watts area of LA
Assignment to a longitudinal primary care clinic
Must do a health disparities related research project and thesis
67-70% URM, 20-24 students in program.

Looked at pre-matriculation vs graduate surveys for students in this program vs regular UCLA med classes over first 10 years of program. Looked at change in intention to practice in underserved area from pre-matriculation to graduation to practice.

		UCLA/Drew	UCLA
Matric	Yes	68%	24%
At grad	Yes	86%	20%

Predictors: being URM, participation in program, intent at matriculation

Grad practice location – in any medical underserved location:

44% of non-URM ................................... much lower
56% of URM

Participants stressed importance of having like-minded students and faculty and the importance of informal interactions.

Reported by:
Stephen C. Shannon, D.O., M.P.H.
President
American Association of Colleges of Osteopathic Medicine

5. The Future of Allopathic Medicine

Moderators: Erick Cheung • Bharath Nath
Speakers: Jordan Chon, MD • Michael M.E. Johns, MD

[It is interesting that the title includes “allopathic” adjective, several years ago it would just be titled the Future of Medicine]

Jordan Cohen, MD:

1. Must consider allopathic medicine in context of workforce shortage projections:
   1. 11 states and 12 MD specialties reporting current shortages
   2. Physicians aging faster than population (1985 majority of physicians < 50, now majority >50)
   3. Factors impacting supply include:
      1. Gender and generational differences
      2. Lifestyle choices
      3. Changing practice patterns
      4. Productivity changes (NPs/PAs)
   4. Factors impacting need include:
      + Population growth
      + Aging
      + Chronic disease epidemic
      + Public expectations
      + Economic growth
      + National investment in healthcare innovations
      ± Improved diagnosis/treatments
      – Changes in organization, delivery, and financing healthcare
      – Cost containment efforts
2. Other issues:
   1. There is a recognized need for robust concentration on primary care but MDs not responding (40% MD grads chose primary care in 1995, 20% in 2004)
   2. Maldistribution is a major problem in HPSAs, rural areas
   3. There is a great diversity gap and it is growing (25% of population minority, but 10% of physicians)
   4. 80% of MD students from top 20% socioeconomic groups
   5. Increased cost and debt for medical students a problem
      1. Growing medical school tuitions unconscionable
      2. Government won't solve problem for “rich doctors”
      3. Advocates from physician community come forward and support medical education, grow scholarship programs
      4. Need growth in such programs as Health Service Corps
      5. “There are challenges to the supposed ‘superiority' of LCME model. Osteopathic medicine is appropriately responding to shortages. If allopathic medicine lags behind this need we must ask how ‘superior' LCME mode really is if we can't respond to this need.”
   6. New AAMC initiative targeting minority recruitment into medicine at: http://www.aspiringdocs.org

Michael Johns, MD, CEO Emory Healthcare:

1. “Quality and humanity of what we do for others is the most important, not the money.”
2. Many new factors will be affecting physicians:
1. Rising expectations re. evidence based medicine (AHRQ)
2. Globalization
3. Acceleration of technological change and medical knowledge, e.g. molecular biology, nanotechnology, imaging, robotics
4. Public-private innovation, e.g. “Medical Home”, CVS Minute Clinics: “Their business model is working out very well financially. Is this good or not? What if high quality of care is demonstrated?”
5. Convergence of many factors a major force, i.e. business models with technology
6. What will emerge is “Predictive healthcare”, where we shift from targeting care to targeting prevention based upon new knowledge of individual risks and population health factors
3. ln answer to question about tuition costs: “We hear about the efficiency of osteopathic medicine. If osteopathic medicine is so efficient, why is their tuition so high?” [a letter from me is forthcoming]

Reported by:
Stephen C. Shannon, D.O., M.P.H.
President
American Association of Colleges of Osteopathic Medicine

back to top

6. LCME Hearing on proposed New Standards [~300 present]

1. First hearing since last changes in 2002
2. Little controversy in standard changes proposed (mom and apple pie)
3. One change would require service learning opportunities in all LCME schools. Currently 121 of 125 have such opportunities, and 20% require in curriculum
4. One standard would require that students be introduced to “basic principles of clinical and translational research.” When standard was questioned as to why so specific, the LCME Co-Chair, Ronald D. Franks, MD (VP for Health Affairs at East Tennessee State University) replied: “One reason for this was that LCME wanted to distinguish MD from osteopathic schools since so much of their curriculum overlaps with ours.”
5. LCME meets in executive session; there are no open hearings and no representatives from the schools undergoing review present. School representatives only are present in appeal situations. Site visit teams consist of volunteers from other schools with LCME staff support.

Reported by:
Stephen C. Shannon, D.O., M.P.H.
President
American Association of Colleges of Osteopathic Medicine

back to top

7. Workforce session on Physicians < 50,

Clese Erikson and Ed Salsberg

PDF File: State of Physician Workforce (1Mb, 22 pages)
[Highlights presented here, with more questions raised than answered at this point]

1. Preliminary (raw) data on survey of physicians < 50 from AMA Masterfile, therefore including DOs, approx 9000 physicians all states/specialties
2. 27% of female physicians vs 4% of males working part time
3. Women physicians all ages working less hours
4. 27% of females vs 34% of males would leave medicine if they could afford to do so
5. 45% of all physicians working with NPs or PAs, and 82% of them feel that it has improved the quality of care
6. Factors rated as important in choosing a practice position:
1. 69% of all physicians rate family/personal time as most important factor (80% for females and 62% for males)
2. 37% practice income
3. 8% ability to serve underserved
7. 49% said their practices allow them to balance work/family time
8. Younger physicians less satisfied with medical career than older physicians
9. Average work hours 58/week for full time, little variance by specialty, therefore the “ROAD (radiology, orthopedic surgery, anesthesiology, dermatology) to happiness” referred to by students selecting residency path partially a myth
10. Over half of primary care physicians are employees
11. Both generational and gender differences identifiable
12. Data will be further analyzed along with that for physician survey >50, and reported over next several months

Reported by:
Stephen C. Shannon, D.O., M.P.H.
President
American Association of Colleges of Osteopathic Medicine

back to top

8. Dual Degrees: Opportunities for Students, Challenges for Administrators

Moderator: Stacey R. McCorison, MBA
Speakers: Carrie Steer-Salazar • Jean Lantz, MA • Gaye W. Sheffler

S. McCorison, MBA, Duke:

Consider what are the best types of dual degrees for the institution, best fit. Need to look at the outcomes. Are the programs meeting their mission?

All types of terms for these programs – joint, dual, combined, concurrent, concomitant; a confused terminology and they are all interchangeable.

More and more students want these programs. It is hard to go back to school once you leave.

Websites need to be accurate and everyone involved needs to be knowledgeable about the program and able to answer questions or refer to right person. Students need individual faculty contacts, someone to answer Q's.

Need to consider all types of internal Q's, e.g.: are their slots available in the other program if student is denied admission to the other?

There are navigation problems, especially if the two programs are in different schools or even different universities. Timing of taking boards is an issue – students need time off to study. Maximum time to graduation and coming up against typical allowed limit may be impacted. Students need to adjust to a different peer group as they move from one graduating class to another. Also, need to be sure there are adequate numbers of rotation slots when the students resume medical studies.

Duke program: 1 yr basic science, 2 yrs clinical science, 3 rd year of research, 4 th year clinical. Fitting in second degree – can go in all different places, depending on school and program. At Duke, 20% of students get a second degree; a very high interest.

C. Steere-Salazar, UCSF:

Financial Aid perspective:

AAMC Institute for Improving Medical Education public policy statement: recommended flexible programming. Many reasons for doing it, e.g., students asking for these options. One reason is as an additional revenue source. At UCSF, medical school is not breaking even; some programs are making money, however. In UC system, the revenue stream stays on campus, unlike core program tuition dollars.

If you are going to do it, need to consider:

• get financial aid office and registrar involved early
• inform accrediting bodies what you are doing and get approval if necessary (Note here: for some programs, certificates, WASC doesn't care. However, UCSF needed to get OK from Department of Education (this had to do with checking that the length of time of program was OK, among other things)
• Need to determine if curricular requirements are met – at UCSF, this involved Faculty Senate
• Separate out the debt to determine what the medical school vs what the other programs part is
• Need to track the programs separately, financially and on transcript; it needs to be correct. Debt management issues are important. Loans may have different limits; available funding may vary; start and stop dates aren't consistent.

Sometimes programs are started by one enthusiastic individual, dynamic teacher, etc and there may even be an outside funding stream; e.g., UCSF certificate in biomedical research, with biotech funds. What happens if source only is for a limited period – who will fund the program after that?

There are many players: admissions, faculty, faculty senate, administration, financial aid, registrar, accrediting bodies, curriculum committee, IT (webmaster), external funders.

J. Lantz, MA, Iowa:

Accepting of units by the other program. Usually there is some. At Iowa, public health accepts 9 U of M.D. program toward MPH degree. Agreements (waivers of hours required to graduate) need to take place in advance. There is a concern about “double dipping” but main thing is that core curricula need to be preserved. As long as those requirements are maintained for each program, they haven't had problems with this issue. They have 18 hrs of core MPH programming. If a student doesn't finish one degree, then the waivers are lifted and all requirements of the other degree must be completed.

Again, stressed that registrar, financial aid, administration, curricular affairs, and student affairs all should be involved early in the discussions. Students are excellent recruiters. These programs need attention – they don't run themselves. Someone needs to check that students are meeting all requirements of both programs and that students don't become alienated from either program. Need to be sure students are maintained on the appropriate group lists, etc.

Scheduling must be flexible. It is easy to state that one route must be followed, but it is nearly impossible to make that stick, as students have all sorts of special issues.

Need to watch out for loan limits; need to pay attention to clerkships as dual programs can impact which clerkships will be available (which will be left) when student is ready to return to rotations.

At Iowa, students do 2 yrs med school, then take step 1 USMLE; then do MPH; then go back to complete last 2 yrs.

Promotions issues – usually 2 committees involved. There should be a handbook (they haven't done it yet).

They also stressed importance of giving both degrees at the same time – maybe students won't finish otherwise?

Good MD/MPH resource person is Rika Maeshiro, AAMC Asst VP for PH and Prevention, Med Ed Division. rmaeshiro@aamc.org She has an MP/MPH listserv.

Reported by:
Barbara M. Kriz, Ph.D.
Associate Dean for Preclinical Education and Research
Touro University College of Osteopathic Medicine

back to top

9. NBME/USMLE/FSMB Update

October 30, 2006
Speakers: Agatha P. Butler, PhD• Gerald Dillon, PhD Robert M. Galbraith, MD • Donald E. Melnick, MD • James N. Thompson, MD

Dr. Dillon reported on results of USMLE Step 2-CS (clinical skills), on a “Comprehensive Review of USMLE project” (CRUP), and on standard-setting activities.

Step 2-CS has tested 70,000 examinees total since implementation in 2004. While U.S. medical students passed at a rate of 96% for first-time takers in 2004-2005, this increased to 98% in 2005-2006. The pass rate for international medical school graduates, however, is lower (83% for first-time takers). USMLE has made a technical adjustment in the standards after looking at data in the Spring of 2006, and has applied this effective July 2006, such that a slight increase in the standard was made for the integrated clinical encounter score, and also an increase in the standard was made for the communication and interpersonal skills component.

Despite having five testing sites for Step 2-CS, with 32,000 examinees annually there are some scheduling “crunches”, and USMLE is piloting an email notification system whereby examinees can sign-up for electronic notifications as to future available test dates at their site(s) of choice. Score reporting dates are published annually, so examinees have a general idea that if they test by a certain cutoff date (e.g. July), they will most likely receive their score reports by a certain date (e.g. October).

Annual school composite reports were added this year in September, such that each school receives information as to the pass rates of their students, as well as performance profiles of their cohort relative to the national averages.

The Comprehensive Review of USMLE Project (CRUP) is underway, with a task force that will collect data from various focus groups and other surveys that are attempting to determine what types of measurements are important for state licensing boards, and also to determine what role the USMLE scores play in medical school promotion, graduation, etc.

Standard-setting for USMLE examinations occurs every three years, using surveys as well as content review by expert panels. For Step 2-CS and Step 2-CK, this process is underway for new cut points in Spring 2007. For Step 1, panels are meeting now and a new standard is expected to be applied for January 2007. Step 3 will have new standards in Spring 2008.

Dr. Butler presented information about services to medical schools, including NBME's new product “Customized Assessment Services”, which allow schools to customize the subject (“shelf”) exams.

Dr. Galbraith reported on activities of NBME's Center for Innovation, and the Stemmler Fund. These include pilot studies of 360 degree-type assessment tools for professionalism, pilots of a electronic portfolio system with a hub at NBME to act as a dynamic virtual personal database, and some rapid item generation software and computer-assisted item development processes being developed at the center.

Dr. Thompson from the Federation of State Medical Boards reported on FSMB activities regarding the maintenance of licensure and certification, and Federation's interest in competencies and the current state of CME. He reported on the Physician Accountability for Physician Competence (PAPC) Summit meetings, and future plans for a National Alliance for Physician Competence. The PAPC has collectively worked on drafting a “Good Medical Practice” competency document, which will soon be available. Besides the FSMB, other organizations recognized for their involvement in the PAPC include ABMS, the AMA, and the AOA.

Reported by:
John R. Gimpel, D.O., M.Ed.
Vice President for Clinical Skills Testing
National Board of Osteopathic Medical Examiners

back to top

10. Expanding Our Horizons: Future Trends in Alumni Affairs

Moderator: Gwendolyn M. Smith-Johnson
Speaker: Ginny Darakjian

Increasing numbers of females are alumni. Need to plan partner events that include things the husbands, families are interested in.

Consider programming and who is on committee: males, females, other cultures; who is on alumni board – this sends a message to the students.

More younger alumni, many who met in school; need plans for the kids and affordable programming at reunions. Fewer speeches, banquets, more fun events: wine-tasting is popular; find out what they would like to do.

Shift to technology – what is best way to reach them: eNewsletters; weblinks

Getting up to date email addresses is a big problem. Assuring alumni that they are not being opened up to spam email; getting approval to send stuff to them.

Reunions are best place to get updated information, either at them or in preparing for them. Limit what you send them. Give alumni opportunity to decline (opt out); let them limit communication to certain types of information.

IT people may require subscription to lists.

Having an alumni directory on line, password-protected.

Many universities use a vendor to manage all sorts of alumni affairs. Some are more flexible than others. With some, you can buy blocks of services: directory; events, class notes, obituaries, posting photos, etc. Almost all places that tried to do this internally felt that it was more difficult and more expensive in the long run.

Some discussion of communication between alumni data base and development/giving data base and how that can be facilitated.

Northwestern has students check out with alumni office as part of graduation exit process.

Develop a relationship with all the other relevant departments that might get information about alumni so it is fed to alumni office to keep records updated: registrar, library (as resources are requested there long after graduation), credentialing office.

A concern: Various search engines/email providers want to provide all sorts of services to students. Maybe advertisement free at first (i.e., for email), but once they graduate, they will start to get ads. The university likes the idea of going with such a service, because it is free or cheap, but alumni office has concern because it will damage the trust they have created. Currently, alumni are getting email forwarding for free; with vendors, this might not be possible.

Ideas: one-stop shopping for alumni – ordering transcripts, alumni information, etc.

Privacy may be less of a concern in the future – students and their expectations of this technology are changing.

Communication tools for reunions – students want to know who else is attending, event module provides this. Posting this information: can just post name. Some do this in open forum, some do it behind password protection. If someone asks for another alum's email address, best way it to forward the query rather than to send the address.

Podcasts: U Michigan is using them. Very well received by faculty and students.

Photos of registrants at all events; one idea is to put the photos on the registration card. Having on-line registration. Digital photos allow for photo directories and can put this in the data base. Helps later in identifying who was who in a group photo, too.

Alumni giving is sometimes a separate dept from alumni relations and sometimes combines – more moving to latter arrangement, lately.

More alums in military; think of ways to honor them: profile them in magazines, newsletters; color guard at events; videostreaming during reunion.

Programming – can get attacks from alumni, if lectures given or articles published on certain scientific topics, or political angles – evolution; medical ethics issues. Opinion given that alumni office should portray pride in the universities accomplishments and not give in to such pressure.

Ways that alumni contribute to the medical school, including ongoing classes: sponsoring white coat ceremony, providing scrubs, giving seminars on the business of medicine, how to interview for a residency, how to prepare a resume, etc.

Reported by:
Barbara M. Kriz, Ph.D.
Associate Dean for Preclinical Education and Research
Touro University - California
College of Osteopathic Medicine

back to top

11. Workshop: Developing High-Quality Multiple Choice Tests to Assess Application of Basic Science Knowledge Using Patient Vignettes

Organizers: Kathleen Z. Holtzman • David B. Swanson, PhD
Faculty: Paul M. Wallach, MD

Covered many aspects of the NBME “red book” on item writing.

Started with a “nonsense” pre-test, which was actually was a demonstration of how someone who is test-savvy can figure the questions out. 7 questions, average score is about 2.6. Not necessarily good to be really test-savvy, may not have to know the material particularly well.

NBME likes all stems to be in the form of a patient vignette, which should include several elements: age, gender, chief complaint, duration of complaint, any relevant date, tests. Then, the lead-in is best done in the form of a question. The number of options can vary – no magic number, although sometimes there is an optimum number for a particular type of lead-in or stem.

NBME now only uses the “one best answer” format, in single items or in sets. Not clear whether they also use T/F (they did say that they don't prefer them);and for sure M/C questions should not have as options just a bunch of T/F statements.

No K-type questions any more – book has a chapter on old question types not used any more.

Avoid vague terms – usually, rarely, frequently, associated with; don't want test taker to have to guess author's intent.

Student should be able to answer the question without seeing the options – ideally, want a “cover the options” style.

Length of stem is a time issue, but you do want all the relevant information to be there. Red herrings in stem are not as good an idea for a beginner question as in a later exam, e.g. in the clinical years. Have to think about intent of exam and stage of development of the student. Is it a minimal competency exam or are you looking to spread the group out.

NBME orders the options alphabetically if they are diagnoses. For treatments, they are listed from least to most invasive.

Lead-ins must be focused. Options should be uniform, e.g., don't want one to be excessively longer (that's often a clue that it is the correct one) and don't want options to be so different in style that it is distracting just to read them.

Don't want negative options – it just turns the question into a T/F type.

Reviewed the pre-test and then examined how a test-savvy person could figure each one out. Flawed items:

• ones that favor a test-wise examinee, e.g., options that eliminate each other
• correct option longer and more detailed than others
• repeating a word in the stem or lead-in and option
• using terms like always or never
• options that involve counting
• item too long
• overlapping numerical options
• numbers should be either in increasing or decreasing order
• none of the above included as an option; another choice is preferred
• roman numerals rank ordering

Stems: complete sentence is helpful in avoiding grammatical issues and fulfills the “cover the options” criterion.

Item content: should be congruent with course objectives, so if student studies “to the test” it is a good use of their time. Basic sciences exams promote horizontal and vertical integration, application of knowledge rather than recall, and reinforce important information. Use clinical vignettes. Pick a vocabulary, clinical terms that are appropriate to level of learner.

Lead-in: complete sentence with a ? at end is preferable.

2-step questions are OK but may want 1 st step to be easy. Good to bring first year knowledge back in 2 nd year testing. “Most likely” is OK in a patient vignette, because that is reality of clinical scenarios.

Encourage use of images for the stem or for the options – photos, diagrams, graphs. Their use will increase in NBME exams. Video options are being considered too.

Writing questions – had a practical example. Worked on improving a set of questions provided by NBME.

In writing questions it is helpful to bring in experts or have others review questions. Interdisciplinary exams, with no course label are good. Assign items based on contact hours. Work in groups - sometimes is helpful to have groups read questions out-loud and then review, for quality, redundancy. Groups should have both clinical and basic scientists. Experience of one school was that for first exam drafts it took about 3 hrs to prepare the test; got a little faster with experience. They have items due centrally (someone in charge), due 10 days before exam. There is a joint meeting 7 days before. Exams are organized and printed centrally, scantron sheets handled centrally. Going to computer testing – some schools. 40-45 Q's per hour. This school kept separate course grades, so Q's for a particular course collected over entire term/year.

Experience of working together and integrating exams generated excellent dialogue about the questions, curricular content, and promoted better organization, collegiality, creativity.

Use a balance of cases with female and male patients; don't have females “complaining and denying,” males “suffering” and other such descriptors.

Reported by:
Barbara M. Kriz, Ph.D.
Associate Dean for Preclinical Education and Research
Touro University - California
College of Osteopathic Medicine

back to top

12. Assessment Across the Continuum: Focus on Self-Assessment

Self-assessment is personal, unguided reflection on one's strengths and weaknesses. Most people don't do it well.

Why can't we self-assess and make performance improvements? Poor performers can't judge or recognize good performance. They leave out crucial details that could improve their performance.

Why do we think we can? We make sense of ambiguous situations and tend to pay selective attention, justify, and discount certain details. We make predictions about what is going to happen next. We are often quite self-aware but are not able to mentally aggregate across past experience. Rely more on salient or recent events. Discount things that are inconsistent with recent experience.

What to do about it? Reconsider the nature of self-assessment. Create opportunities for students to discover their boundaries – testing helps in retention. Don't emphasize accurate self-assessment over accurate assessment. Focus on skill development. Performance isn't well correlated with self-perceived skill.

People do self-assess, whether accurate or not. Benchmarks help – discussion, or listening to one, helps and give clues to gaps in knowledge. Feedback helps; self-assessment in isolation is useless. Staying connected is important – CME, societies, collaborations, internet, journals.

How do residents self-assess? On their own, through portfolios, exams, surveys. Residents don't like self-assessment forms if they don't have objectives, benchmarks.

Feedback – timing is key. Mid-rotation evaluation, e.g.; need time to improve. Mentorship – advisor meetings most helpful to residents.

Barriers to self-assessment:

• feared and actual repercussions; won't get the plum fellowship
• gossiping attendings
• looking for someone to blame
• residents highly suspicious; afraid of incriminating themselves; need to cover up gaps
• not being told what self-assessment will be used for
• not wanting to praise themselves too much or be too harsh on themselves

Facilitation of accurate, meaningful self-assessment:

• mentorship; establishing trust
• problem solving; not a punitive environment
• dedicated time for the discussion
• giving opportunities, time for improvement

People respect themselves and others, including attendings, if they admit mistakes. Self-assessment is only useful if done well and is a genuine learning experience.

Reported by:
Barbara M. Kriz, Ph.D.
Associate Dean for Preclinical Education and Research
Touro University - California
College of Osteopathic Medicine

back to top

13. Implementing a Vision for Medical Education in the United States: GEA Report for AAMC and IIME

October 29, 2006
Moderator: Lois M. Nora
Speakers: Ajit K. Sachdeva, MD • Carol A. Aschenbrener, MD • Lois M. Nora, MD, ND, MPN

Dr. Nora gave history, andoverview of the Institute for Improving Medical Education (IIME). There is a critical but poorly analyzed link between education and patient outcomes, including health outcomes and satisfaction with care. Challenges: calls for expanded learning and shorter time-frames; interdependence of silos: training and political realities; medical education and student affairs; importance of research, collaboration, and need for funding.

Dr. Sachdeva addressed from general perspective and particular perspective of surgical education:

Report – put in context of the six ACGME core competencies and in context of maintenance of certification programs and additional ABMS competencies, including commitment to life-long learning and evidence of evaluation of performance in practice.

Strategies to address recommendations of report:

• pursuit of competency based education across the continuum of professional development; e.g., through OSCEs on patient communication for entering residents; OSCEs for technical skills; OSCEs on management of adverse events. Need team leadership, mutual trust. Much has improved, more to do at each stage along development of practicing physician. Need to focus on professionalism and communication from day 1. Patient is a partner.
• specific focus on practice-based learning and improvement; continuous professional development; focus on CQI; need to get away from punitive approach.
• use simulation in education all types and at all levels. Need research to show it has added value.
• train and reward faculty how develop new teaching, learning, and assessment methods. Educator track should not be second class track.
• also need a re-entry system for physicians who take time to do other things – raise families, do research, do administration, do global practice, do political service.

C. Aschenbrener, Chair of NBME, asked what AAMC can do to help medical schools advance these initiatives: meetings, encouragement of research; connect schools, educators to develop team learning (learning communities); develop a consensus on the bar for the competencies.

Reported by:
Barbara M. Kriz, Ph.D.
Associate Dean for Preclinical Education and Research
Touro University - California
College of Osteopathic Medicine

back to top

14. Moving Educational Activities into Scholarship: Results and Recommendation from the AAMC-GEA Consensus Conference on Educational Scholarship

October 30, 2006
Presenters: Deborah E. Simpson, PhD • Ruth-Marie E. Fincher, MD • Janet P. Hafler, EdD • David M. Irby, PhD • Boyd F. Richards, PhD • Gary C. Rosenfeld, PhD • Thomas R. Viggiano, MD

D.Simpson, Ph.D.

Parameters of Scholarship: it must be public, peer-reviewed, and presented via a platform that others can build upon.

Documentation: show how your work is informed by existing literature, best practices, resources in field, and/or colleagues locally to internationally – that is show how your work draws from other scholarly work. Then show how it contributes to scholarship, through dissemination (papers, peer-reviewed forums, presentations) and/or by impact on the field.

R. Fincher, M.D.

Educator activity categories: teaching, curriculum development, advising and mentoring, educational leadership/administration, learner assessment. All these fall within the educator category.

Teaching is any activity that fosters learning; in and of itself, it is not scholarship. It can be judged by its quality and quantity. Include lectures, facilitation, role modeling, creating associated instructional materials – handouts, interactive materials, media.

Curriculum development: a longitudinal set of designed educational activities, can occur at any training level, venue, or in any delivery format. Requires goals and objectives, learning experiences to achieve the goals and objectives, organization and sequencing to ensure effective learning, and evaluation of effectiveness.

T. Viggiano, M.D.

Advising and Mentoring: involves a developmental relationship in which an educator provides guidance or counsel to facilitate accomplishment of a learner's or colleague's goals. For example, the outcome or proof can be in the production of a scholarly work by the advisee, or recognition by that person of the role the mentor played.

Educational Leadership and Administration: exceptional leadership that transforms educational programs and advances the field. A leader in this sense is one who pursues excellence, evaluates and engages in self-reflection, builds on the work of others, disseminates results (advances the field) and garners and maximizes human and fiscal resources.

B. Richards, Ph.D.

Learner Assessment: activities associated with measuring learners' knowledge, skills, and attitudes. Documentation should include evidence of adherence to Glassick's six criteria: clear goals, adequate preparation, appropriate methods, significant results, effective presentation, and reflective critique.

Conclusion: Educator's contributions must be valued in academic promotion. Promotion standards must be in balance with the educational infrastructure and with mission of institution. Does a faculty member have to be excellent and doing everything in order to be promoted?

For educational research, one still needs an infrastructure – for example, data collection and analysis, just as with bench research.

Reported by:
Barbara M. Kriz, Ph.D.
Associate Dean for Preclinical Education and Research
Touro University - California
College of Osteopathic Medicine

back to top

15. Organizational Infrastructure to Support Scholarship in Education

October 20, 2006
Moderator: John H. Littlefield, PhD
Discussants: Deborah E. Simpson, PhD • Debra A. DaRosa, PhD

Presented four frames for analyzing organizational infrastructure:

• Structural – org charts, policies, and procedures; identify resources available to support teaching scholarship
• Human resources – advancement of faculty knowledge, skills, and behavior as educators
• Political – educators directly or informally influencing decisions about priorities and resource allocation
• Symbolic – certain activities communicate organizational values; how are educators' activities publicly displayed.

Structural frame at the department level. An example given of a surgery department in which Vice-Chair of Education position created at an equal status level with Vice-Chair of Clinical Affairs or of Research. This elevates importance of education, provides built in mentorship, team-building approach that promotes scholarship. Key to have support from the top for this. Accreditation (i.e., external) pressure can sometimes be useful. In this example, the individual is providing faculty development on how people learn, building communication skills, etc. Educational research does require IT/data collection, entry, and evaluation support.

Structural frame at the medical school or academic health sciences center levels. Gave example of an institution where there is an Office of Education for the campus, with an Associate Dean for Education for each component School. This office is staffed by people trained in medical education, providing services and faculty development in curriculum design, teaching techniques, assessment, etc. Graduate students from other departments, schools, may be used to help faculty in the medical school. IT infrastructure, podcasting, standardized clinical assessment all run through this office. The Office of Education does all the course/instructor evaluations, testing, assessment.

In this model there is a promotions track for clinician-educators. Extramural funds are available for medical education, but still need an infrastructure for grants management and support. Med ed researchers can benefit by interaction with more “hard core” researchers to help them in promoting and disseminating their work.

Valued clinician-educators are involved in the faculty recruitment and interview process when new faculty are hired.

This institution has a fall convocation at which those demonstrating educational excellence and/or innovation are recognized. They make education visible in newsletters, and through various symbolic ways – perpetual plaque, honored at dinners, etc.

Reported by:
Barbara M. Kriz, Ph.D.
Associate Dean for Preclinical Education and Research
Touro University - California
College of Osteopathic Medicine

back to top

16. Planning for Class Size Increases

Two institutions told how they have responded to a call by the AAMC to increase the medical school population by 30% to meet the anticipated workforce projections by 2015.

In Texas, the decision was to increase the number of medical students in the three smallest programs. Presenter stressed the importance of thorough assessment of infrastructure, preclinical, and clinical costs, maintaining standards, and obtaining the buy-in of the faculty.

University of British Columbia took a completely different approach. They established full 4-year programs in two additional locations, one of which was quite rural. They used this as an opportunity for outreach, bringing medical care to a more remote part of the country, thereby addressing the existent mal-distribution of physicians as well as the overall shortage. This also opened opportunities for local economic development and for new types of research (e.g., in aboriginal health). Educational technology was very important – videoconferencing, etc., keeping connections with main campus.

Last part of this presentation was by AAMC, which explained the data and services that are available through AAMC so that member schools can determine such things as what percent of applicants are being accepted at each individual school in comparison with all schools. Many other types of information are available too, e.g., on majors, gender, ethnicity, etc. It is possible to use this information to target recruitment efforts as schools attempt to increase enrollments. It was said that when there are 1.5 applicants or more per available seat, AAMC feels that there is a large enough pool for expansion. When the number falls below 1.3 there is a concern. Currently the number is just about 1.5.

Reported by:
Barbara M. Kriz, Ph.D.
Associate Dean for Preclinical Education and Research
Touro University - California
College of Osteopathic Medicine

back to top

17. The Under-performing Medical Student: How to Identify and Address Learning and Emotional Difficulties in the Preclinical and Clinical Years

In this workshop very typical cases were analyzed by the group, to determine whether or not an under-performing student was simply adjusting to medical school, not studying adequately, had need for assistance with study techniques, test-taking, etc., or had moderate to severe (depending on the case) emotional or learning issues.

A Learning Survey was distributed and shared as a useful tool for those to whom such students might be referred, e.g., a learning resource center, student affairs dean, etc. It can be filled out by the student and may indicate to the evaluator where certain problems cluster.

Loring Brinckerhoff, Ph.D., a psychologist who works with ETS, and with Harvard and other institutions spoke about neuropsychological testing, types of learning disabilities, characteristics of students with LD and/or ADHD, and accommodations. One message was that not all psychometric evaluations will secure accommodations under the ADA and many “evaluators” are not qualified to make the diagnosis. The National Board of Medical Examiners is very strict in what they will and will not accept.

Reported by:
Barbara M. Kriz, Ph.D.
Associate Dean for Preclinical Education and Research
Touro University - California
College of Osteopathic Medicine

back to top

18. Formative and Summative Computerized Assessments in Medical Education

This was a very interactive presentation from the Dept of Pathology of the University of Alabama School of Medicine, primarily delivered by their instructional designer, an individual with a Master's in Education, specifically focusing on educational technology – a nice resource person to have!

They obtained a grant initially and developed a computerized exam that combines path, pharm, and micro. They did caution that when you get a grant you need to think about how you are going to maintain the program that you start.

One of the big take-home pieces of advice from this session was that, if you are considering moving to CBT (computer-based testing), don't try to build your own. There are commercial products out there and a lot of the session had to do with the different types of software available. Clearly, researching these and selecting one was favored over trying to invent something locally. There was a lot of individual opinion expressed in this session about what the pros and cons of different approaches are – which software, using wireless environments vs hard-wired computer labs, etc. However, it seems very clear that medical education is moving more to virtual, on-line environments and our colleges need to plan accordingly.

Reported by:
Barbara M. Kriz, Ph.D.
Associate Dean for Preclinical Education and Research
Touro University - California
College of Osteopathic Medicine

back to top

19. Academies Collaborative Annual Meeting

Moderators: Molly M Cooke, MD • Jessica Muller, PhD • George Thibault, MD • Leslie Zimmerman, MD

Purposes in developing a faculty academy included:

• To draw attention to education
• To provide standards and support for promotion and tenure
• To encourage scholarship in teaching, service, advocacy for teaching, cross-discipline discussions, and educational leadership

Shared issues in the development and growth of a Faculty Academy included:

• Where should the infrastructure of the academy fit organizationally?
• How do we move away from exclusivity and toward inclusivity? How do we deal with applicants who are not accepted
• How do we establish critical relationships to faculty development effort?
• How do we deal with the time-consuming process of mentoring and selecting members?
• What should we expect from faculty members who are already overburdened and unrewarded?
• How do we fund activities?
• How do we involve GME in our academies?
• How do we set appropriate expectations for inductees?
• How can we use mid-term reviews to maximize contributions of members?
• How can we provide evidence for the Dean that the Academy is making a difference?

Reported by:
Linda Heun, Ph.D. <meded@aacom.org>
Vice President for Medical Education
American Association of Colleges of Osteopathic Medicine

back to top

20. First Annual Meeting of Directors of Medical Education Fellowships

Handout: PDF File: Faculty Fellowship Programs (910k, 18 pages)

A survey of US medical schools was conducted and led to the November, 2006 issue of Academic Medicine. In that survey a ‘fellowship' was defined as a longitudinal faculty development activity for a cohort of individuals. The survey resulted revealed a focus on the following topics: teaching skills, networking with other faculty, scholarly dissemination, program evaluation, learning evaluation, curricular design, educational theory, educational leadership, use of educational literature, educational research, career advisement and reflective practice.

Program Evaluation utilized satisfaction questionnaires, self-assessment of skill development, follow-up interviews; cataloguing of educational activities, peer observation and portfolios.

The following goals and issues related to a collaboration of fellowship directors were identified:

1. Curriculum Development:
   1. how could we effectively share our curricula?
   2. how do we define a curriculum? Is there a core set of learnings?
   3. how do we decide if the curriculum is meeting the fellows needs?
   4. can we work toward standards for evaluation?
   5. how can we involve administrators to identify what they would expect differently from a fellow? programs must also allow the fellow to innovate
   6. how can curriculum establish a cadre of interactive learners?
   7. does curriculum help faculty understand how people learn?
2. Models for mentorship of fellows:
3. Developing educational change agents regarding the work culture; promotion standards
4. Collaborative studies in faculty development; how about a retrospection of programs with a history?

An action plan and listserv will be developed to take this effort forward.

Reported by:
Linda Heun, Ph.D. <meded@aacom.org>
Vice President for Medical Education
American Association of Colleges of Osteopathic Medicine

back to top

21. Searching for Diversity: Conducting Successful Searches

Moderator: Cynda A. Johnson, MD, MBA
Panel: William T. Mallon, EdD • M. Roy Wilson, MD, MS • Susan Shurin, MD • Virginia Valian, PhD

Handout: PDF File: Successful Searches (1.14Mb, 28 pages)

[M. Roy Wilson, MD] - An Institutional Perspective

• Commitment must be institution wide and include a business plan
• The management team related to diversity searched must itself be diverse
• There must be an appropriate allocation of resources, including having an ‘opportunity fund' for special cases

1. The Search Committee

• Ask for a diverse and good-sized pool, come up with an unranked list of finalists, and conduct an active not a passive search (don't just post a job description-go out and solicit potential members)
• Final composition of search committee should be diverse, respected senior-level staff and/or bring in members from other departments or the community
• Provide human resource training for the committee or include a human resource person
• Include a legal expert
• If you're seeking women member, use ELAM as a resource

2. An effective search process is only the beginning, retention is as important as recruitment

• Avoid the service trap – if overused in service work, there are no tenure-related credits; set up a mentoring program

[Susan Shurin, MD] Executive Search Firm/Senior Staff at NIH

Diversity must be seen as a core value for universities because such institutions are a main source for leaders. Unfortunately institutional leadership is inherently conservative and not geared for change. She indicated that when seeking to expand the number of leaders who are women and underrepresented minorities into an educational system, such leadership must be nurtured. New leaders must be encouraged to transcend themselves and the extant leaders must be sensitive to what the culture is telling new leaders about what they can do. We need to be more open minded about what a leader looks like and build a safety net that permits a learning curve.

[Virginia Vallian, respondent, from the Gender Equity Project]

• People tend to write letters of recommendation differently for men and women. Adjectives for men include brilliant; for women they include careful, persistent. Doubt-raising comments are more likely in letters for women
• The University of Michigan trains a cadre across the university about such social science findings regarding women and minorities which advises searches in all departments

Reported by:
Linda Heun, Ph.D. <meded@aacom.org>
Vice President for Medical Education
American Association of Colleges of Osteopathic Medicine

back to top

22. Reforming CME: Whose Responsibility is It?

Moderator/Respondent: Michael Fordis, MD
Speakers: (Note: AACOM only heard respondent comments)

The respondent expressed concern that those individuals currently responsible for CME might not have the skill set or orientation to implement the performance-based learning and assessment described by the speakers. He asked if current CME professionals would have to experience reentry or retraining? “Will it be done through us, with us, through what process? We are told to facilitate and assess performance improvement which involves more than getting data about performance and turning it into CME programming. The new approach involves interaction with physicians, followed by intervention, then reassessment. Are we equipped?”

He clearly identified the need for professional development for those in CME. Professional needed to learn about and move toward a) evidence-based CME, b) credit systems that reward important behaviors already being done and provide the motivation to try new behaviors and c) practice-based CME.

Responding to questions about the need to collaborate with other health care professionals from the audience, Michael Fordis, President of AAMC's Society for Academic Continuing Medical Education suggested the need to focus on interprofessional group learning and communities of practice. Asked about the need to cope with the strong perception that current CME efforts are ineffective, he suggested that leaders needed to focus on a few goals and work toward them, to stop thinking of CME as a vacation and think of it in terms of patient outcomes. He further suggested that while there had always been the creativity and desire to focus on improvement patient care, with the internet and IT there are now the tools to do so.

Reported by:
Linda Heun, Ph.D. <meded@aacom.org>
Vice President for Medical Education
American Association of Colleges of Osteopathic Medicine

back to top

23. Research Paper Presentations: Measuring Clinical Skills

October 29, 2006
Discussant: Mark Albanese, PhD
Moderator: Alex Mechaber, MD

RIME (Research in Medical Education) Research Paper Presentations

Courtneay Barlow, MD presented a paper titled “Early Clinical Skills Improves Confidence in Clerkships”

Investigators observed that Transition to 3 rd year entails anxiety provoking change from familiar classroom to unfamiliar learning situation in clinical settings. Moreover, investigators observe that medical students feel most prepared for clinical training when clinical skills are introduced early (first and second year) and are real patient encounters and not passive lecturing.

The hypothesis of this non-randomized prospective cohort study was: Students will be more comfortable in 3 rd yr following a new curriculum that introduced clinical skills early than they were with old curriculum that did not feature early introduction of clinical skills. Data was gathered using a 5 point Likert scale.

Investigators concluded that their study verified that confidence in clerkships rose with early introduction of clinical skills.

This study only looked at student comfort level, not competence. Future research will look at OSCE for correlation with increased measured competence.

Dr. Karen Szauter ( University of Texas Medical Branch)

Do Students Do What They Write and Write What They Do?: The Match Between the Patient Encounter and the Patient Note.

The note is important because it is the written record of health care issues. It serves in billing, research, litigation, and communication between providers.

Problem: faculty members use patient notes to determine skill in data gathering and integration, but review of the note is often remote from setting in which it was written so we must ASSUME the note is complete and accurate as to what actually happened.

Research Question: Does the patient note accurately, completely and correctly portray the patient encounter?

There have been some studies in past that involved post facto patient interviews or SP checklist. In this study, investigators reviewed videotapes from high stakes 4 th year standardized patient based assessment in which students used USMLE note format (available on USMLE website). 4 reviewers met in advance, established how to assess, then compared note with video of what was done in session. Reviewers transcribed every move student made and then compared with note. They didn't look at whether students did what case called for, just whether note recorded what happened. Next investigators classified as match or non-match. Non-matches included “did it but didn't report it,” “reported but didn't do it,” or “did it and reported it but reported it incorrectly.” Statistical investigation revealed high inter-rater reliability.

Findings: 96% of notes had at least one type of mismatch.

No student had a complete match in GI, 3% in Cardio, and 7% in Resp. In GI: 78% didn't do, but recorded. 64% in card, and 50% in resp. Done incorrectly, recorded was 65% in card, 47% in gi and 54% in resp. Incorrect abnormal findings were 14% in Cards, 47% in GI and 23% in Resp.

Not done/ done incorrectly, recorded as done were 82%

Dr. Edward Wu

There are 21 procedures students should do and know by end of medical school. Are they and do they? What procedures are they doing?

At end of 3rd year what are student experience and confidence levels? How do experiences affect their comfort?

Investigators asked how many times they did the procedures, how comfortable to do unsupervised, and something else I missed.

There were 7 skills a majority reported having never done. Also 7 a big majority reported low confidence. Skills w/ lower perceived importance were consistent w/ those they weren't comfortable.

2% reported formal evaluation of paracentesis and thoracentesis.

Presence of curricular materials was associated with greater incidence of performance.

Students are not learning and performing the procedures. Should they be? Which?

Reported by:
Glenn Davis, MS
Curriculum Director
Touro University College of Osteopathic Medicine

back to top

24. Addressing Medical Student Professionalism

October 29, 2006
Moderator: Heidi Chumley-Jones, MD
Discussant: Maxine Papadakis, MD

The P-Mex (professionalism mini-evaluation exercise) preliminary investigation
“P-MEX: A Tool to Evaluate Professional Behavior”
Richard Cruess, MD ( McGill University)
Jodi Herold Mcilroy, PhD
Sylvia Cruess, MD
Shiphra Ginsburg, MD, Med
Yvonne Steinert, PhD

Influenced by and built on Mini-CEX

Started w/ 142 behaviors and condensed to 24. Now it is down to 21. Wanted to measure as many different aspects of professionalism as possible, though not all w/ P-MEX

Construct validity examined w/ factor analysis.

Scores are reliable w/ a minimum of 4 evaluations.

Critical event requires immediate action within 24 hrs (sleeping w/ patient) whereas unacceptable rating merely requires attention of the dean.

They will evaluate professionalism of faculty in future, though probably not w/ this form. The behaviors will be the same.

The use of form is initiated by faculty member, not student.

Form is meant to be formative. Some faculty felt like it was too much like a traffic ticket, even looks like one.

They are going to make it part of the permanent record.

Issue of differences among raters as to what “acceptable” means is attenuated by multiple iterations (i.e. minimum of 4 observations required to obtain valid results)

Does Community Service During Medical School Make for Better Physicians?
by Amy Blue, PhD

New LCME standard requiring service learning.

http://www.lcme.org/hearing.htm

Are students who provide more community service during medical school academically stronger, better clinical skills, or more positively evaluated by residency directors?

Students with no community service were more likely to be male and white.

More community service was associated with higher GPA and USMLE2 up to the second quartile in GPA but there was a drop in service hours among students in the top quartile. There was no association with clinical skills. There was a trend toward better evaluation by residency but not significant.

This author observes there is clearly an association between rate of community service and academic achievement, but questions whether it is causal. It was suggested that admissions committees try to select students with record of community service as undergraduate. I suspect that paying extra attention to premed service levels would select for 2nd quartile students. A more intriguing question is how can we increase the level of service among top quartile?

Medical Student Professionalism: Are We Measuring the Right Behaviors: Relevant Issues Regarding Professionalism in Medicine by Michael Ainsworth

There are limits to relying on formal academic reports about unprofessional behavior. Faculty don't see all behaviors and may be inhibited for other reasons like avoidance of confrontation, fear of legal consequence, etc.

They use an “Early Concern Note”

Allows longitudinal tracking

Separate from academic record

Managed centrally through associate dean

Confidential but not anonymous

Intervention is voluntary and requires consent of student

Are unprofessional behaviors by students similar to those of physicians reported to state medical boards?

Reported by:
Glenn Davis, MS
Curriculum Director
Touro University College of Osteopathic Medicine

back to top