Choosing the Right Learner

Choosing the Right
Learner: Benefits of a
Structured Interview
Melissa McNeil, MD, MPH

How Do You Pick a New Learner?
 Medical students/residents/fellows are

selected based on a variety of criteria


Academic performance








Standardized tests
Grades
Research/publications

Evidence of leadership
Letters of recommendation
The interview

The Traditional Interview
 The interviewer generally asks a serious of

questions about the candidate’s previous
experience and/or goals for the future
 Generally open ended




Why do you want to be a physician?
Where do you see yourself in 5 or 10 years?
Identify strengths/weaknesses

 Can often be anticipated and prepared for by

the candidate

The Perfect Interview
 Would identify behavioral characteristics

deemed critical to the job at hand
 Would satisfy the usual psychometric criteria
of any selection process





Standardization
Reliability
Validity
Fairness

 Must discriminate fairly between applicants
 Methodology must be acceptable to

applicants

The Interview
 What are your goals in the interview?
 What questions go you usually ask?
 Do you get the information you need to meet

your goals?

The Ideal Interview
 Generally not designed to evaluate academic

qualifications; grades and scores suffice
 Rather, the perfect interview would help to
evaluate other domains






Integrity
Leadership
Empathy
Ability to work as a team
Work ethic

New Strategies
 Increasingly the business world is moving to

structured interview techniques
 Widespread belief that structured interviews
offer better insight and more consistency of
result; easier to compare candidates and
more reliable
 Two strategies have gained popularity:



Situational Judgement Tests
Behavioral Based Interviewing (BBI)

Objectives for Today
 Encourage you to think about what you are

really looking to evaluate in an interview and
whether or not your current strategies are
working
 Review and contrast two popular structured
interview techniques
 Outline basic constructs for developing
questions in each technique

Situational Judgement Tests
 Type of psychological test which

presents the applicant with realistic,
hypothetical scenarios
 Goal is to assess judgement required for
problem solving in typical work situations
 Applicant asked what he/she would do in
specific situations
 Generally multiple choice format
 Goal is to ask questions that directly assess
relevant job related behaviors

Situational Judgement Tests
 Unlikely that practice can improve

performance as answers are not always
arrived at logically
 Can tap into a wide range of constructs from
problem solving and decisional skills to
interpersonal constructs

Item Characteristics
 SJT items vary widely in format.
 Like most forms of multiple choice items, they

have a stem and several responses.
 Item stem: Everyone in your work group has
received a new computer except you. What
would you do?
 Item responses are possible actions.

Example Question 1
This morning, you found a fax in your in-box
that seems to concern a colleague's home
business. She normally does not use the fax
for business purposes.
 The most effective response to this situation
would be:
 The least effective response to this situation
would be:

Answers:
a) Politely, tell your co-worker that you will inform the

b)
c)
d)
e)

manager the next time you catch her using office
resources for private business.
Report the incident to the manager.
Put the fax in the manager's mailbox without saying
anything to anyone.
Put the fax in your co-worker's mailbox without
saying anything to anyone.
Give the fax to your co-worker and remind her that
office equipment is not supposed to be used for
personal use.

Example Question 2
You and a co-worker are working on a complex project
that demands a great deal of effort from both of you.
Your co-worker is frequently absent as a result of
burnout and stress from his personal problems. You
do not know much about the circumstances, nor have
you known him for long. Your co-worker contributes
very little to the project, and, as a result, you are
putting in an excessive amount of overtime in order to
keep the project moving ahead. You feel that your
health may begin to suffer if you continue to work as
many hours.
 The most effective response:
 The least effective response:

Answers to Sample Question 2
a) Ask other co-workers to help you manage
b)

c)
d)
e)

your workload.
Raise the issue with your manager and
request additional help to ensure that the
project is completed on schedule.
Meet with your co-worker to request that he
does his share of the work.
Continue to put in overtime to keep the
project moving ahead.
Offer to help your co-worker deal with his
personal problems.

Are Such Questions Useful?
 Depends on how the questions are written
 Requires careful identification of what types

of judgment the examiner is looking for
 Need organizational consensus on what is
the right answer
 Biggest benefit is that once developed, the
tests are standardized and not subject to
interpretation

Overview of SJT Test
Development
 Identify a job or job

 Generate item

class for which a
responses
SJT is to be
 Edit item responses
developed
 Determine response
 Write critical
instructions
incidents
 Develop a scoring
 Sort critical incidents
key
 Turn selected critical
incidents into item
stems

Identifying Critical Incidents
 Think about a time when someone did a

really good job.
 Think about a time when someone could
have done something differently.
 Think of a recent work challenge you faced
and how you handled it.
 Think of something you did in the past that
you were proud of.

Turn Critical Incidents into Stems
 Working from the critical incidents, write item

stems.
 Consider asking the same question in
different ways or scenarios.
 The same item does not need to be written
twice, but you need to decide how redundant
the items are permitted to be.

Components of Questions
 Stem Length
 Stem Complexity
 Response Instructions
 Evaluation of the responses

Stem Length
 Some stems are very short (You find the fax

in your mailbox).
 Other stems present very detailed
descriptions of situations (Your coworker is
under stress and is not carrying his end of the
load; you are now feeling burned out and
falling behind in the project)

Stem Complexity
 Complexity: Stems vary in the complexity of

the situation presented.
 Low complexity: One has difficulty with a new
assignment and needs instructions.
 High complexity: One has multiple
supervisors who are not cooperating with
each other, and who are providing conflicting
instructions concerning which of your
assignments has highest priority.

Response Instructions
 Variety of ways to instruct the applicants to

respond:







What would you most likely do?
What would you most likely do? What would
you least likely do?
Pick the best answer.
Pick the best answer and then pick the worst
answer.
Rate each response for effectiveness.
Rate each response on likelihood that you
would do the behavior.

Response Instructions
 The various response instructions fall into two

categories:



Knowledge
Behavioral tendency

Generate Item Responses
 Have subject matter experts with different

levels of experience/expertise write additional
responses for each stem.
 Prompts for writing responses:




What would you do?
What is the best thing to do?
What is a bad response that you think many
people would do?

Response Instruments and Faking
 Applicants may recognize that what they

would most likely do is not the most effective
response.
 Some applicants may choose to misrepresent
their behavioral tendency.

Behavioral Interviewing
 Designed to assess specific skills
 Touted as being better at assessing how an

applicant will perform on the job
 Commonly used in business sector
 Much less commonly employed in the field of
medicine
 Principle of the technique is that the best
predictor of future behavior is past behavior

Behavioral Interviewing
 When asked yes/no, or hypothetical

questions, the applicant can tell the
interviewer what he/she wants to hear
 Eg: If asked, “what would you do if the
deadline on a project was moved up”, answer
would be to put in extra time
 Easy for the applicant to guess the “right
answer”
 Difficult for the interviewer to assess what a
candidate would actually do

Behavioral Interviewing
 Compare to: What have you done in the past

when you needed to meet a tight deadline?
 Follow with probing questions:





How many hours did you put in?
Were you successful in completing the task?
How did you feel about having to put in
overtime?
How were you compensated?

Behavioral Interviewing
 Applicant is asked a series of standardized

questions
 Designed to get the applicant to respond to
how he/she handled specific situations in the
past
 Candidate is expected to describe situations
from the past and his/her feelings and/or
observations about the situation
 Allows the interviewer to assess proficiency in
one or more job related areas

Steps in the Process
 First step in the process is the identification of

the skills that you are looking for



Which are the most important?
Which are the least critical?

 Once the skills are identified, need to develop

questions that will ask about behaviors that
reflect the desired skills
 Then need to analyze the answers


How do the answers reflect the needed skills?

Identification of Skills
 Name five skills that you want in a future

physician (ie a medical student)
 Rank them in order of importance
 How does your ranking change if the
applicant wants to go into:





Internal Medicine?
Pediatrics?
Pathology?

 Should the skill set be differentiated at this

level?

Sample Skills Analysis
 Coping

 Goal Setting

 Tolerance of Ambiguity

 Written Communication

 Decisiveness

 Commitment to Task

 Spoken Communication

 Interaction

 Assertiveness

 Perceptivity

 Energizing

 Organization and Planning

 Policy and Procedure

 Creativity

 Alertness

 Versatility

 Analytical Problem Solving

 Reading the System

 Decision Making and

 Team Building

Problem Solving
 Leadership

Competencies
 Another way to approach this issue is to

consider the six ACGME competencies







Medical knowledge
Patient care
Interpersonal Communication skills
Professionalism
Practice Based Learning
Systems Based Learning

Developing the Questions
 Classically begin with opening statements

such as:



Tell me about a time when. . . .
Can you describe a situation where . . . .

 Can be casually inserted into an interview OR

more formally asked
 Generally followed up with more questions to
more specifically explore all aspects of the
situation

Typical Questions: Identify the Skill
 Describe a difficult problem that you tried to solve;

how did you identify the problem? How did you go
about trying to solve it?
 Describe a time when you tried to persuade someone

to do something that they were not inclined to do?
 Describe a time you decided on your own that

something needed to be done; what do you do to get
it done?

Practice Interviews
 Pair off in twos
 Each pair should identify:



The applicant
The interviewer

 Take 10 minutes and “conduct” the interview

 Reflect on how the interview went



As the applicant, how did the interview feel?
As the interviewer, what kind of knowledge
about the “applicant” did you obtain?