Interobserver agreement
Interobserver agreement, or IOA, is a check of how closely two people's data match when they record the same behavior in the same session. High agreement suggests the definition is clear and the data are trustworthy. Observers must record independently, without looking at each other's sheets.
What is interobserver agreement?
IOA compares data from two independent observers who watched the same session and used the same definition and method. The result is an agreement percentage. It supports the reliability of the data, meaning others would have recorded about the same thing, and it highlights definitions that need clearer wording.
How do you calculate IOA?
For total count IOA, divide the smaller count by the larger and multiply by 100. For interval-by-interval IOA, divide the number of intervals both observers scored the same by the total number of intervals, then multiply by 100. If two observers agree on 17 of 20 intervals, the IOA is 85 percent.
Total count example: one observer records 16 instances of out-of-seat behavior and the other records 20. The IOA is 16 divided by 20, which is 80 percent. Interval example: 17 agreements out of 17 plus 3 disagreements gives 85 percent.
What makes an IOA check valid?
The two observers must be watching the same session, using the same definition and recording system, without seeing each other's data. If the second observer glances at the first person's tally, the numbers are no longer independent and agreement is inflated. Supervisors often run IOA during overlap sessions.
- Same session, same definition, same method
- Observers record independently
- No comparing or discussing until both are done
What is observer drift?
Observer drift is a slow shift in how an observer applies a definition, so that over time they count slightly different things. A technician who starts scoring light taps as hitting, though the definition says otherwise, has drifted. Regular IOA checks, retraining, and rereading the written definition help catch and correct it.
If your counts and the supervisor's disagree, say so openly. The point of IOA is to find and fix differences in how a definition is used, not to find fault.
What exam traps involve IOA?
Common traps include the wrong formula for the data type, forgetting that observers must be independent, and treating a high IOA as proof the plan works. IOA only speaks to the reliability of the measurement. It says nothing about whether the intervention is effective. Recording from memory also weakens reliability.
Practice questions
A supervisor and a technician each record Kofi's hand mouthing in the same session, without looking at each other's data, and then compare. What is the main purpose?
Two observers count Lena's out-of-seat behavior during the same session. One records 16 instances and the other records 20. What is the total count IOA?
Two observers record the same 20 intervals. They agree on 17 intervals and disagree on 3. What is the interval-by-interval IOA?
During an IOA check, the second observer glances at the primary technician's tally sheet a few times to "stay on track." Why does this weaken the check?
Months into a case, a technician has started counting light taps as "hitting," though the definition requires forceful contact. Nobody told her to change. What is this called?
Questions people ask
What if my IOA is low?
Tell your supervisor. The definition may need clearer wording, or the team may need to review how the behavior is scored.
Is IOA the same as treatment integrity?
No. IOA checks that data match between observers. Treatment integrity checks that the plan is carried out as written.
How ready are you?
Ten questions, about 3 minutes, no account. Get your readiness score and your weakest domains.
Check my readiness