Data reliability and procedural fidelity

Data are reliable when two trained observers watching the same session record the same thing, which is checked with interobserver agreement (IOA). Procedural fidelity, also called treatment integrity, is a separate check of whether the plan was carried out as written. Both protect the decisions a supervisor makes from the data.

What does task A.8 cover?

Task A.8 asks you to implement procedures that keep data accurate and treatment delivered as planned. That includes taking part in IOA checks, recording independently of the other observer, collecting data in the moment rather than from memory, guarding against drift from the definition, and being observed for procedural fidelity without changing how you normally work.

How do IOA and procedural fidelity differ?

IOA compares two people's data on the learner's behavior to see whether the measurement is trustworthy. Procedural fidelity compares what the implementer actually did with what the written plan says, to see whether the intervention was delivered correctly. You can have perfect IOA on a session where the plan was run wrongly, so neither check replaces the other.

Interobserver agreementProcedural fidelity
QuestionDid two observers record the same behavior?Was the plan carried out as written?
Who is measuredThe data on the learnerThe implementer's steps
ExampleBoth observers count out-of-seat behaviorSupervisor checks each 3-second wait before a prompt
ResultPercentage agreementPercentage of steps done correctly

How do you calculate common IOA types?

For total count IOA, divide the smaller count by the larger and multiply by 100: counts of 16 and 20 give 80%. For interval-by-interval IOA, divide intervals with agreement by total intervals and multiply by 100: 17 agreements out of 20 intervals give 85%. Your supervisor sets the acceptable level and how often checks happen.

  • Total count IOA: smaller count / larger count x 100.
  • Interval-by-interval IOA: agreements / (agreements + disagreements) x 100.
  • Total duration IOA: shorter duration / longer duration x 100.

What threatens reliable data in everyday sessions?

Observer drift is a slow, unnoticed shift away from the definition, such as starting to count light taps as hitting months into a case. Recording from memory at the end of the session invites errors. Looking at the other observer's sheet during an IOA check makes agreement meaningless. Expectations about how the learner should be doing can also bend what you record.

Worked scenario at a clinic: during an IOA check, the second observer glances at the primary technician's tally. The resulting 100% agreement says nothing about reliability, because the observers were no longer independent. The honest step is to tell the supervisor and redo the check.

Worked scenario at home: a plan calls for a 3-second wait before each prompt, but the technician usually prompts after about 1 second. A fidelity check would show this, and the fix is retraining and practice, not changing the plan to match the habit.

Protecting data quality, step by step

Home example: before a session a technician re-reads the definition of hitting, which excludes high fives and accidental contact during play. When the learner taps her arm to get attention she checks the definition rather than guessing, and asks her supervisor at the next check-in whether taps should count.

None of these habits are about catching anyone out. They exist because a supervisor deciding whether to fade a prompt or change a reduction plan is only as right as the numbers in front of them.

  • Re-read operational definitions regularly, not only at the start of a case.
  • Record each response as it happens, or at the planned recording points.
  • During IOA, position yourself so you cannot see the other observer's sheet, and do not discuss scoring until the end.
  • Ask for a definition review if you notice yourself hesitating over what counts.
  • Welcome fidelity checks and ask for feedback on any step you are unsure of.
  • Report honestly when you missed data, rather than reconstructing it.

What is the RBT's part versus the BCBA's?

The supervisor schedules IOA and fidelity checks, sets acceptable thresholds, calculates or reviews the results, and retrains or revises definitions when agreement or fidelity is low. The technician records independently and accurately, follows procedures as written, accepts feedback without defensiveness, and raises definition problems early. Low agreement is a signal to fix the system, not a personal failing to hide.

Practice questions

Question 1 · Data collection and graphing

Two observers count Lena's out-of-seat behavior during the same session. One records 16 instances and the other records 20. What is the total count IOA?

Question 2 · Data collection and graphing

Two observers record the same 20 intervals. They agree on 17 intervals and disagree on 3. What is the interval-by-interval IOA?

Question 3 · Data collection and graphing

Months into a case, a technician has started counting light taps as "hitting," though the definition requires forceful contact. Nobody told her to change. What is this called?

Question 4 · Data collection and graphing

Elena's plan calls for a 3-second wait before each prompt, but her technician usually prompts immediately. After a month, the data show little progress. Why is this a problem for the team's decisions?

Question 5 · Data collection and graphing

During an IOA check, the second observer glances at the primary technician's tally sheet a few times to "stay on track." Why does this weaken the check?

Questions people ask

Does high IOA prove the data are accurate?

No. Two observers can agree and both be wrong. High IOA shows consistency, which is necessary but not sufficient for accuracy.

Is procedural fidelity the same as treatment integrity?

Yes. The terms are used interchangeably for how closely a plan is carried out as written.

What if my IOA score is low?

Tell your supervisor. The usual next steps are reviewing the definition and practising together, not adjusting your own counts.

How ready are you?

Ten questions, about 3 minutes, no account. Get your readiness score and your weakest domains.

Check my readiness