VR Got Two Trainees to Criterion on Functional Analysis. Both Failed the Live Test.

Clay and colleagues trained six preservice clinicians — all women in their early twenties, two already RBTs — to run functional analysis conditions in a virtual reality simulation, then tested whether the skill survived contact with a human being.

It mostly did not, and which trainees it failed for is the finding.

What happened

Baseline accuracy was 0–58% correct in the first cohort and 0–67% in the second. VR with immediate prompting produced an immediate level change for every participant. Two of the six, Ofelia and Mercedes, hit criterion across all conditions on VR alone, so they skipped the feedback phase and went straight to the transfer probe. The other four needed brief feedback.

Then a researcher role-played the client. From the paper: “Mercedes and Ofelia scored below criterion across all conditions in the treatment-extension phase; their scores ranged from 14% to 83% correct. In contrast, Carmen scored above criterion across all conditions in the treatment-extension phase; her scores ranged from 91% to 100% correct.”

The two who looked strongest in the simulation were the two who had no feedback loop, and they are the two the live test caught.

The money, since that is why anyone asks

The authors published a cost table. VR: headset $300, laptop $699, simulation $15,000 — $15,999, once. In person: “$550 per person (20 staff per clinic) … Every training (2x per year),” $22,000, recurring. Their note says the in-person figure was “pulled from large international behavioral consultation provider.”

But the authors also flag what the table leaves out: “our cost analysis did not account for the time required to deliver feedback, which was brief (1–2 min per session) but variable across participants.” The cheaper column is only cheaper if you do not need the thing it omits, and four of six trainees did.

Read it against the video-scoring study

On September 2 we covered a study in which trainees reached discrete-trial-instruction mastery purely by scoring videos, with no modeling, rehearsal or feedback. This one points the other way: the simulation produced in-simulation mastery, and in-simulation mastery did not transfer for the two who never got feedback. The two findings are not in conflict about whether low-cost training works. They disagree about whether a score inside the training environment is evidence of anything outside it.

The limits, in the authors' terms

“First, baseline responding was not consistently stable across participants … fewer data points limit the strength of conclusions. Second, we could not analyze error types in all control condition sessions due to corrupted video files. Third, social validity data were not formally collected.” They add that the goal of reducing reliance on trainers was not met — “personnel were still needed to provide feedback and collect data” — that baseline data were not collected in the transfer context, that only role-play was used, and that participants “could not physically interact with the virtual client (e.g., blocking SIB), which limits training in safety-related procedures.”

And the size of the comparison that carries the headline: the transfer contrast is three people in one cohort. Every participant in the second cohort went through feedback, so there is no no-feedback comparison there at all.

What you must know or do

• Do not sign off competency from a score earned inside the training tool. Whatever your simulation, video module or e-learning reports, the criterion that counts is performance with a live person. Two trainees here passed the tool and failed the room.
• Keep the feedback step even for the trainees who do not appear to need it. In this study those were exactly the two who did not transfer, and they skipped feedback because they had scored well.
• If you are pricing a simulation against in-person training, add the supervisor's feedback time to the simulation column. The authors say plainly that theirs does not include it.
• Do not use simulation for anything you would physically block. The trainees could not interact with the virtual client, so SIB blocking and other safety procedures are untrained by this method.