Human factors validation testing is the final usability testing to be completed prior to regulatory submission. A mistake or an omission may result in follow-up testing being required if you uncover unanticipated use errors during testing. Careful planning is highly recommended.
The following aspects should be considered for human factors validation planning. Each topic could be a chapter unto itself but for now we will keep it succinct.
A good starting point for more information is of course the standards, IEC 62366-1, IEC/TR 62366-2 and the FDA guidance document: Applying Human Factors and Usability Engineering to Medical Devices. This blog covers:
- Review how the test environment and the patient will be simulated
- Review the user profiles for optimization
- Understand hazard related use scenarios and critical tasks
- Establish representative training plans
- Plan for a dry run
- Seek advice, confirm the protocol
How will the Test Environment and the Patient be Simulated?
The use environment as per 62366-1 is the “actual conditions and settings in which users interact with the medical device”. Established early in the project through usability evaluations, the use environment will need to be translated into a test environment. Requirements for the test environment will have to factor in aspects such as lighting level, temperature, noise, distraction, and interactions with other equipment. The important aspects of the use environment that may contribute to use errors should be included in the test environment.
ANSI/AAMI HE75:2009 Human Factors Engineering – Design of Medical Devices, can be a start to find guidance or you may have to review literature to find requirements for your use environment. Some human factors validation testing might be able to be simulated in an office environment, while others may necessitate sourcing of an operation room or imaging suite. Regardless, ensure you plan time and budget to renting an appropriate space. Keep in mind that the location of your chosen use environment may impact recruitment.
The simulated use environment is only one part of setting up an effective test environment. As part of the use specification, the patient population and “intended part of the body or type of tissues applied to or interacted with[1]” must be defined. The patient and anatomical interface need to be reflected in the human factors validation test setup.
For many medical devices it is neither feasible nor practical to use human subjects as part of the test setup. Simulated patients such as manikins (anatomical models that simulate properties of tissue, bone and morphology) may be appropriate. Take into account how the chosen materials may affect the way users interact with the medical device. Does the setup give the same visual cues, tactile response? For example, when looking at imaging modalities, simulation models require adequate visual cues for users to navigate the simulated anatomy so that any difficulties observed are due to the medical device and not testing artifacts.
Review User Profiles for Optimization
User profiles should be established early in the project as part of the use specification. These should be reviewed in combination with the task analysis to determine how many distinct user profiles will be tested. It may be possible to create user groups in which two or more user profiles are combined. This is only possible if they are similar enough in experience / education and the tasks that they typically perform. A good example would be scrub nurses and scrub technicians. In some situations, they may perform similar duties in the operating room and combining them into a group may be possible. This has the benefits of simplifying recruitment and keeping the testing focused.
Understand Hazard Related Use Scenarios and Critical Tasks
One of the most important activities to finalize prior to creating a human factors validation protocol is a clear and thorough understanding of the use scenarios that can result in use errors (hazard-related use scenarios) using use related risk analysis (URRA). One method to do this is to perform a task analysis combined with a uFMEA to establish use errors, associated harms and mitigations. Previous formative analyses should be used to inform the potential use errors for each task. A validation method can be attributed for all use errors to establish the effectiveness of the risk mitigation measures. Done in collaboration with the team and clinicians, this can be a powerful method to establish the hazard related use scenarios. Note the word “team” – while the human factors team may lead the URRA activity, input must be gathered from the design team and key stakeholders (e.g. clinicians, patients).
For human factors validation one can either test: 1) all hazard-related use scenarios, 2) a subset based on severity of the harm, or 3) a subset based on the severity of the harm and additional circumstances specific to the device. When using option 2) only the critical tasks are evaluated in human factors validation testing.
Critical tasks are identified as “a user task which, if performed incorrectly or not performed at all, would or could cause serious harm to the patient or user, where harm is defined to include compromised medical care”[2]. For a definition of serious harm one can refer to 21 CFR 803.3 (w). Given these two definitions and using the five-point severity scale outlined in ISO 14971 one could infer that a use error that causes harm with a severity above 3 constitutes a critical task. Hazard related use scenarios in the URRA with critical tasks would be selected to be tested as part of human factors validation testing. Thus, identification of critical tasks is key to defining the scope of the human factors validations testing.
Establish Representative Training Plans
As part of the risk management process, training will often be used as a risk mitigation measure. Depending upon the nature of the training this can be relatively straightforward (e.g., a video), or complex (e.g., didactic training session followed by a hands-on training scenarios). These training sessions need to be advanced enough that they are representative of what will occur when the product is released. Don’t assume you know what training is best for your user. What works for you might be highly ineffective for your target user profile. Literature can be used to identify learning styles (e.g., David Kolb’s model, Fleming’s VAK/VARK model) that are most effective for your target user profile, in addition to surveying potential users. By tailoring your training to the user, you will establish successful mitigations for use errors. Like all mitigations it is always best to perform formative usability testing or dry run testing to establish some confidence that the training will be an effective mitigation. Introducing training first in the human factors validation testing is risky.
Plan for a Dry Run
At a minimum one should plan for an extensive dry run session internally within the organization to ensure that the human factors validation testing runs smoothly, timing is understood, and the prompts are unambiguous. However, if the test is complex and/or mitigations (e.g., training) have not been previously tested in formative evaluations, one may consider running a trial human factors validation test with a smaller user group. Keep in mind that for human factors validation testing you cannot use subjects that have been involved previously in usability testing on the product. If you use a subject for dry run testing you will not be able to use the same subject for human factors validation testing. Dry run testing with a small user group allows any design changes, label changes, and packaging changes to be tested to ensure no new use errors have been generated. Think of dry run testing as a mitigation to unforeseen complications during human factors validation testing.
Seek Advice, Confirm the Protocol
Finally, submit your human factors validation testing protocol to the FDA for their review. The FDA “encourages manufacturers to submit a draft of the human factors testing protocol prior to conducting the test so we can ensure that the methods you plan to use will be acceptable.” This process can be lengthy so factor that into the plan. However, given that human factors validation testing can be costly and does take significant effort for planning and execution, having the protocol evaluated to ensure methods are acceptable is an excellent way to ensure regulatory success.
The above gives a brief overview of some considerations for human factors validation testing planning. Starting with all of the required input and generating a plan will lead to more positive outcomes. Like an iceberg, testing is the part people notice but all the planning and information gathering is what keeps it afloat.
Happy testing,
[1] 62366-1:2016, Medical devices – Part 1: Application of usability engineering to medical devices
[2] FDA Guidance Document, Applying Human Factors and Usability Engineering to Medical Devices, Feb 3, 2016
Paul Hulme is a StarFish Medical Human Factors Engineer. His professional experience includes working with the Canadian Space Agency and 5 years in Switzerland working at Zimmer GmbH. Paul studied Mechanical Engineering at the University of Victoria, completed his Masters of Mechanical Engineering at the University of Calgary, and his PhD in Biomedical Engineering at the University of Bern.
