Diagnosis of axillary lymph node status in primary breast cancer without a surgical procedure - a human factors validation test

ISRCTN	ISRCTN99301435
DOI	https://doi.org/10.1186/ISRCTN99301435
ClinicalTrials.gov (NCT)	Nil known
Clinical Trials Information System (CTIS)	Nil known
Protocol serial number	Nil known
Sponsor	Region Skåne/South Swedish Health Care Region
Funders	The Governmental Funding of Clinical Research within the National Health Services Sweden, The Swedish Cancer Society, Lunds University, Region Skåne/South Swedish Health Care Region, The Swedish Research Council, The Erling Persson Foundation

Submission date: 10/11/2024
Registration date: 15/11/2024
Last edited: 16/12/2024

Recruitment status: No longer recruiting
Overall study status: Completed
Condition category: Cancer

Prospectively registered

Protocol

Statistical analysis plan

Results

Individual participant data

Record updated in last year

Plain English summary of protocol

Background and study aims
NILS (Non-invasive Lymph node Staging) is a web-based software that uses artificial intelligence to predict sentinel lymph node status in women who are planned for primary surgery due to breast cancer and are clinically axillary lymph node-negative. NILS is not yet CE-marked and is not available on the market. Previously, formative usability engineering evaluation has been performed with NILS. This study is a summative evaluation study in preparation of market access. The main aim is to present evidence that the NILS interface is considered safe. The study will be conducted according to the standard IEC 62366-1:2015. For the purpose of this pre-market study, a study under simulated conditions will be performed.

Who can participate
This study is for licensed medical physicians specialized in general surgery or oncology (including medical and radiation oncologists).

What does the study involve?
During the simulated-use testing, the test participant will be given the opportunity to use the web-calculator independently and as naturally as possible in their own work environment, without unnecessary interference from the test leader or observer. The test participants will be asked to perform five scenarios, with five different simulated patient cases. After calculating each case using NILS, the test participants will fill in the appropriate clinical pathway based on the NILS result and other available information. The participants’ interaction with the interface will be controlled by a test leader and an observer who will follow a predefined protocol. All tasks have been identified (risk management procedure according to ISO 14971) and critical tasks specified.

The study will take place in three different health care regions and 4 different hospitals in Sweden. The conditions under which the simulated-use testing will be conducted will be sufficiently realistic so that the results of the testing are generalizable to actual intended use. Therefore, fabricated, but realistic, patient cases will be used for the testing. In addition, cases are created to ensure that all critical tasks are performed during the test and that the test conditions are sufficiently realistic to represent actual conditions of use.

After each case the test participant will be asked to fill in the clinical pathway that they find most appropriate based on the information they have received and the NILS calculation. Several pathways can be appropriate: “Consider omitting SLNB”, “Consider performing SLNB”, “Definitely perform SLNB”, and “NILS cannot assist in making the clinical decision”. The test participants will following be asked to fill in a validated questionnaire about their experience of the device. Interview questions will be asked after the use of the interface if any tasks were incorrectly or close to incorrectly executed.

What are the possible benefits and risks of participating?
The participants will receive a small symbolic compensation for the participation. No risks of participating have been identified.

Where is the study run from?
Lunds University Cancer Center in cooperation with the South Swedish Health Care Region.

When is the study starting and how long is it expected to run for?
November 2024 to June 2025

Who is funding the study?
1. The Governmental Funding of Clinical Research within the National Health Services (Sweden).
2. The Swedish Cancer Society (Sweden).
3. Lunds University (Sweden).
4. Region Skåne/South Swedish Health Care Region (Sweden)
5. The Swedish Research Council (Sweden)
6. The Erling Persson Foundation (Sweden).

Funding resources had no role in the study design, data collection, analyses, data interpretation, writing of the manuscript, or the decision to submit the manuscript for publication.

Who is the main contact?
MD Anna Allfelt, anna.allfelt@med.lu.se
MD, PhD Ida Skarping, ida.skarping@med.lu.se

Contact information

Prof Lisa Rydén
Principal investigator

Bröst och melanommottagning, SUS
Tora Kjellgrens gata 27
Malmö
205 02
Sweden

ORCID ID	0000-0001-7515-3130
Phone	+46 (0)46176241
Email	lisa.ryden@med.lu.se

Dr Ida Skarping
Scientific

Klinisk fysiologi och nuklearmedicin, SUS
Entrégatan 7
Lund
221 85
Sweden

ORCID ID	0000-0002-1265-8649
Phone	+46 (0)46177309
Email	ida.skarping@med.lu.se

Mrs Anna Allfelt
Public

Bröst och melanommottagning, SUS
Tora Kjellgrens gata 27
Malmö
205 02
Sweden

ORCID ID	0009-0002-7889-9821
Phone	+46 (0)40335357
Email	anna.allfelt@med.lu.se

Study information

Primary study design	Observational
Study design	Pre-market human factors validation test under simulated conditions.
Secondary study design	Human factor vaildation test
Study type	Participant information sheet
Scientific title	Human factors validation study of an artificial neural network‑based preoperative decision‑support tool for noninvasive lymph node staging (NILS) in women with primary breast cancer
Study objectives	From a usability point-of-view evaluating the final product with intended-use testing, the hypothesis is to present evidence that the interface for NILS is considered safe.
Ethics approval(s)	Ethics approval not required
Ethics approval additional information	This usability study is not a clinical study, and it does not require an ethical approval from the Swedish Ethical Review Authority. The usability study is approved by the Head of Research in Region Skåne regarding the intended conduction of the study and Region Skåne is thus the legal authority for the study.
Health condition(s) or problem(s) studied	Pre-market human factor validation test with intended (actual) users.
Intervention	The test participants will be greeted by the test leader and be provided a written instruction for use manual. The test participants will be presented with five simulated cases and are expected to fill in the patient information correctly and press calculate. The test participants are then expected to interpret the result of the calculation correctly and consider the calculated results of probability of a benign axillary nodal status together with other sources of information when subsequently determining the most appropriate clinical pathway for the patient. The hypothetical treatment decision per se will not be evaluated. This will be done five times covering all the cases. The test participants will following be asked to fill in a validated questionnaire about their experience of the device. Lastly a short oral interview will take place if indicated. The total time required per test-participant is approximately 45-60 min. No follow-up is planned.
Intervention type	Other
Primary outcome measure(s)	1. Frequency of observed outcome and result during the test, analysis per task. Observed outcome is defined as correct use, close call, use difficulty, and use error. Result is defined as passed or failed. 2. Frequency of passed critical tasks per scenario. 3. Frequency of observed outcome and result during the test, analysis per task description in association with potential harm (false estimated indication of benign sentinel lymph node). Observed outcome is defined as correct use, close call, use difficulty, and use error. Result is defined as passed or failed. 4. Average point (median) on Likert scale 1-5 and 1-7: 1 “strongly disagree” to 5 respectively 7 “strongly agree” in validated questionnaire (Lewis J.R 2018 and 1995). 5. Review of all comments in validated questionnaire (Lewis J.R 2018 and 1995) and interview data with the purpose to detect potential safety and performance concerns. Since NILS has not yet been released to the market and has had limited contact with the intended users, it is likely that some usability issues may be detected. However, a large amount of user errors is not acceptable. Therefore, the team has set the acceptance criteria to 90%. Each test protocol interview data will be reviewed and all comments with potential safety and performance concerns will be analyzed to assess any required changes in design. In case any significant change of the NILS web page will be needed, the change will be re-assessed using a new scenario with the purpose to re-evaluate the potential concern. The qualitative evaluation will be acceptable when no potential safety and performance concerns remain to be analyzed or addressed.
Key secondary outcome measure(s)	There are no secondary outcome measures
Completion date	30/06/2025

Eligibility

Participant type(s)	Health professional
Age group	Adult
Lower age limit	18 Years
Upper age limit	65 Years
Sex	All
Target sample size at registration	20
Key inclusion criteria	1. Intended users - licensed medical physicians specialized in general surgery or oncology (including medical and radiation oncologists) 2. >18 years 3. Both sexes
Key exclusion criteria	1. Any former contact with the device
Date of first enrolment	12/09/2024
Date of final enrolment	31/12/2024

Locations

Countries of recruitment

Sweden

Study participating centres

Skånes universitetssjukhus Malmö

Inga Marie Nilssons gata 47
Malmö
214 28
Sweden

Skånes universitetssjukhus Lund

Entrégatan 7
Lund
222 42
Sweden

Centralsjukhuset Kristianstad

J A Hedlunds väg 5
Kristianstad
291 33
Sweden

Blekingesjukhuset Karlskrona

Lasarettsvägen
Karlskrona
371 41
Sweden

Centrallasarettet Växjö

Strandvägen 8
Växjö
352 34
Sweden

Results and Publications

Individual participant data (IPD) Intention to share	Yes
IPD sharing plan summary	Available on request
IPD sharing plan	The dataset generated and analyzed during the current study will be available upon reasonable request from Anna Allfelt, anna.allfelt@med.lu.se

Study outputs

Output type	Details	Date created	Date added	Peer reviewed?	Patient-facing?
Other files	Instructions for participants		15/11/2024	No	No
Other files	Instructions for use		15/11/2024	No	No
Participant information sheet	Participant information sheet	11/11/2025	11/11/2025	No	Yes
Study website	Study website	11/11/2025	11/11/2025	No	Yes

Additional files

46376 Instructions for use.pdf: Instructions for use
46376 Instructions for participants.pdf: Instructions for participants

Editorial Notes

16/12/2024: The sponsor email was changed.
15/11/2024: Trial's existence confirmed by Region Skane.