文献阅读笔记|影像组学方法论|Why did European Radiology reject my radiomic biomarker paper? How to correctly evaluate imaging biomarkers in a clinical setting

目录

• 研究的临床意义很重要
• 在多变量模型中评估影像biomarker加入后的意义
• 不要忽视影像之外的临床特征，即使他们很常见/不显著
• 样本量和特征量的关系
• 模型选择
• 模型评估
• 模型结论
• 定义——诊断模型？预后模型？预测模型？
• 其他观点

题目：Why did European Radiology reject my radiomic biomarker paper? How to correctly evaluate imaging biomarkers in a clinical setting
期刊：European Radiology
重要观点摘要：

研究的临床意义很重要

1、Early research should determine appropriate diagnostic cut points for the biomarker (e.g. positivity threshold, operating point), so these are agreed before larger studies begin. Subsequent multi-centre pragmatic studies should focus on whether the biomarker(s) works in the intended clinical setting.
2、We have noted that much radiomic research attempts to predict a clinical outcome, sometimes near future events like diagnosis, or later events, like survival. Predicting events of no clinical significance is irrelevant. For example, the clinical value of a model that predicts tumour genetics is minimal if the same information can be easily (and more accurately) obtained from a biopsy that is always performed routinely.

在多变量模型中评估影像biomarker加入后的意义

3、Assessment of multiple biomarkers is facilitated if they are combined within a multivariable model that yields a single result (e.g. “ response ” or “ no response ” ).
4、clinical assessment of an imaging biomarker is best achieved by incorporating it within a multivariable model, where its additive contribution to the overall result can be judged.
5、One of us assessed whether perfusion CT could predict rectal cancer metastasis by simply comparing baseline CT parameters between patients with and without subsequent metastases [16]; this is simply a measure of association, not prediction, and does not indicate clinical utility. While the paper was highly cited (N = 92), it did not influence clinical practice. This approach is only reasonable for earlystage development

不要忽视影像之外的临床特征，即使他们很常见/不显著

6、Ignoring important non-radiological variables is a common mistake we often identify during review.
7、While we encourage combining clinical and radiological variables, a balance must be struck between number and simplicity. Accordingly, final models should retain factors “ working hardest ” and discard those not “ pulling their weight ” . Retained radiomics and other imaging biomarkers must contribute as effectively as clinical variables.
8、Factors well established for a specific disease should always be included regardless of their statistical significance in a particular dataset.
9、Clinicians will be surprised to learn that statisticians might ignore statistical significance when selecting variables. Rather, the recommended approach is to ask clinical experts to indicate factors believed important, only then adding less established factors. Unfortunately, the worst but most widespread approach is to select factors via their significance in univariable analysis, followed by retaining those that remain significant within the model
10、A recent article pleaded that models include all relevant clinical data before adding “ omics ” .
11、It is imperative that models do not exclude factors simply because they are perceived as commonplace.

样本量和特征量的关系

12、 A simple “ rule of thumb ” for prognostic research requires a minimum of ten events per individual predictor investigated, (noting that this is an initial step and additional sample size considerations remain important). Moreover, this rule applies to the smaller group, i.e. if the sample comprises 100 breast cancer patients, 30 of whom respond, then just three predictors should be investigated. 【predictor太多，events太少，会导致1类错误概率增加】

模型选择

13、Because they are inherently multivariable, models are usually developed using multivariable regression, with linear regression used for continuous outcomes, logistic regression for binary outcomes, Cox proportional hazards for time-to-event outcomes (notably survival), and additional methods such as Lasso and ridge regression. There are also methods that fit to classification rules, such as neural network and machine learning methods.

模型评估

14、所有模型都必须被evaluate，内部验证是必须（Bootstrap 200-1000，Cross Validation，leave-out-one等），外部验证最好有
15、测试集和验证集划分不要随机划分，可选：按采集时间划分（前2/3训练，后1/3验证）；按采集机器划分（单机器数据训练，多机器验证）；按不同中心划分（一个中心数据训练，多中心验证）

模型结论

16、However, the final model equation recommended for clinical use should be presented so that others can evaluate it.
Publication without the equation is rather like a recipe that omits the quantities of ingredients! 【最好要有适用于临床的计算公式or诺莫图】
The equation is the final mathematical combination of variables and their weight. A simple example is the Nottingham Prognostic Index: Breast cancer survival = (0.2 × tumour diameter cm) + nodal stage + tumour grade. Lower scores predict better survival. Models must be simple to use and interpret or they will be ignored, even when accurate.
Model presentation is complex and should consider the intended clinical setting (are computers available?), who will use it (clinicians or laypeople?), and when (during a fraught emergency or calm outpatient clinic?). Online calculators facilitate complex equations. Output must be easy to understand, using points, groups, or graphs. Nomograms are slightly more complex.

定义——诊断模型？预后模型？预测模型？

17、Broadly, diagnostic models use reference data to establish current outcomes whereas prognostic models use follow-up data to identify “ true ” future outcomes. Oncologists often state that “ prognosis ” predicts outcomes independent of treatment (e.g. TNM stage), whereas “ prediction ” evaluates outcomes after treatment.

其他观点

18、We have stressed that biomarkers should not be assessed without factors already known to be useful and that it is generally more desirable to evaluate/update existing models than develop new models.
评估指标：AUC、ACC、SEN、SPE、PPV、NPV、F1score