Abstract
Artificial intelligence (AI) is currently regaining enormous interest due to the success of machine learning (ML), and in particular deep learning (DL). Image analysis, and thus radiomics, strongly benefits from this research. However, effectively and efficiently integrating diverse clinical, imaging, and molecular profile data is necessary to understand complex diseases, and to achieve accurate diagnosis in order to provide the best possible treatment. In addition to the need for sufficient computing resources, suitable algorithms, models, and data infrastructure, three important aspects are often neglected: (1) the need for multiple independent, sufficiently large and, above all, high-quality data sets; (2) the need for domain knowledge and ontologies; and (3) the requirement for multiple networks that provide relevant relationships among biological entities. While one will always get results out of high-dimensional data, all three aspects are essential to provide robust training and validation of ML models, to provide explainable hypotheses and results, and to achieve the necessary trust in AI and confidence for clinical applications.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.References
Marcus G. 2018. Deep learning: A critical appraisal. arXiv:1801.00631.
Zhang Q, Yang LT, Chen Z, Li P. A survey on deep learning for big data. Inf Fus 2018;42:146–57.
Komaroff AL. The variability and inaccuracy of medical data. Proc IEEE 1979;67(9):1196–207.
Kadmon G, Pinchover M, Weissbach A, Kogan Hazan S, Nahum E. Case not closed: prescription errors 12 years after computerized physician order entry implementation. J Pediatr 2017;190 :236–40.e2. https://doi.org/10.1016/j.jpeds.2017.08.013.
Holzinger A, Jurisica I. Knowledge discovery and data mining in biomedical informatics: the future is in integrative, interactive machine learning solutions. Interactive knowledge discovery and data mining in biomedical informatics: state-of-the-art and future challenges. Lecture notes in computer science LNCS 8401. In: Holzinger A and Jurisica I, editors. Springer; 2014. p. 1–18.
Wang B, et al. Similarity network fusion for aggregating data types on genomic scale. Nat Methods 2014;11 (3):3377.
Holzinger A. Biomedical informatics: discovering knowledge in big data. New York: Springer; 2014.
Hutson M. Artificial intelligence faces reproducibility crisis. Science 2018;359(6377):725–6.
Goodman SN, Fanelli D, Ioannidis JPA. What does research reproducibility mean? Sci Transl Med 2016; 8(341):341ps12.
Turkay C, Jeanquartier F, Holzinger A, Hauser H. On computationally-enhanced visual analysis of heterogeneous data and its application in biomedical informatics. Interactive knowledge discovery and data mining: state-of-the-art and future challenges in biomedical informatics. Lecture notes in computer science LNCS 8401. Springer; 2014. p. 117–140.
Collins FS, Varmus H. A new initiative on precision medicine. England J Med 2015;372(9):793–5.
Schiphof D, Oei EH, Hofman A, Waarsing JH, Weinans H, Bierma-Zeinstra SM. Sensitivity and associations with pain and body weight of an MRI definition of knee osteoarthritis compared with radiographic Kellgren and Lawrence criteria: a population-based study in middle-aged females. Osteoarthritis Cartilage 2014;22(3): 440–6.
Mlecnik B, Bindea G, Kirilovsky A, Angell HK, Obenauf AC, Tosolini M, Church SE, Maby P, Vasaturo A, Angelova M, Fredriksen T, Mauger S, Waldner M, Berger A, Speicher MR, Pages F, Valge-Archer V, Galon J. The tumor microenvironment and immunoscore are critical determinants of dissemination to distant metastasis. Sci Transl Med 2016;8(327):327ra26.
Sanduleanu S, Woodruff HC, de Jong EEC, van Timmeren JE, Jochems A, Dubois L, Lambin P. Tracking tumor biology with radiomics: a systematic review utilizing a radiomics quality score. Radiotherapy and Oncology: Journal of the European Society for Therapeutic Radiology and Oncology 2018;127(3):349– 60.
Vargas HA, Veeraraghavan H, Micco M, Nougaret S, Lakhman Y, Meier AA, Sosa R, Soslow RA, Levine DA, Weigelt B, Aghajanian C, Hricak H, Deasy J, Snyder A, Sala E. A novel representation of inter-site tumour heterogeneity from pre-treatment computed tomography textures classifies ovarian cancers by clinical outcome. Eur Radiol 2017;27(9):3991– 4001.
Yin Q, Hung SC, Rathmell WK, Shen L, Wang L, Lin W, Fielding JR, Khandani AH, Woods ME, Milowsky MI, Brooks SA, Wallen EM, Shen D. Integrative radiomics expression predicts molecular subtypes of primary clear cell renal cell carcinoma. Clin Radiol 2018;73(9):782–91.
Liao Xin, Bo Cai, Tian Bin, Luo Yilin, Song Wen, Li Yinglong. 2019. Machine-learning based radiogenomics analysis of MRI features and metagenes in glioblastoma multiforme patients with different survival time. J Cell Mol Med, 4.
Danala G, Thai T, Gunderson CC, Moxley KM, Moore K, Mannel RS, Liu H, Zheng B, Qiu Y. Applying quantitative CT image feature analysis to predict response of ovarian cancer patients to chemotherapy. Acad Radiol 2017;24(10):1233–9.
Cook GJR, Azad G, Owczarczyk K, Siddique M, Goh V. Challenges and promises of PET radiomics. Int J Radiat Oncol Biol Phys 2018;102(4):1083–89.
Lin G, Lai CH, Yen TC. Emerging molecular imaging techniques in gynecologic oncology. PET Clin 2018;13(2):289– 99.
Xu Y, Hosny A, Zeleznik R, Parmar C, Coroller T, Franco I, Mak RH, Aerts HJWL. 2019. Deep learning predicts lung cancer treatment response from serial medical imaging. Clinical Cancer Research.
Baessler B, Weiss K, Pinto Dos Santos D. Robustness and reproducibility of radiomics in magnetic resonance imaging: a phantom study. Invest Radiol 2019;54(4):221–8.
Lau SK, Boutros PC, Pintilie M, Blackhall FH, Zhu CQ, Strumpf D, Johnston MR, Darling G, Keshavjee S, Waddell TK, Liu N, Lau D, Penn LZ, Shepherd FA, Jurisica I, Der SD, Tsao MS. Three-gene prognostic classifier for early-stage non small-cell lung cancer. J Clin Oncol 2007;25 (35):5562–9.
Shedden K, Taylor JM, Enkemann SA, Tsao MS, Yeatman TJ, Gerald WL, Eschrich S, Jurisica I, Giordano TJ, Misek DE, Chang AC, Zhu CQ, Strumpf D, Hanash S, Shepherd FA, Ding K, Seymour L, Naoki K, Pennell N, Weir B, Verhaak R, Ladd-Acosta C, Golub T, Gruidl M, Sharma A, Szoke J, Zakowski M, Rusch V, Kris M, Viale A, Motoi N, Travis W, Conley B, Seshan VE, Meyerson M, Kuick R, Dobbin KK, Lively T, Jacobson JW, Beer DG. Gene expression-based survival prediction in lung adenocarcinoma: a multi-site, blinded validation study. Nat Med 2008;14(8):822–7.
Boutros PC, Lau SK, Pintilie M, Liu N, Shepherd FA, Der SD, Tsao MS, Penn LZ, Jurisica I. Prognostic gene signatures for non-small-cell lung cancer. Proc Natl Acad Sci USA 2009;106(8):2824–8.
Beaulieu-Jones BK, Greene CS. Reproducibility of computational workflows is automated using continuous analysis. Nat Biotechnol 2017;34(4):342–6.
Munafò MR, Nosek BA, Bishop VMD, Button SB, Chambers CD, du Sert N, Simonsohn U, Wagenmakers E-J, Ware JJ, Ioannidis JPA. A manifesto for reproducible science. Nat Hum Behav 2017; 1(1):s41562–016–0021.
Lamb J, Crawford ED, Peck D, Modell JW, Blat IC, Wrobel MJ, Lerner J, Brunet J-P, Subramanian A, Ross KN, Reich M, Hieronymus H, Wei G, Armstrong SA, Haggarty SJ, Clemons PA, Wei R, Carr SA, Lander ES, Golub TR. The connectivity map: using gene-expression signatures to connect small molecules, genes, and disease. Science 2006;313(5795):1929–35.
Sun R, Limkin EJ, Vakalopoulou M, Dercle L, Champiat S, Han SR, Verlingue L, Brandao D, Lancia A, Ammari S, Hollebecque A, Scoazec JY, Marabelle A, Massard C, Soria JC, Robert C, Paragios N, Deutsch E, Ferte C. A radiomics approach to assess tumour-infiltrating cd8 cells and response to anti-pd-1 or anti-pd-l1 immunotherapy: an imaging biomarker, retrospective multicohort study. Lancet Oncol 2018;19(9):1180–91.
Aerts HJ, Grossmann P, Tan Y, Oxnard GR, Rizvi N, Schwartz LH, Zhao B. Defining a radiomic response phenotype: a pilot study using targeted therapy in NSCLC. Sci Rep 2016;6:33860.
Tokar T, Pastrello C, Ramnarine VR, Zhu CQ, Craddock KJ, Pikor LA, Vucic EA, Vary S, Shepherd FA, Tsao MS, Lam WL, Jurisica I. Differentially expressed microRNAs in lung adenocarcinoma invert effects of copy number aberrations of prognostic genes. Oncotarget 2018;9(10):9137–55.
Viswanath SE, Tiwari P, Lee G, Madabhushi A. Dimensionality reduction-based fusion approaches for imaging and non-imaging biomedical data: concepts, workflow, and use-cases. BMC Med Imaging 2017;17(1):2.
Ingalhalikar M, Smith A, Parker D, Satterthwaite TD, Elliott MA, Ruparel K, Hakonarson H, Gur RE, Gur RC, Verma R. Sex differences in the structural connectome of the human brain. Proc Natl Acad Sci USA 2014;111(2):823–8.
Bigler ED. Systems biology, neuroimaging, neuropsychology, neuroconnectivity and traumatic brain injury. Front Syst Neurosci 2016;10:55.
Cui LB, Liu L, Wang HN, Wang LX, Guo F, Xi YB, Liu TT, Li C, Tian P, Liu K, Wu WJ, Chen YH, Qin W, Yin H. Disease definition for schizophrenia by functional connectivity using radiomics strategy. Schizophrenia Bull 2018;44(5):1053–9.
Jeong H, Mason SP, Barabasi AL, Oltvai ZN. Lethality and centrality in protein networks. Nature 2001;411(6833):41–2.
Przulj N, Wigle DA, Jurisica I. Functional topology in a network of protein interactions. Bioinformatics 2004;20(3):340–8.
Ye P, Peyser BD, Pan X, Boeke JD, Spencer FA, Bader JS. Gene function prediction from congruent synthetic lethal interactions in yeast. Molec Syst Biol 2005;1:1.
Sharan R, Ulitsky I, Shamir R. 2007. Network-based prediction of protein function. Mol Syst Biol, 3.
Ulitsky I, Shamir R. 2007. Identification of functional modules using network topology and high-throughput data. BMC Syst Biol, 1.
Chuang HY, Lee E, Liu YT, Lee D, Ideker T. 2007. Network-based classification of breast cancer metastasis. Mol Syst Biol, 3.
Fortney K, Kotlyar M, Jurisica I. Inferring the functions of longevity genes with modular subnetwork biomarkers of Caenorhabditis elegans aging. Genome Biol 2010;11(2):R13.
Kotlyar M, Fortney K, Jurisica I. Network-based characterization of drug-regulated genes, drug targets, and toxicity. Methods 2012;57(4):499–507. Protein-Protein Interactions.
Hinze L, Pfirrmann M, Karim S, Degar J, McGuckin C, Vinjamur D, Sacher J, Stevenson KE, Neuberg DS, Orellana E, Stanulla M, Gregory RI, Bauer DE, Wagner FF, Stegmaier K, Gutierrez A. Synthetic lethality of Wnt pathway activation and asparaginase in drug-resistant acute leukemias. Cancer Cell 2019;35(4):664–676.e667.
Wong SWH, Pastrello C, Kotlyar M, Faloutsos C, Jurisica I. Sdregion: fast spotting of changing communities in biological networks. Proceedings of the ACM SIGKDD international conference on knowledge discovery and data mining. New York: ACM; 2018. p. 867–875.
Lambin P, Rios-Velazquez E, Leijenaar R, Carvalho S, van Stiphout RG, Granton P, Zegers CM, Gillies R, Boellard R, Dekker A, Aerts HJ. Radiomics: extracting more information from medical images using advanced feature analysis. Eur J Cancer 2012;48(4):441–6.
Ger RB, Zhou S, Chi PM, Lee HJ, Layman RR, Jones AK, Goff DL, Fuller CD, Howell RM, Li H, Stafford RJ, Court LE, Mackin DS. Comprehensive investigation on controlling for CT imaging variabilities in radiomics studies. Sci Rep 2018;8(1):13047.
Jorgensen P, Nelson B, Robinson MD, Chen Y, Andrews B, Tyers M, Boone C. High-resolution genetic mapping with ordered arrays of Saccharomyces cerevisiae deletion mutants. Genetics 2002;162 (3):1091–9.
Suthers PF, Zomorrodi A, Maranas CD. 2009. Genome-scale gene/reaction essentiality and synthetic lethality analysis, Vol. 5.
Chan EM, Shibue T, McFarland JM, Gaeta B, Ghandi M, Dumont N, Gonzalez A, McPartlan JS, Li T, Zhang Y, Liu JB, Lazaro J-B, Gu P, Piett CG, Apffel A, Ali SO, Deasy R, Keskula P, Ng RWS, Roberts EA, Reznichenko E, Leung L, Alimova M, Schenone M, Islam M, Maruvka YE, Liu Y, Roper J, Raghavan S, Giannakis M, Tseng Y-Y, Nagel ZD, D’Andrea A, Root DE, Boehm JS, Getz G, Chang S, Golub TR, Tsherniak A, Vazquez F, Bass AJ. Wrn helicase is a synthetic lethal target in microsatellite unstable cancers. Nature 2019;568(7753): 551–6.
Holzinger A, Langs G, Denk H, Zatloukal K, Mueller H. 2019. Causability and explainability of AI in medicine. Wiley Interdisciplinary Reviews, Data Mining and Knowledge Discovery.
Krittanawong C, Zhang HJ, Wang Z, Aydar M, Kitai T. Artificial intelligence in precision cardiovascular medicine. J Am Coll Cardiol 2017;69(21):2657–64.
Hamet P, Tremblay J. Artificial intelligence in medicine. Metabolism 2017;69:S36–40.
Robertson S, Azizpour H, Smith K, Hartman J. Digital image analysis in breast pathology—from image processing techniques to artificial intelligence. Transl Res 2018;194:19–35.
Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med 2019;25 (1):44–56.
Hosny A, Parmar C, Quackenbush J, Schwartz LH, Aerts H. Artificial intelligence in radiology. Nat Rev Cancer 2018;18(8):500–510.
Kang J, Rancati T, Lee S, Oh J, Kerns SL, Scott JG, Schwartz R, Kim S, Rosenstein BS. Machine learning and radiogenomics: lessons learned and future directions. Frontiers Oncol 2018;8:228.
Sun R, Limkin E, Vakalopoulou M, Dercle L, Champiat S, Han S, Verlingue L, Brandao D, Lancia A, Ammari S, Hollebecque A, Scoazec J-Y, Marabelle A, Massard C, Soria J-C, Robert C, Paragios N, Deutsch E, Ferté C. A radiomics approach to assess tumour-infiltrating CD8 cells and response to anti-PD-1 or anti-PD-L1 immunotherapy: an imaging biomarker, retrospective multicohort study. Lancet Oncol 2018;19(9):1180–91.
Iorio F, Knijnenburg TA, Vis DJ, Bignell GR, Menden MP, Schubert M, Aben N, Gonçalves E, Barthorpe S, Lightfoot H, Cokelaer T, Greninger P, van Dyk E, Chang H, de Silva H, Heyn H, Deng X, Egan RK, Liu Q, Mironenko T, Mitropoulos X, Richardson L, Wang J, Zhang T, Moran S, Sayols S, Soleimani M, Tamborero D, López-Bigas N, Ross-Macdonald P, Esteller M, Gray NS, Haber DA, Stratton MR, Benes CH, Wessels LFA, Saez-Rodriguez J, McDermott U, Garnett MJ. A landscape of pharmacogenomic interactions in cancer. Cell 2016;166(3):740–54.
Seashore-Ludlow B, Rees MG, Cheah JH, Cokol M, Price EV, Coletti ME, Jones V, Bodycombe NE, Soule CK, Gould J, Alexander B, Li A, Montgomery P, Wawer MJ, Kuru N, Kotz JD, Hon S-YC, Munoz B, Liefeld T, Dancik V, Bittker JA, Palmer M, Bradner JE, Shamji AF, Clemons PA, Schreiber SL. Harnessing connectivity in a large-scale small-molecule sensitivity dataset. Cancer Discovery.
Haverty PM, Lin E, Tan J, Yu Y, Lam B, Lianoglou S, Neve RM, Martin S, Settleman J, Yauch RL, Bourgon R. Reproducible pharmacogenomic profiling of cancer cell line panels. Nature 2016; 533(7603):333–7.
Gao H, Korn JM, Ferretti S, Monahan JE, Wang Y, Singh M, Zhang C, Schnell C, Yang G, Zhang Y, Balbin AO, Barbe S, Cai H, Casey F, Chatterjee S, Chiang DY, Chuai S, Cogan SM, Collins SD, Dammassa E, Ebel N, Embry M, Green J, Kauffmann A, Kowal C, Leary RJ, Lehar J, Liang Y, Loo Alice, Lorenzana E III, Robert E, McLaughlin ME, Merkin J, Meyer R, Naylor TL, Patawaran M, Reddy A, Röelli C, Ruddy DA, Salangsang F, Santacroce F, Singh AP, Tang Y, Tinetto W, Tobler S, Velazquez R, Venkatesan K, Arx F, Wang H, Wang Z, Wiesmann M, Wyss D, Xu F, Bitter H, Atadja P, Lees E, Hofmann F, Li E, Keen N, Cozens R, Jensen M, Pryer NK, Williams JA, Sellers WR. High-throughput screening using patient-derived tumor xenografts to predict clinical trial drug response. Nat Med 2015;21(11):1318–25.
LeCun Y, Bengio Y, Hinton G. Deep learning. Nature 2015;521(7553):436.
Chartrand G, Cheng PM, Vorontsov E, Drozdzal M, Turcotte S, Pal CJ, Kadoury S, An T. Deep learning A primer for radiologists. Radiographics Rev Publ Radiological Soc North Am Inc 2017;37(7): 2113–31.
Sharp G, Fritscher KD, Pekar V, Peroni M, Shusharina N, Veeraraghavan H, Yang J. Vision 20/20: perspectives on automated image segmentation for radiotherapy. Med Phys 2014;41(5):050902.
Cardenas CE, Yang J, Anderson BM, Court LE, Brock KB. Advances in auto-segmentation. Semin Radiat Oncol 2019;29(3):185–97. Adaptive Radiotherapy and Automation.
Kreimeyer K, Foster M, Pandey A, Arya N, Halford G, Jones SF, Forshee R, Walderhaug M, Botsis T. 2017. Natural language processing systems for capturing and standardizing unstructured clinical information: a systematic review. J Biomed Inform, 73.
Asim M, Wasim M, Khan M, Mahmood W, Abbasi H. 2018. A survey of ontology learning techniques and applications. Database, 2018.
Deshpande PR, Rajan S, Sudeepthi LB, Abdul CN. Patient-reported outcomes: a new era in clinical research. Perspectives Clin Res 2011;2(4):137–44.
Loh E. Medicine and the rise of the robots: a qualitative review of recent advances of artificial intelligence in health. BMJ Leader 2018;2(2):59–63.
Dias D, Cunha J. Wearable health Devices—Vital sign monitoring, systems and technologies. Sensors 2018; 18(8):2414.
Guyon I, Bennett K, Cawley G, Escalante HJ, Escalera S, Ho TK, Macia N, Ray B, Saeed M, Statnikov A. Design of the 2015 ChaLearn AutoML Challenge. 2015 International joint conference on neural networks (IJCNN). IEEE; 2015. p. 1–8.
Silver D, Huang A, Maddison CJ, Guez A, Sifre L, van den Driessche G, Schrittwieser J, Antonoglou I, Panneershelvam V, Lanctot M, Dieleman S, Grewe D, Nham J, Kalchbrenner N, Sutskever I, Lillicrap T, Leach M, Kavukcuoglu K, Graepel T, Hassabis D. Mastering the game of go with deep neural networks and tree search. Nature 2016;529(7587):484–9.
Stickel C, Ebner M, Steinbach-Nordmann S, Searle G, Holzinger A. Emotion detection: application of the valence arousal space for rapid biological usability testing to enhance universal access. Universal access in human–computer interaction. Addressing diversity, lecture notes in computer science, LNCS 5614. In: Stephanidis C, editors. Springer; 2009. p. 615–24.
Esteva A, Kuprel B, Novoa RA, Ko J, Swetter SM, Blau HM, Thrun S. Dermatologist-level classification of skin cancer with deep neural networks. Nature 2017;542(7639):115–8.
Holzinger A. Interactive machine learning for health informatics: when do we need the human-in-the-loop? Brain Inform 2016;3(2):119–31.
Holzinger A, Stocker C, Ofner B, Prohaska G, Brabenetz A, Hofmann-Wellenhof R. Combining HCI, natural language processing, and knowledge discovery - potential of IBM content analytics as an assistive technology in the biomedical domain. Springer lecture notes in computer science LNCS 7947. Springer; 2013.
Holzinger A, Plass M, Holzinger K, Crisan GC, Pintea C-M, Palade V. Towards interactive machine learning (IML): applying ant colony algorithms to solve the traveling salesman problem with the human-in-the-loop approach. Springer lecture notes in computer science LNCS 9817. Springer; 2016. p. 81–95.
Funding
This study was funded in part by Ontario Research Fund (34876) to IJ. B.H.K. was supported by the Gattuso-Slaight Personalized Cancer Medicine Fund at Princess Margaret Cancer Centre and the Artificial Intelligence and Microbiome Program supported by the Princess Margaret Cancer Foundation.
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
Author AH declares no conflict of interest. Author BHK declares no conflict of interest. Author IJ declares no conflict of interest.
Ethical approval
This article does not contain any studies with human participants or animals performed by any of the authors.
Additional information
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
This article is part of the Topical Collection on Advanced Image Analyses (Radiomics and Artificial Intelligence)
Appendix: Abbreviations
Appendix: Abbreviations
- AI ...:
-
Artificial Intelligence
- CPU ...:
-
Central Processing Unit
- CT ...:
-
Computer Tomography
- ECG ...:
-
Electrocardiography
- EEG ...:
-
Electroencephalography
- EOG ...:
-
Electrooculography
- EPR ...:
-
Electronic Patient Record
- fMRI ...:
-
Functional Magnetic Resonance Imaging
- GPU ...:
-
raphical Processing Unit
- ICD ...:
-
International Classification of Diseases
- ML ...:
-
Machine Learning
- MRI ...:
-
Magnetic Resonance Imaging
- NGS ...:
-
Next-Generation Sequencing
- SNOMED CT ...:
-
Standard Nomenclature of Medicine Clinical Terms
- TILs ...:
-
Tumor Infiltrated Lymphocytes
Rights and permissions
About this article
Cite this article
Holzinger, A., Haibe-Kains, B. & Jurisica, I. Why imaging data alone is not enough: AI-based integration of imaging, omics, and clinical data. Eur J Nucl Med Mol Imaging 46, 2722–2730 (2019). https://doi.org/10.1007/s00259-019-04382-9
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00259-019-04382-9