Machine Learning on Small Size Samples: a Synthetic Knowledge Synthesis

Machine learning on small size samples: A synthetic knowledge synthesis

Peter Kokol, Marko Kokol*, Sašo Zagoranski* Faculty of Electrical Engineering and Computer Science, University of Maribor, Maribor, Slovenia *Semantika, Maribor, Slovenia

ITRODUCTION

Periods of scientific knowledge doubling have become significantly becoming shorter and shorter (R. Buckminster Fuller, 1983; Bornmann, Anegón and Leydesdorff, 2010; McDeavitt, 2014). This phenomena, combined with information explosion, fast cycles of technological innovations, Open Access and Open Science movements, and new Web/Internet based methods of scholarly communication have immensely increased the complexity and effort needed to synthesis scientific evidence and knowledge. However, above phenomena also resulted in the growing availability of research literature in a digital, machine-readable format. To solve the emerging complexity of knowledge synthesis and the simultaneous new possibilities offered by digital presentation of scientific literature, Blažun et all (Blažun Vošner et al., 2017) and Kokol et al (Kokol et al., 2020; Kokol, Zagoranski and Kokol, 2020) developed a novel synthetics knowledge synthesis methodology based on the triangulation of (1) distant reading (Moretti, 2013), an approach for understanding the canons of literature not by close (manual) reading, but by using computer based technologies, like text mining and machine learning, (2) bibliometric mapping (Noyons, 2001) and (3) content analysis (Kyngäs, Mikkonen and Kääriäinen, 2020; Lindgren, Lundman and Graneheim, 2020). Such triangulation of technologies enables one to combine quantitative and qualitative knowledge synthesis in the manner to extend classical bibliometric analysis of publication metadata with the machine learning supported understanding of patterns, structure and content of publications (Atanassova, Bertin and Mayr, 2019). One of the increasingly important technologies dealing with the growing complexity of the digitalisation of almost all human activities is the Artificial intelligence, more precisely machine learning (Krittanawong, 2018; Grant et al., 2020). Despite the fact, that we live in a “Big data” world where almost “everything” is digitally stored, there are many real world situation, where researchers are faced with small data samples. Hence, it is quite common, that the database is limited for example by the number of subjects (i.e. patients with rare disseises), the sample is small comparing to number of features like in genetics or biomarkers detection, sampling, there is a lot of noisy or missing data or measurements are extremely expensive or data iimbalanced meaning that the size of one class in a data set has very few objects (Thomas et al., 2019; Vabalas et al., 2019). Using machine learning on small size datasets pose a problem, because, in general, the “power”of machine learning in recognising patterns is proportional to the size of the dataset, the smaller the dataset, less powerful and less accurate are the machine learning algorithms. Despite the commonality of the above problem and various approaches to solve it we didn’t found any holistic studies concerned with this important area of the machine learning. To fill this gap we used synthetic knowledge synthesis presented above to aggregate current evidence relating to the »small data set problem« in machine learning. In that manner we can overcome the problem of isolated findings which might be incomplete or might lack possible overlap with other possible solutions. In our analysis, we aimed to extract, synthetise and multidimensionally structure the evidence of as possible complete corpus of scholarship on small data sets. Additionally, the study seeks to identify gaps which may require further research. Thus the outputs of the study might help researchers and practitioners to understand the broader aspects of small size sample` research and its translation to practice. On the other hand, it can help a novice or data science professional without specific knowledge on small size samples to develop a perspective on the most important research themes, solutions and relations between them.

METHODOLOGY

Following research question research questions was set

What is the small data problem and how it is solved?

Synthetic knowledge synthesis was performed following the steps below:

1. Harvest the research publications concerning small data sets in machine learning to represent the content to analyse. 2. Condense and code the content using text mining. 3. Analyse the codes using bibliometric mapping and induce the small data set research cluster landscape. 4. Analyse the connections between the codes in individual clusters and map them into sub- categories. 5. Analyse sub-categories to label cluster with themes. 6. Analyse sub-categories to identify research dimensions 7. Cross-tabulate themes and research dimension and identify concepts The research publications were harvested from the Scopus database, using the advance search using the command TITLE-ABS-KEY(("small database" or "small dataset" or "small sample" ) and "machine learning" and not ("large database" or "large dataset" or "large sample" )) The search was performed on 7th of January, 2021. Following metadata were exported as a CSV formatted corpus file for each publication: Authors, Authors affiliations, Publication Title, Year of publication, Source Title, Abstract and Author Keywords. Bibliometric mapping and text mining were performed using the VOSViewer software (Leiden University, Netherlands). VOSViewer uses text mining to recognize publication terms and then employs the mapping technique called Visualisation of Similarities (VoS) to create bibliometric maps or landscapes (van Eck and Waltman, 2014). Landscapes are displayed in various ways to present different aspects of the research publications content. In this study the content analysis was performed on the author keywords cluster landscape, due to the fact that previous research showed that authors keywords most concisely present the content authors would like to communicate to the research community (Železnik, Blažun Vošner and Kokol, 2017). In this type of landscape the VOSviewer merges author keywords which are closely associated into clusters, denoted by the same cluster colours (van Eck and Waltman, 2010). Using a customized Thesaurus file, we excluded the common terms like study, significance, experiment and eliminated geographical names and time stamps from the analysis.

RESULTS AND DISCUSSION

The search resulted in 1254 publications written by 3833 authors. Among them were 687 articles, 500 conference papers, 33 review papers, 17 book chapters and 17 other types of publications. Most productive countries among 78 were China (n=439), United States (n=297), United Kingdom (n=91), Germany (n=63), India (n=52), Canada (n=51) and Spain (n=44), Australia (n=42), Japan (n=34), South Korea (n=30) and France (n=30). Most productive institutions are located in China and United States. Among 2857 institution, China Academy of Sciences is the most productive institution (n=52), followed by Ministry of Education China (n=28), Harvard Medical School, USA (n=14), Harbin Institute of Technology, China (n=14), National Cheng Kung University, China (13), Shanghai University, China (n=13), Shanghai Jiao Tong University, China (n=12), Georgia Institute of Technology, USA (n=12) and Northwestern Polytechnical University, China (n=11). The first non China/USA institution is the University of Granada (n=8) on 19th rank. Majority of the productive countries are member of G9 countries, with strong economies and research infrastructure.

Most prolific source titles (journals, proceedings, books) are the Lecture Notes In Computer Science Including Subseries Lecture Notes In Artificial Intelligence And Lecture Notes In Bioinformatics (n=64), IEEE Access (n=17), Neurocomputing (n=17), Plos One (n=17), Advances in Intelligent Systems and Computing (n=14), Expert System with application (n=14), ACM International Conference Proceedings Series (n=13) and Proceedings Of SPIE The International Society For Optical Engineering (n=13). The more productive source titles are mainly from the computer science research area and are ranked in the first half of Scimago Journal ranking (Scimago, Elsevier, netherlands). In average their h-indexs is between 120 an 380.

Naslov grafikona 300

250

200

150

100

2014 2018 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2015 2016 2017 2019 2020

Total number of publications Articles Conference papers Reviews

Figure 1. The research literature production dynamics

First two papers concerning the use of machine learning on small datasets indexed in Scopus were published in 1995 (Forsström et al., 1995; Reich and Travitzky, 1995). After that the publications were rare till 2002, trend starting to rise linearly in 2003, and exponentially in 2016. In the beginning till the year 2000 all publications were published in journals, after that conference papers started to emerge. Uninterrupted production of review papers started much later in 2016. According to the ratio between articles, reviews and conference paper and the exponential trend in production for the first two we asses that machine learning on small datasets in the third stage of Schneider scientific discipline evolution model (Shneider, 2009). That means that the terminology and methodologies are already highly developed, and that domain specific original knowledge generation is heading toward optimal research productivity.

On the other hand Pestana et al (Pestana, Sánchez and Moutinho, 2019) characterised the scientific discipline maturity as a set of extensive repeated connections between researchers that collaborate on the publication of papers on the same or related topics over time. In our study we analysed those connection with the co-authors networks induced by VOSviewer. The co-author network showed that among countries with the productivity of 10 or more papers (n=27) presented in Figure 2. an elaborate international co-authorship network emerged, confirming that maturity is already reached a high level. The most intensive cooperation is between United states and China, and the “newest” countries joining the network are Russian federation, Soth Korea and Brasil.

Figure 2. The co-authorship network

The content analysis of keyword cluster landscape shown in Figure 3. resulted in codes, SUB- categories and themes presented in Table 1. The content analysis revealed four prevailing themes. The largest two themes are related to dimension reduction in complex big data analysis and data augmentation techniques in deep learning.

Figure 3. The author keyword cluster landscape

Table 1: Small size sample research themes (numbers in parenthesis present the number of papers in which an author keyword occurred)

Theme Colour More frequent codes Prevailing sub-categories Dimension Red Machine learning (339), Machine learning reduction in Classification (51), Feature algorithms in complex big data selection (50), Artificial neural bioinformatics, analysis network (28), Neural network Classification on small (19), Natural language samples with feature processing (16), Ensemble selection, Ensemble learning (14), Bioinformatics learning for solving under- (13), Decision tree (12), Breast sampling in personalised cancer (11), Machine learning medicine, Solving missing algorithms (10), Computer data in breast cancer using vision (10), Microarray (9), deep neural networks, Small datasets (9), Text Dimensionality reduction. classification (9), Big data (9), with feature extraction in Dimensionality reduction (9), cancer classification, Social media (7), Natural language processing in social media and text analysis Data Blue Deep learning (89), Deep and transfer learning augmentation and Convolutional neural networks using in image techniques in yellow (45), Transfer learning (38), classification using data deep learning for Pattern recognition (14), Image augmentation and synthetic pattern classification (14), data data, complex neural recognition and augmentation (12), Generative networks, Pattern classification adversarial networks (8), recognition in mobile LSTM (6), Resnet – Residual sensing using Principal neural network (n=4) component analysis and LTSM, Resnet and Desnet based deep learning using auto encoder on case of glaucoma, Support vector Green Support vector machines (109), Support vector machines machines, Random forest (24), Prediction and random forests in Random forest (18), Fault diagnosing (18), prediction, forecasting and and Genetic Forecast (10), Genetic fault diagnosis, Genetic algorithms in algorithms (8), Regression (8), algorithms in optimisation statistical Gaussian process (7), and time series analysis learning Optimisation (7), Time series (6), Statistical learning theory (6) Data mining on Viollet Small datasets (2), Data mining Small datasets small datasets (15), Support vector regression augmentation with virtual with support (11), Virtual sample (7) samples, Data mining with support vector regression vectors regression

The higher abstraction analysis of the Table 1. revealed four categories, which we call Research dimensions. Four identified dimensions are: Small data set problem, Machine learning algorithms, Small-data pre-processing technique and Application Area. Cross tabulation of themes and research dimension resulted in a taxonomy presented in Table 2. The entries in the table presents the most popular concepts in each taxonomical entity.

The most frequently reported difficulties causing the small data problem are the small size of the dataset, high/low dimensionality of datasets and unbalanced data. Small size datasets can cause problems when machine learning is applied in material sciences (Zhang and Ling, 2018), engineering (Babič et al., 2014; Feng, Zhou and Dong, 2019) and various omics fields (Ko, Choi and Ahn, 2021) due to the high cost of sampling; differentiating between autistic and non-autistic patients (Vabalas et al., 2019) and diagnosing rare diseases (Spiga et al., 2020) due to the small number of available patients, or unavailability of other subjects for example PhD students in prediction of their grades (Abu Zohair, 2019); and new pandemic prediction due fact that samples are scare when pandemic occurs and there is lack of medical knowledge (Fong et al., 2020). Similar reasons can also lead to too high or low dimensionality of the datasets. The first case can occur even if the sample size is not small, however the ratio between number of features and the sample size is large, like for example in particle physics (Komiske et al., 2018) or bioinformatics (Zhang et al., 2021). The second case can occur when the number of instances is not problematic, however the number of features is very small, like for example in characterisation of high-entropy alloys (Dai et al., 2020). Unbalanced data present a long standing problem in machine learning and still remains a challenge in various applications like face recognition, credit scoring, fault diagnosing and anomaly detection where mayor class has much more instances than one or more of remaining classes (He et al., 2020; Liang et al., 2020; Marceau et al., 2020).

The most used machine learning algorithms used on small datasets are support vector machines (Alvarsson et al., 2016; Cao et al., 2017; Razzak et al., 2020), decision trees/forests (Zhang et al., 2017; Shaikhina et al., 2019), convolutional neural networks (Liu and Deng, 2016; Yamashita et al., 2018; Guo et al., 2020) and transfer learning (Hall et al., 2020; Tits, El Haddad and Dutoit, 2020). Most frequently employed data pre-processing techniques to overcame the small size problem are liner and nonlinear Principal component analysis (Jalali, Mallipeddi and Lee, 2017; Feng, Zhou and Dong, 2019; Athanasopoulou, Papacharalampopoulos and Stavropoulos, 2020), Discriminant analysis (Abu Zohair, 2019; Li et al., 2019; Silitonga et al., 2021), Data augmentation(Han, Liu and Fan, 2018; Hagos and Kant, 2019; Fong et al., 2020), Virtual sample (Gong et al., 2017; MacAllister, Kohl and Winer, 2020; Zhu et al., 2020), Feature extraction (Kumar et al., 2018; Dai et al., 2020) and Auto-encoder (Feng, Zhou and Dong, 2019; Pei et al., 2021).

Most affected areas are Bioinformatics (Giansanti et al., 2019; Khatun et al., 2020; Ko, Choi and Ahn, 2021), image classification and analysis (Han, Liu and Fan, 2018; Hagos and Kant, 2019; Yadav and Jadhav, 2019), fault diagnosing (Saufi et al., 2020; Wen, Li and Gao, 2020), forecasting and prediction (Croda, Romero and Morales, 2019; Fong et al., 2020), social media analysis (Khan et al., 2019; Renault, 2020) and health care (Rajpurkar et al., 2020; Subirana et al., 2020).

Table 2: Taxonomy of Themes and Research dimensions categories (numbers in parenthesis present the number of publications)

Theme Small data Machine learning Small – data Application size problem algorithms pre-processing area technique Dimension Missing data Bagging (9), Bayesian Dimensionality Image reduction in (3), networks (7), Boosting reduction (9), analysis (18), complex big Unbalanced (6), Decision trees and Feature genetics and data analysis data (8), forests (43), deep extraction (12), bioinformatics Under neural network (12), engineering (27), sampling (3) Ensemble learning (4), fusion (2), personalised (21), manifold learning Lasso (6), and precision (5), neural network Linear and medicine (6) (30), naïve Bayes (2), multiple neuroimaging semi supervised discriminant (7), mental learning (17), analysis (13), health (25) supervised learning PCA (16), (19), unsupervised autoencoder learning (4), FMRI (7) (10) Data Convolutional neural Data Image augmentation network (42), Dense augmentation classification for deep CNN (3), Generative (12), Domain (22), genetics learning in adversarial networks adaption (5), (3), mobile pattern (8), LSTM (6), Resnet data fusion (3), sensing (10), recognition (4), transfer learning synthetic data predictive and (40), semi-supervised (3), modelling (6), classification learning (17) autoencoder semantic (10) segmentation (3) Statistical Small sample Support vector Feature Forecasting based machine (98) machines (89). Least engineering (13), Fault learning square SVM (10), (3), meta- diagnosing genetic algorithms, learning (5), (18), random forest (24), Monte Carlo regression Particle swarm (27), (3), Small (19), support vector sample prediction regression (11) learning (5), (26), pattern virtual sample recognition (11) (17), rock mechanics (5), security 7), computer vision (10), data mining (15) Data mining Big data (9) Text and on small social media datasets analysis (21), Precision medicine (3), Natural language processing (16), Sentiment analysis (9), clustering (8)

Strengths and limitations

The main strength of the study is that it is the first bibliometrics and content analysis of the small dataset research. One of the limitation is that the analysis was limited to publications indexed in Scopus only, however Scopus is indexing the largest and most complete set of information titles, thus assuring the integrity of the data source. Additionally, the analysis alsp included qualitative components, which might bias the results of our study.

Conclusion

Our bibliometric study showed the positive trend in the number of research publications concerning the use of small dataset and substantial growth of research community dealing with the small dataset problem, indicating that the research field is moving toward the higher maturity levels. Despite notable international cooperation, regional concentration of research literature production in economically more developed countries was observed.

The results of study present a multi-dimensional facet and science landscape of the small datasets problem which can help machine learning researchers and practitioners to improve their understanding of the area and can catalyse further knowledge development. On the other hand, it can inform novice researchers, interested readers or research mangers and evaluators without specific knowledge and help them to develop a perspective on the most important small dataset research dimensions. Finally, the study output can serve as a guide to further research.

REFERENCES

Abu Zohair, L. M. (2019) ‘Prediction of Student’s performance by modelling small dataset size’, International Journal of Educational Technology in Higher Education, 16(1), p. 27. doi: 10.1186/s41239-019-0160-3. Alvarsson, J. et al. (2016) ‘Large-scale ligand-based predictive modelling using support vector machines’, Journal of Cheminformatics, 8(1), p. 39. doi: 10.1186/s13321-016-0151-5.

Atanassova, I., Bertin, M. and Mayr, P. (2019) ‘Editorial: Mining Scientific Papers: NLP-enhanced Bibliometrics’, Frontiers in Research Metrics and Analytics, 4. doi: 10.3389/frma.2019.00002.

Athanasopoulou, L., Papacharalampopoulos, A. and Stavropoulos, P. (2020) ‘Context awareness system in the use phase of a smart mobility platform: A vision system for a light-weight approach’, in. Procedia CIRP, pp. 560–564. doi: 10.1016/j.procir.2020.05.097.

Babič, M. et al. (2014) ‘Using of genetic programming in engineering’, Elektrotehniski Vestnik/Electrotechnical Review, 81(3), pp. 143–147.

Blažun Vošner, H. et al. (2017) ‘Trends in nursing ethics research: Mapping the literature production’, Nursing Ethics, 24(8), pp. 892–907. doi: 10.1177/0969733016654314.

Bornmann, L., Anegón, F. de M. and Leydesdorff, L. (2010) ‘Do Scientific Advancements Lean on the Shoulders of Giants? A Bibliometric Investigation of the Ortega Hypothesis’, PLOS ONE, 5(10), p. e13327. doi: 10.1371/journal.pone.0013327.

Cao, J. et al. (2017) ‘An accurate traffic classification model based on support vector machines’, International Journal of Network Management, 27(1), p. e1962. doi: https://doi.org/10.1002/nem.1962.

Croda, R. M. C., Romero, D. E. G. and Morales, S.-O. C. (2019) ‘Sales Prediction through Neural Networks for a Small Dataset’, IJIMAI, 5(4), pp. 35–41.

Dai, D. et al. (2020) ‘Using machine learning and feature engineering to characterize limited material datasets of high-entropy alloys’, Computational Materials Science, 175. doi: 10.1016/j.commatsci.2020.109618. van Eck, N. J. and Waltman, L. (2010) ‘Software survey: VOSviewer, a computer program for bibliometric mapping’, Scientometrics, 84(2), pp. 523–538. doi: 10.1007/s11192-009-0146-3. van Eck, N. J. and Waltman, L. (2014) ‘Visualizing Bibliometric Networks’, in Ding, Y., Rousseau, R., and Wolfram, D. (eds) Measuring Scholarly Impact: Methods and Practice. Cham: Springer International Publishing, pp. 285–320. doi: 10.1007/978-3-319-10377-8_13.

Feng, S., Zhou, H. and Dong, H. (2019) ‘Using deep neural network with small dataset to predict material defects’, Materials and Design, 162, pp. 300–310. doi: 10.1016/j.matdes.2018.11.060.

Fong, S. J. et al. (2020) ‘Finding an Accurate Early Forecasting Model from Small Dataset: A Case of 2019-nCoV Novel Coronavirus Outbreak’, International Journal of Interactive Multimedia and Artificial Intelligence, 6(1), p. 132. doi: 10.9781/ijimai.2020.02.002.

Forsström, J. J. et al. (1995) ‘Using data preprocessing and single layer perceptron to analyze laboratory data’, Scandinavian Journal of Clinical and Laboratory Investigation, 55(S222), pp. 75–81. doi: 10.3109/00365519509088453.

Giansanti, V. et al. (2019) ‘Comparing Deep and Machine Learning Approaches in Bioinformatics: A miRNA-Target Prediction Case Study’, in Rodrigues, J. M. F. et al. (eds) Computational Science – ICCS 2019. Cham: Springer International Publishing (Lecture Notes in Computer Science), pp. 31–44. doi: 10.1007/978-3-030-22744-9_3. Gong, H.-F. et al. (2017) ‘A Monte Carlo and PSO based virtual sample generation method for enhancing the energy prediction and energy optimization on small data problem: An empirical study of petrochemical industries’, Applied Energy, 197, pp. 405–415. doi: 10.1016/j.apenergy.2017.04.007.

Grant, K. et al. (2020) ‘Artificial Intelligence in Emergency Medicine: Surmountable Barriers With Revolutionary Potential’, Annals of Emergency Medicine, 75(6), pp. 721–726. doi: 10.1016/j.annemergmed.2019.12.024.

Guo, F. et al. (2020) ‘Improving cardiac MRI convolutional neural network segmentation on small training datasets and dataset shift: A continuous kernel cut approach’, Medical Image Analysis, 61, p. 101636. doi: 10.1016/j.media.2020.101636.

Hagos, M. T. and Kant, S. (2019) ‘Transfer Learning based Detection of Diabetic Retinopathy from Small Dataset’, arXiv:1905.07203 [cs]. Available at: http://arxiv.org/abs/1905.07203 (Accessed: 19 February 2021).

Hall, L. O. et al. (2020) ‘Finding Covid-19 from Chest X-rays using Deep Learning on a Small Dataset’, arXiv:2004.02060 [cs, eess]. Available at: http://arxiv.org/abs/2004.02060 (Accessed: 18 February 2021).

Han, D., Liu, Q. and Fan, W. (2018) ‘A new image classification method using CNN transfer learning and web data augmentation’, Expert Systems with Applications, 95, pp. 43–56. doi: 10.1016/j.eswa.2017.11.028.

He, Z. et al. (2020) ‘Support tensor machine with dynamic penalty factors and its application to the fault diagnosis of rotating machinery with unbalanced data’, Mechanical Systems and Signal Processing, 141, p. 106441. doi: 10.1016/j.ymssp.2019.106441.

Jalali, A., Mallipeddi, R. and Lee, M. (2017) ‘Sensitive deep convolutional neural network for face recognition at large standoffs with small dataset’, Expert Systems with Applications, 87, pp. 304–315. doi: 10.1016/j.eswa.2017.06.025.

Khan, J. Y. et al. (2019) ‘A Benchmark Study on Machine Learning Methods for Fake News Detection’, arXiv:1905.04749 [cs, stat]. Available at: http://arxiv.org/abs/1905.04749 (Accessed: 19 February 2021).

Khatun, Mst. S. et al. (2020) ‘Evolution of Sequence-based Bioinformatics Tools for Protein-protein Interaction Prediction’, Current Genomics, 21(6), pp. 454–463. doi: 10.2174/1389202921999200625103936.

Ko, S., Choi, J. and Ahn, J. (2021) ‘GVES: machine learning model for identification of prognostic genes with a small dataset’, Scientific Reports, 11(1), p. 439. doi: 10.1038/s41598-020-79889-5.

Kokol, P. et al. (2020) ‘Enhancing the role of academic librarians in conducting scoping reviews’, Library Philosophy and Practice (e-journal). Available at: https://digitalcommons.unl.edu/libphilprac/4293.

Kokol, P., Zagoranski, S. and Kokol, M. (2020) ‘Software Development with Scrum: A Bibliometric Analysis and Profile’, Library Philosophy and Practice (e-journal). Available at: https://digitalcommons.unl.edu/libphilprac/4705. Komiske, P. T. et al. (2018) ‘Learning to classify from impure samples with high-dimensional data’, Physical Review D, 98(1), p. 011502. doi: 10.1103/PhysRevD.98.011502.

Krittanawong, C. (2018) ‘The rise of artificial intelligence and the uncertain future for physicians’, European Journal of Internal Medicine, 48, pp. e13–e14. doi: 10.1016/j.ejim.2017.06.017.

Kumar, I. et al. (2018) ‘A Comparative Study of Supervised Machine Learning Algorithms for Stock Market Trend Prediction’, in. Proceedings of the International Conference on Inventive Communication and Computational Technologies, ICICCT 2018, pp. 1003–1007. doi: 10.1109/ICICCT.2018.8473214.

Kyngäs, H., Mikkonen, K. and Kääriäinen, M. (2020) ‘The Application of Content Analysis in Nursing Science Research’. doi: 10.1007/978-3-030-30199-6.

Li, W. et al. (2019) ‘Detecting Alzheimer’s Disease on Small Dataset: A Knowledge Transfer Perspective’, IEEE Journal of Biomedical and Health Informatics, 23(3), pp. 1234–1242. doi: 10.1109/JBHI.2018.2839771.

Liang, X. W. et al. (2020) ‘LR-SMOTE — An improved unbalanced data set oversampling based on K- means and SVM’, Knowledge-Based Systems, 196, p. 105845. doi: 10.1016/j.knosys.2020.105845.

Lindgren, B.-M., Lundman, B. and Graneheim, U. H. (2020) ‘Abstraction and interpretation during the qualitative content analysis process’, International Journal of Nursing Studies, 108, p. 103632. doi: 10.1016/j.ijnurstu.2020.103632.

Liu, S. and Deng, W. (2016) ‘Very deep convolutional neural network based image classification using small training sample size’, in. Proceedings - 3rd IAPR Asian Conference on Pattern Recognition, ACPR 2015, pp. 730–734. doi: 10.1109/ACPR.2015.7486599.

MacAllister, A., Kohl, A. and Winer, E. (2020) ‘Using high-fidelity meta-models to improve performance of small dataset trained Bayesian Networks’, Expert Systems with Applications, 139, p. 112830. doi: 10.1016/j.eswa.2019.112830.

Marceau, L. et al. (2020) ‘A comparison of Deep Learning performances with other machine learning algorithms on credit scoring unbalanced data’, arXiv:1907.12363 [cs, stat]. Available at: http://arxiv.org/abs/1907.12363 (Accessed: 18 February 2021).

McDeavitt, J. T. (2014) ‘Medical Education: Toy Airplane or Stone Flywheel?’ Available at: http://wingofzock.org/2014/12/23/medical-education-toy-airplane-or-stone-flywheel/ (Accessed: 29 April 2017).

Moretti, F. (2013) Distant Reading. London: Verso.

Noyons, E. (2001) ‘Bibliometric mapping of science in a science policy context’, Scientometrics, 50(1), pp. 83–98. doi: 10.1023/A:1005694202977.

Pei, Z. et al. (2021) ‘Data augmentation for rolling bearing fault diagnosis using an enhanced few-shot Wasserstein auto-encoder with meta-learning’, Measurement Science and Technology. doi: 10.1088/1361-6501/abe5e3.

Pestana, M. H., Sánchez, A. V. and Moutinho, L. (2019) ‘The network science approach in determining the intellectual structure, emerging trends and future research opportunities – An application to senior tourism research’, Tourism Management Perspectives, 31, pp. 370–382. doi: 10.1016/j.tmp.2019.07.006.

R. Buckminster Fuller (1983) Critical Path. Century Hutchinson Ltd.

Rajpurkar, P. et al. (2020) ‘AppendiXNet: Deep Learning for Diagnosis of Appendicitis from A Small Dataset of CT Exams Using Video Pretraining’, Scientific Reports, 10(1), p. 3958. doi: 10.1038/s41598- 020-61055-6.

Razzak, I. et al. (2020) ‘Randomized nonlinear one-class support vector machines with bounded loss function to detect of outliers for large scale IoT data’, Future Generation Computer Systems, 112, pp. 715–723. doi: 10.1016/j.future.2020.05.045.

Reich, Y. and Travitzky, N. (1995) ‘Machine learning of material behaviour knowledge from empirical data’, Materials and Design, 16(5), pp. 251–259. doi: 10.1016/0261-3069(96)00007-6.

Renault, T. (2020) ‘Sentiment analysis and machine learning in finance: a comparison of methods and models on one million messages’, Digital Finance, 2(1), pp. 1–13. doi: 10.1007/s42521-019-00014-x.

Saufi, S. R. et al. (2020) ‘Gearbox Fault Diagnosis Using a Deep Learning Model with Limited Data Sample’, IEEE Transactions on Industrial Informatics, 16(10), pp. 6263–6271. doi: 10.1109/TII.2020.2967822.

Shaikhina, T. et al. (2019) ‘Decision tree and random forest models for outcome prediction in antibody incompatible kidney transplantation’, Biomedical Signal Processing and Control, 52, pp. 456–462. doi: 10.1016/j.bspc.2017.01.012.

Shneider, A. M. (2009) ‘Four stages of a scientific discipline; four types of scientist’, Trends in Biochemical Sciences, 34(5), pp. 217–223. doi: 10.1016/j.tibs.2009.02.002.

Silitonga, P. et al. (2021) ‘Comparison of Dengue Predictive Models Developed Using Artificial Neural Network and Discriminant Analysis with Small Dataset’, Applied Sciences, 11(3), p. 943. doi: 10.3390/app11030943.

Spiga, O. et al. (2020) ‘Machine learning application for development of a data-driven predictive model able to investigate quality of life scores in a rare disease’, Orphanet Journal of Rare Diseases, 15(1). doi: 10.1186/s13023-020-1305-0.

Subirana, B. et al. (2020) ‘Hi Sigma, do I have the Coronavirus?: Call for a New Artificial Intelligence Approach to Support Health Care Professionals Dealing With The COVID-19 Pandemic’, arXiv:2004.06510 [cs]. Available at: http://arxiv.org/abs/2004.06510 (Accessed: 19 February 2021).

Thomas, R. M. et al. (2019) ‘Dealing with missing data, small sample sizes, and heterogeneity in machine learning studies of brain disorders’, in Machine Learning: Methods and Applications to Brain Disorders, pp. 249–266. doi: 10.1016/B978-0-12-815739-8.00014-6.

Tits, N., El Haddad, K. and Dutoit, T. (2020) ‘Exploring Transfer Learning for Low Resource Emotional TTS’, in Bi, Y., Bhatia, R., and Kapoor, S. (eds) Intelligent Systems and Applications. Cham: Springer International Publishing (Advances in Intelligent Systems and Computing), pp. 52–60. doi: 10.1007/978-3-030-29516-5_5.

Vabalas, A. et al. (2019) ‘Machine learning algorithm validation with a limited sample size’, PLoS ONE, 14(11). doi: 10.1371/journal.pone.0224365. Wen, L., Li, X. and Gao, L. (2020) ‘A transfer convolutional neural network for fault diagnosis based on ResNet-50’, Neural Computing and Applications, 32(10), pp. 6111–6124. doi: 10.1007/s00521-019- 04097-w.

Yadav, S. S. and Jadhav, S. M. (2019) ‘Deep convolutional neural network based medical image classification for disease diagnosis’, Journal of Big Data, 6(1), p. 113. doi: 10.1186/s40537-019-0276- 2.

Yamashita, R. et al. (2018) ‘Convolutional neural networks: an overview and application in radiology’, Insights into Imaging, 9(4), pp. 611–629. doi: 10.1007/s13244-018-0639-9.

Železnik, D., Blažun Vošner, H. and Kokol, P. (2017) ‘A bibliometric analysis of the Journal of Advanced Nursing, 1976–2015’, Journal of Advanced Nursing, 73(10), pp. 2407–2419. doi: 10.1111/jan.13296.

Zhang, Y. et al. (2021) ‘Evaluating and selecting features via information theoretic lower bounds of feature inner correlations for high-dimensional data’, European Journal of Operational Research, 290(1), pp. 235–247. doi: 10.1016/j.ejor.2020.09.028.

Zhang, Y. and Ling, C. (2018) ‘A strategy to apply machine learning to small datasets in materials science’, npj Computational Materials, 4(1). doi: 10.1038/s41524-018-0081-z.

Zhang, Z. et al. (2017) ‘Topological Analysis and Gaussian Decision Tree: Effective Representation and Classification of Biosignals of Small Sample Size’, IEEE Transactions on Biomedical Engineering, 64(9), pp. 2288–2299. doi: 10.1109/TBME.2016.2634531.

Zhu, Q.-X. et al. (2020) ‘Dealing with small sample size problems in process industry using virtual sample generation: a Kriging-based approach’, Soft Computing, 24(9), pp. 6889–6902. doi: 10.1007/s00500-019-04326-3.