functional data analysis dissertation

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• BMC Med Res Methodol

Applications of functional data analysis: A systematic review

Shahid ullah.

1 Flinders Centre for Epidemiology and Biostatistics, School of Medicine, Faculty of Health Sciences, Flinders University, Adelaide, SA, 5001, Australia

Caroline F Finch

2 Centre for Healthy and Safe Sports (CHASS), University of Ballarat, SMB Campus, Ballarat, VIC, 3353, Australia

Functional data analysis (FDA) is increasingly being used to better analyze, model and predict time series data. Key aspects of FDA include the choice of smoothing technique, data reduction, adjustment for clustering, functional linear modeling and forecasting methods.

A systematic review using 11 electronic databases was conducted to identify FDA application studies published in the peer-review literature during 1995–2010. Papers reporting methodological considerations only were excluded, as were non-English articles.

In total, 84 FDA application articles were identified; 75.0% of the reviewed articles have been published since 2005. Application of FDA has appeared in a large number of publications across various fields of sciences; the majority is related to biomedicine applications (21.4%). Overall, 72 studies (85.7%) provided information about the type of smoothing techniques used, with B-spline smoothing (29.8%) being the most popular. Functional principal component analysis (FPCA) for extracting information from functional data was reported in 51 (60.7%) studies. One-quarter (25.0%) of the published studies used functional linear models to describe relationships between explanatory and outcome variables and only 8.3% used FDA for forecasting time series data.

Conclusions

Despite its clear benefits for analyzing time series data, full appreciation of the key features and value of FDA have been limited to date, though the applications show its relevance to many public health and biomedical problems. Wider application of FDA to all studies involving correlated measurements should allow better modeling of, and predictions from, such data in the future especially as FDA makes no a priori age and time effects assumptions.

Recent increased interest in the application of statistical modeling to medicine, biomedicine, public health, biology, biomechanics and environmental science has largely been driven by the need for good data to inform government policy and planning processes for health service delivery and disease prevention. Importantly, such models will only be useful in the long term if they are accurate, based on good quality data, and generated through application of robust appropriate statistical methods. Functional data analysis (FDA) is one such approach towards modeling time series data that has started to receive attention in the literature, particularly in terms of its public health and biomedical applications.

Commonly, time series data are treated as multivariate data because they are given as a finite discrete time series. This usual multivariate approach completely ignores important information about the smooth functional behavior of the generating process that underpins the data [ 1 ]. It also suffers from issues associated with highly correlated measurements within each functional object. The basic idea behind FDA is to express discrete observations arising from time series in the form of a function (to create functional data) that represents the entire measured function as a single observation, and then to draw modeling and/or prediction information from a collection of functional data by applying statistical concepts from multivariate data analysis. In doing so, it has the advantage of generating models that can be described by continuous smooth dynamics, which then allow for accurate estimates of parameters for use in the analysis phase, effective data noise reduction through curve smoothing, and applicability to data with irregular time sampling schedules. Ramsay [ 2 , 3 ] presents a strong argument for FDA.

Ramsay and Dalzell [ 4 ] present several practical reasons for considering functional data:

1) smoothing and interpolation procedures can yield functional representations of a finite set of observations;

2) it is more natural to think through modeling problems in a functional form; and

3) the objectives of an analysis can be functional in nature, as would be the case if finite data were used to estimate an entire function, its derivatives, or the values of other functionals.

Müller has recently described important characteristics of FDA [ 5 ]. The FDA approach is highly flexible in the sense that the timing intervals for data observations do not have to be equally spaced for all cases and can vary across cases. Importantly, FDA methods are not necessarily based on the assumption that the values observed at different times for a single subject are independent. Although functional data themselves are not new, a new conceptualization of them has become necessary because of the increasing sophistication of available data collections [ 4 ]. Data collection technology has evolved over recent decades, allowing more dense sampling of observations over time, space, and other continuum measures. Such data are usually interpreted as reflecting the influence of certain smooth functions that are assumed to underlie and to generate the observations. Although classical multivariate statistical techniques are often applied to such data, they do not take advantage of additional information that could be implied by the smoothness of underlying functions. In particular, FDA methods can often extract additional information contained in the function and its derivatives [ 6 , 7 ] that is not normally available from application of traditional statistical methods [ 1 ]. Because the FDA approach essentially treats the whole curve as a single entity, there is also no concern about correlations between repeated measurements. This represents a change in philosophy towards the handling of time series and correlated data [ 8 ].

There are a number of good illustrations of applications of FDA; for example, Ramsay and Silverman [ 9 , 10 ] using curves as data, Locantore et al. [ 11 ] with images as data, and Yushkevich and Pizer [ 12 ] where the data points are shape representations of body parts. Application of FDA has also been published across various scientific fields including analysis of child size evolution [ 9 ], climatic variation [ 4 , 13 ], handwriting in Chinese [ 14 ], acidification processes [ 15 ], land usage prediction based on satellite images [ 16 ], medical research [ 17 - 19 ], behavioral sciences [ 20 ], term-structured yield curves [ 21 ], and spectrometry data [ 22 ]. Most recently, Ullah and Finch [ 23 ] found FDA to be an effective exploratory and modeling technique for highlighting trends and variations in the shape of the age–falls injury incidence relationship over time.

In contrast to most other methods commonly used to model trends in time series data, a key strength of the FDA approach is that it makes no parametric assumptions about age or time effects. The FDA methods for modeling and forecasting data across a range of health and demographic issues also have significant advantages for better understanding trends, risk factor relationships, and the effectiveness of preventive measures [ 24 , 25 ]. In the book Functional Data Analysis , Ramsay and Silverman [ 9 ] give an accessible overview of the foundations and applications of FDA. In an earlier book entitled Applied Functional Data Analysis , the same authors [ 10 ] provide many examples that share the property of being functional forms of a continuous variable, most often age or time. In 2004, Statistica Sinica published a special issue that included two relevant review articles that dealt exclusively with the close connection between longitudinal data and functional data [ 26 , 27 ]. In his PhD thesis, Ullah [ 28 ] described the significance and application of FDA in demographic data settings. Software developed for MATLAB, S-PLUS and R by Ramsay and Silverman specifically to support FDA is available from < http://www.psych.mcgill.ca/misc/fda/ >.

Because the application of FDA is still relatively novel, especially to public health and biomedical data, this paper reviews applications of the approach to date with the aim of encouraging researchers to adopt FDA in future studies. This paper begins with a systematic review of the focus and application features of published peer-reviewed FDA studies. In doing so, it provides a summary of the extent to which FDA has been applied in different fields, including an overview of the nature of the time series variables/data used. For each of the identified studies, this paper also describes the features of FDA that were used, including the:

(1) representation of data via principal components analysis, which plays a key role in defining smoothness and continuity conditions of the resulting data;

(2) classification of data, which produces different functional groups (or clusters) for gaining more sophisticated knowledge of different pathways and/or functions for large scale data;

(3) functional linear models used for testing the effects on outcomes in functional form; and

(4) forecasting via stochastic methods, to measure the forecast uncertainty through the estimation of a prediction interval.

This review was conducted according to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) Statement [ 29 ]. We conducted a systematic search of 11 electronic databases to identify peer-reviewed FDA application studies published between January 1995 and December 2010. The databases used were Academic Search Premier, ScienceDirect, SpringerLink, Cambridge Journals, MEDGE (Informit), Oxford Journals, PubMed, Sage Journals Online, Web of Science, Wiley Interscience Journals, and MEDLINE. We used the phrase functional data analysis to identify relevant articles , and considered only English language articles published in peer-reviewed journals. In addition to the electronic database search, the search strategy included secondary searching of the reference lists of identified articles.

Inclusion and exclusion criteria

Studies were eligible for inclusion if they were original research articles in peer-reviewed journals reporting an application of FDA. We excluded studies of statistical methodology development without application, and abstracts, letters, and conference papers.

Identification of studies

The first author, with the assistance of two research assistants, sourced and screened all identified articles. This included viewing titles and reading abstracts. We obtained full text versions of potentially eligible articles, assessed them against the exclusion/inclusion criteria, and removed obvious exclusions.

In the first review phase, 334 articles were identified. Figure ​ Figure1 1 summarizes the numbers of studies identified and the reasons for exclusion at each stage. Searching the titles and abstracts of identified studies excluded 160 (47.9%) articles that were not directly relevant to statistical FDA applications. These included reports of functional magnetic resonance imaging (fMRI) to assess patterns of brain activation in patients suffering from chronic traumatic brain injury [ 30 ], functional performance in participants with functional ankle instability [ 31 ], and the relationship between neurocognitive function and noncontact anterior cruciate ligament injuries [ 32 ].

Systematic search strategy used to identify 84 peer-review studies with published application of functional data analysis (FDA).

In the second phase, we conducted a complete detailed review of the remaining 174 retrieved articles to ensure they fully met the inclusion and exclusion criteria. A further 102 articles were excluded at this stage, leaving 72 peer reviewed articles for the third phase. Studies excluded at this stage were mainly those that justified FDA theory rather than presenting examples of its application [ 8 , 33 - 35 ]. A further 12 articles were found in the manual search of reference lists of the 72 retained articles.

We retained a final set of 84 articles for detailed review. The lead author reviewed each paper in terms of key FDA criteria, as outlined below, and assessed its field of application and the specific FDA methods applied. Figure ​ Figure1 1 uses the PRISMA [ 29 ] flowchart to summarize all stages of the paper selection process.

Overview of the published FDA studies

Table  1 summarizes the final set of reviewed papers, and shows fields of application, outcome of interest, and use of the following important FDA features:

Areas of application and the functional data analysis (FDA) features used in the 84 peer-review papers reporting application of FDA

FPCA - Functional principal component analysis; FEM - Functional embedding; HCA - Hierarchical cluster analysis; SVM - Support vector machine; LDA- Linear discriminant analysis; QDA - Quadratic discriminant analysis; KNN- K-nearest neighbours; MBC - Model based clustering; CART - Classification and regression tree; EDO - Estimated differential operators; FLM - Functional linear model; FRM - Functional linear regression model; FANOVA - Functional ANOVA; FMANOVA - Functional MANOVA; FFT - Functional F test; FLRM - Functional logistic regression model; FAR - Functional auto regressive model.

• smoothing technique;

• use of functional principal component analysis (FPCA);

• type of clustering adjustment;

• functional linear modeling (FLM) approach adopted to relate explanatory and outcome variables; and

• type of forecasting (if any).

The earliest identified application of FDA was in 1995 and 75% of the reviewed articles were published since 2005. This reflects increasing recognition of the important features of functional data and awareness of the development of new statistical approaches and software for handling them.

While diverse fields were covered in the published studies, almost 21% of the studies related specifically to biomedical science (18 identified papers), followed by biomechanics applications (11 papers). Other fields of application were medicine (10), linguistics (6), biology (4), ecology (4), psychology (4), meteorology (4), environmental studies (4), demography (3), finance (3), neurology (2), economics (2), engineering (2), agriculture (1), physiology (1), information technology (1), education (1), chemistry (1), geophysics (1), and behavioral science (1). In relation to specific health issues, the most common topics were analyses of kinematics gait data (9 papers), magnetic resonance imaging (6 papers), and yeast cell cycle temporal gene expression profiles (6 papers).

Important features of the published FDA studies

Table  1 summarizes the published studies in terms of their use of the following key features of FDA: the reported smoothing technique, FPCA, clustering, the adopted forms of the FLM and forecasting. The importance of each of these features is explained below and an overview given of how they were handled in the published studies.

Smoothing techniques

Smoothing is the first step in any FDA, and its purpose is to convert raw discrete data points into a smoothly varying function. This emphasizes patterns in the data by minimizing short-term deviations due to observational errors, such as measurement errors or inherent system noise. When reporting FDA studies, it is important to state the smoothing approach used because observational errors always exist in longitudinal data.

Table  1 summarizes the various smoothing techniques used to estimate functions from the discrete observations reported in the reviewed literature. Overall, all except twelve of the studies (i.e. 85.7% of the reviewed studies) provided information about the type of smoothing technique used. Although some authors believe that FDA can be considered as a smoothed version of multivariate data analysis, smoothing techniques should still be used to reduce some of the inherent randomness in the observed data [ 1 , 25 , 113 ].

Overall, B-spline smoothing was the most popular smoothing technique used (25 papers), presumably because of its simplicity and flexibility for tackling a wide range of nonparametric and semiparametric modeling situations. A common approach towards B-spline smoothing is to construct a large number of knots (as the smoothing parameter) to reduce the effective degrees of freedom and increase smoothness in the overall function estimate [ 114 , 115 ]. Other smoothing techniques adopted in the published studies included use of Fourier smoothing (8 papers), regression splines (6), kernel smoothing (7), polynomial splines (5), cubic splines (3), smoothing splines (3), wavelet bases (3), roughness penalties (2), local polynomials (2), local quadratics (1), local weighted regression (1), P-splines (1) and log-splines (1).

Ramsay and Silverman [ 9 ] emphasize that the choice of smoothing technique is dependent upon the underlying behavior of the data being analyzed. Ideally, the smoother should reflect or have features that match those of the data. For example, Fourier smoothers are traditionally used when the data are cyclical or periodic. Environmental diurnal ozone and NO x cycles [ 71 , 116 ], trends in ecologically meaningful water quality variates in ecology [ 81 ], cash flows in finance [ 92 ] and fetal heart rate monitoring in medicine [ 18 , 19 ] are examples of the application of Fourier smoothers. Splines (regression splines, polynomial splines, B-spline) are typically chosen to represent noncyclical nonperiodic data [ 25 , 51 , 84 ], and wavelet bases are chosen to represent data displaying discontinuities and/or rapid changes in behavior [ 117 , 118 ]. Most recently, Ullah and Finch [ 23 ] used constrained penalized regression splines with a monotonic constraint to represent their smooth curves of falls incidence rates.

Data reduction

The FPCA is one of the most popular multivariate analysis techniques for extracting information from functional data [ 119 , 120 ]. This approach reduces the dimensions of a data set in which there are a large number of interrelated variables, while still holding as much of the total variation as possible. This reduction is obtained by transforming the data to a new set of variables, or principal components, that are uncorrelated and ordered so that the first few retain most of the variation present in all of the original variables.

The use of FPCA was reported in 51 (60.7%) of the reviewed studies (Table  1 ). It has been successfully applied to real life problems such as modeling the curvature of the cornea in the human eye [ 11 ], in a set of density curves where the argument variable is log income [ 121 ], and fMRI scans of areas in the human brain [ 88 ]. Many different applications of principal component analysis to functional data have been developed, including a useful extension of FPCA that allows the estimation of harmonics from fragments of curves [ 122 ]. Although FPCA is an important feature of FDA, not all studies reported it because they did not undertake data reduction. For example, Roislin et al. [ 48 ] used a functional regression model to estimate the effects of gender, age, and walking speed on kinematic gait data; Park et al. [ 58 ] classified gene functions using a support vector machine (clustering) for time-course gene expression data; and Lucero [ 93 ] used only a B-spline to smooth the harmonics-to-noise ratio of voice signals. None of these applications required FPCA to reduce the data.

While FPCA results in dimension reduction, FPCA vector scores can be used for clustering different functions/components using standard clustering methods. Clustering is one of the most frequently used techniques for partitioning a dataset into subgroups that contain instances that are similar to each other while being clearly dissimilar to those of other groups. In a functional context, clustering helps to identify representative curve patterns and individuals who are very likely to be involved in the same or similar processes. For example, in time-course microarray experiments, thousands of gene expression measures are taken over time [ 123 ] and an important problem is to discover functionally related genes that could then be the target for new gene regulatory networks or functional pathways. Clustering of data allows identification of groups of genes with similar expression patterns to identify such networks and pathways.

A number of clustering methods were reported in the reviewed literature (Table  1 ) and most of these were exploratory techniques for gene expression data. Overall, 15 studies (17.9%) reported some form of clustering. Biologists and ecologists used clustering to classify genes [ 68 , 76 ] and ecological components [ 81 , 90 ] within their studies. The most commonly applied clustering algorithms were hierarchical in nature (7 papers). Hierarchical algorithms define a dendrogram (tree) relating similar objects in the same sub-trees. In each step, similar sub-trees (clusters) are merged to form a dendrogram that clearly shows the different distinct clusters. Other reported clustering methods were linear discriminant analysis (LDA) (2 papers), k-nearest neighbors (KNN) (3), support vector machine (SVM) approaches (2), model-based clustering (MBC) (1), quadratic discriminant analysis (QDA) (1) and estimated differential operators (EDO) (1).

The LDA and QDA are both classical clustering methods and commonly used in microarray analysis [ 124 , 125 ]. Application of LDA is based on finding linear combinations of gene expression levels called discriminants that maximize the ratio of between-group variation to within-group variation. The QDA approach is a generalization of the linear classifier, allowing covariance matrices to be heterogeneous, whereas LDA functions are based on the assumption that covariance matrices of each of the classes are equal. This assumption relaxation can prevent individuals from being placed into classes with larger variance on their covariance matrix diagonals. The KNN is a nonparametric classification method based on the distance between individuals [ 126 , 127 ]. For example, Song et al. [ 59 ] proposed KNN to classify time-course gene expression profiles based on information from the data patterns. The SVM approach [ 128 ] is an extremely powerful methodology for classification problems and has a wide range of applications. Recently, this method has received much attention in classification problems that arise with the analysis of microarray data [ 58 , 59 ]. The MBC method assumes that the data are generated by a multivariate normal mixture distribution with appropriate means and covariance matrix [ 129 ]. Song et al. [ 68 ] have applied this method of clustering time-course gene expression data.

Functional linear models

An interesting application of FDA involves the construction of functional models that describe the relation between an outcome variable and an explanatory variable. Such models are termed functional linear models (FLMs). The number of published applications involving functional data has been steadily growing. In functional linear models, the functions could be the outcome or the predictors or both.

Of the reviewed studies in Table  1 , 21 (25.0%) reported some form of FLM. The approach most favored by authors was a basic functional linear regression model (12 papers). When the outcome variable is in its functional form and the relationship is almost linear, the methodology is called functional linear regression model, or FRM. Functional ANOVA (FANOVA) was used in eight studies. Vines et al. [ 85 ] developed a functional F test (FFT) for linear models with functional outcomes in their psychological study for measuring tension judgment in music. They illustrated how to apply the FFT to longitudinal data where intrasubject repeated measures are viewed as discrete samples from an underlying curve with a continuous functional form. One study applied a functional logistic regression model (FLRM) to fetal heart rates [ 18 ] and another applied functional multivariate analysis of variance (FMANOVA) to temporal fertility trajectories of medfly populations [ 47 ].

Forecasting framework

The recent introduction of stochastic methods for forecasting functional data has significant advantages over the standard approaches for better understanding trends, risk factor relationships, and the effectiveness of preventive measures. A major advantage of these methods is that they can measure forecast uncertainty through the estimation of prediction intervals for future data. For this reason, the FDA forecasting approach has started to receive attention in both demographic and medical applications [ 24 , 25 , 28 , 60 ]. To date there has only been limited application of FDA to epidemiological studies relating to the prediction of incidence/prevalence rates, with only one recent study applying it to forecast the incidence of fall-related severe head injuries [ 23 ].

Overall, only seven of the reviewed studies (8.3%) reported any FDA-derived forecasting. A state space model was the most common approach for forecasting functional data in these studies (5 papers). In the forecasting process, the authors estimated the coefficients from a time series, with one value representing each time point, and a state space model was used to model and forecast these time series coefficients [ 23 - 25 , 28 , 130 ].

Modern data analysis has greatly benefited from the development of FDA methods and their application to time series data. Although used by statisticians for many years, FDA provides a relatively novel approach to modeling and prediction that is highly suitable for public health and biomedical applications. This paper has summarized papers describing FDA applications with a main emphasis on five popular features: smoothing, FPCA, clustering, FLM, and forecasting.

Overall, the published FDA application studies demonstrate the value of this approach for exploring complex multivariate functional relationships and its major strength of being able to model the functional form of time series data. Different approaches allow for FDA representations as smooth functions, and the published studies used a range of smoothing techniques for the estimation of discretely observed data. The FDA approach of initially smoothing the data and then using the smoothed observations for modeling and forecasting is a major methodological improvement over methods that simply fit linear/non-linear trends to observed data. These FDA approaches are very suitable for widespread public health and biomedical applications. Although some authors believe that FDA can be considered as a smoothed version of multivariate data analysis, recent work has shown the advantage of direct application of smoothing techniques to reduce some of the inherent randomness in the observed data [ 1 , 25 , 113 ].

The theoretical and practical developments that have occurred over recent years mean that researchers can now successfully apply FPCA to many practical problems, with main attention given to the reduction of data dimensions to a finite level and identification of the most significant components of the data. High dimensional data significantly slow down conventional statistical algorithms and in some cases it is not feasible to use them in practice. This means that standard classification methods can suffer from difficulties when handling such data. Some studies need to compress their data to facilitate exploration of the most important features (e.g., characteristics of genes from entire time-course data). In such instances, dimension reduction should be applied to keep only the relevant information and for removing correlations. This will both speed up and improve the accuracy of subsequent analyses and modeling. The FPCA has proven to be a key technique for dimension reduction, reported in most of studies reviewed here. It can also be used to investigate the variability of data with respect to individual curve shapes [ 131 ].

One of the major application areas highlighted in this review is an apparent increasing interest in clustering and classification techniques, especially for time-course gene expression data. The clustering is useful for detecting patterns and clusters in high dimensional functional data. Functional clustering is used to search for natural groupings of data with similar characteristics. Unlike conventional clustering that requires measuring multivariate data at the same time points to calculate Euclidean ‘distances’ between observations, functional clustering can derive a broader class of distance measures even if the original measurements are not time-aligned among sampling units, as is common in public health applications. The reason for the popularity of functional clustering is that it can classify time series data into different classes without requiring a priori knowledge of data.

A very interesting application of FDA involves the construction of linear models that describe the relation between an outcome variable and explanatory variables with functional nature. The FLMs have recently gained popularity and the related literature has been steadily growing with several studies using covariates to explain functional variables. Overall, FRM and FANOVA methods were the most prominent in the reviewed literature. Reasons for not using FLM techniques are unclear but might include a lack of knowledge about the value of building functional models for public health and biomedical data. However, the use of FLM is not always necessary and depends on the specific research questions.

Public health researchers now recognize the importance of understanding trends in high dimensional time series data. Policy makers, for example, need information about predicted trends to inform their decision-making about public health and economic investments to reduce the burden into the future [ 132 ]. It is critical that such predictions are robust and based on the best available statistical modeling approaches to minimize possible errors in the forecasts. This is also true for other areas of public health and biomedicine. The new FDA forecasting approaches [ 23 , 25 ] are a natural extension of methods developed for mortality and fertility forecasting that have evolved over the last two decades in demography [ 25 , 133 , 134 ]. The methodology has therefore been used in a number of demographic applications and there have been various extensions and modifications proposed [ 25 , 134 ]. Somewhat surprisingly, the use of FDA forecasting in public health and biomedical applications has been limited to date.

In summary, this paper describes FDA and its important features as applied to time series data from various fields. Functional data analysis provides a relatively novel modeling and prediction approach, with the potential for many significant applications across a range of public health and biomedical applications. Importantly, not all FDA features always need to be used in a single study and the selection of specific analysis features will depend on the underlying behavior of the data, the nature of study and the specific research questions being posed. Consideration should be given to wider application of FDA and its important modeling features so that more accurate estimates for public health and biomedical applications can be generated.

Abbreviations

FDA: Functional data analysis; FPCA: Functional principal component analysis; FLM: Functional linear modeling; LDA: Linear discriminant analysis; KNN: K-nearest neighbours; SVM: Support vector machine; MBC: Model based clustering; QDA: Quadratic discriminant analysis; EDO: Estimated differential operators; FRM: Functional linear regression model; FANOVA: Functional ANOVA; FFT: Functional F-test; FLRM: Functional logistic regression model; FMANOVA: Functional multivariate analysis of variance.

Competing interests

The authors have no conflicts of interest that are directly relevant to the content of this review.

Authors’ contributions

As first author, SU conceived and designed the study reported in this paper. He took the lead role in drafting the manuscript and reviewed all relevant articles and analysed their content. The second author, CF, provided expertise in the conduct of systematic reviews and also contributed to the writing and editing of the paper. She reviewed the article and revised it critically for important intellectual content. Both authors read and approved the final manuscript.

Pre-publication history

The pre-publication history for this paper can be accessed here:

http://www.biomedcentral.com/1471-2288/13/43/prepub

Acknowledgements

The study was funded (at least in part) through the Early Career Researcher development funding program at the University of Ballarat. Professor Caroline Finch was supported by a National Health and Medical Research Council (NHMRC) Principal Research Fellowship (ID: 565900). Peter Richardson and Dr Saad Saleem assisted with the original literature searches and retrieval of published articles for the review. Eileen Clark proofread and copy edited the manuscript.

• Green PJ, Silverman BW. Nonparametric regression and generalized linear models: A roughness penalty approach. London: Chapman and Hall; 1994. [ Google Scholar ]
• Ramsay JO. When the data are functions? Psychometrika. 1982; 47 :379–396. doi: 10.1007/BF02293704. [ CrossRef ] [ Google Scholar ]
• Ramsay JO. Monotone regression splines in action. Statist Sci. 1988; 3 :425–441. doi: 10.1214/ss/1177012761. [ CrossRef ] [ Google Scholar ]
• Ramsay JO, Dalzell CJ. Some tools for functional data analysis. J R Stat Soc Series B Stat Methodol. 1991; 53 :539–572. [ Google Scholar ]
• Müller HG. Functional data analysis. StatProb: The Encyclopedia Sponsored by Statistics and Probability Societies; 2011. [ Google Scholar ]
• Ferraty F, Mas A, Vieu P. Advances in nonparametric regression for functional variables. Aust N Z J Stat. 2007; 49 :1–20. [ Google Scholar ]
• Mas A, Pumo B. Functional linear regression with derivatives. J Nonparametr Stat. 2009; 21 :19–40. doi: 10.1080/10485250802401046. [ CrossRef ] [ Google Scholar ]
• Levitin DJ, Nuzzo RL, Vines BW, Ramsay JO. Introduction to functional data analysis. Can Psychol. 2007; 48 :135–155. [ Google Scholar ]
• Ramsay JO, Silverman BW. Functional data analysis. 2. New York: Springer; 2005. [ Google Scholar ]
• Ramsay JO, Silverman BW. Applied functional data analysis . New York: Springer; 2002. [ Google Scholar ]
• Locantore N, Marron JS, Simpson DG, Tripoli N, Zhang JT, Kohen KL. Robust principal component analysis for functional data. TEST. 1999; 8 :1–73. doi: 10.1007/BF02595862. [ CrossRef ] [ Google Scholar ]
• Yushkevich P, Pizer SM, Joshi S, Maron JS. Intuitive, localized analysis of shape variability. 2001, 2082/2001. pp. 402–408. (Proceedings of the Inf Process Med Imaging).
• Besse PC, Cardot H. Autoregressive forecasting of some functional climate variations. Scand Stat Theory Appl. 2000; 27 :673–687. [ Google Scholar ]
• Ramsay JO. Functional components of variation in handwriting. J Am Stat Assoc. 2000; 95 :9–15. doi: 10.1080/01621459.2000.10473894. [ CrossRef ] [ Google Scholar ]
• Abraham C, Cornillon PA, MatznerLober E, Molinari N. Unsupervised curve clustering using B-splines. Scand Stat Theory Appl. 2003; 30 :581–595. [ Google Scholar ]
• Besse PC, Cardot H, Faivre R, Goulard M. Statistical modelling of functional data. Appl Stoch Model Bus Ind. 2005; 21 :165–173. doi: 10.1002/asmb.539. [ CrossRef ] [ Google Scholar ]
• Pfeiffer RM, Bura E, Smith A, Rutter JL. Two approaches to mutation detection based on functional data. Stat Med. 2002; 21 :3447–3464. doi: 10.1002/sim.1269. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Ratcliffe SJ, Heller GZ, Leader LR. Functional data analysis with application to periodically stimulated foetal heart rate data. II: functional logistic regression. Stat Med. 2002; 21 :1115–1127. doi: 10.1002/sim.1068. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Ratcliffe SJ, Leader LR, Heller GZ. Functional data analysis with application to periodically stimulated foetal heart rate data. I: functional regression. Stat Med. 2002; 21 :1103–1114. doi: 10.1002/sim.1067. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Rossi N, Wang X, Ramsay JO. Nonparametric item response function estimates with the EM algorithm. J Educ Behav Stat. 2002; 27 :291–317. doi: 10.3102/10769986027003291. [ CrossRef ] [ Google Scholar ]
• Kargin V, Onatski A. Curve forecasting by functional autoregression. J Multivar Anal. 2008; 99 :2508–2526. doi: 10.1016/j.jmva.2008.03.001. [ CrossRef ] [ Google Scholar ]
• Reiss PT, Ogden RT. Functional principal component regression and functional partial least squares. J Am Stat Assoc. 2007; 102 :984–996. doi: 10.1198/016214507000000527. [ CrossRef ] [ Google Scholar ]
• Ullah S, Finch CF. Functional data modelling approach for analysing and predicting trends in incidence rates-An application to falls injury. Osteoporos Int. 2010; 21 :2125–2134. doi: 10.1007/s00198-010-1189-2. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Erbas B, Hyndman RJ, Gertig DM. Forecasting age-specific breast cancer mortality using functional data models. Stat Med. 2007; 26 :458–470. doi: 10.1002/sim.2306. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Hyndman RJ, Ullah MS. Robust forecasting of mortality and fertility rates: a functional data approach. Comput Stat Data Anal. 2007; 51 :4942–4956. doi: 10.1016/j.csda.2006.07.028. [ CrossRef ] [ Google Scholar ]
• Rice JA. Functional and longitudinal data analysis: perspectives on smoothing. Statist Sci. 2004; 14 :631–647. [ Google Scholar ]
• Davidian M, Lin X, Wang JL. Introduction: emerging issues in longitudinal and functional data analysis. Statist Sci. 2004; 14 :613–614. [ Google Scholar ]
• Ullah S. Demographic forecasting using functional data analysis. Monash University: PhD Thesis; 2007. [ Google Scholar ]
• Moher D, Liberati A, Tetzlaff J, Altman DG. The PRISMA Group. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med. 2009; 6 :e1000097. doi: 10.1371/journal.pmed.1000097. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Rasmussen IJ, Antonsen IK, Berntsen EM, Xu J, Lagopoulos J, Haberg AK. Brain activation measured using functional magnetic resonance imaging during the Tower of London task. Acta Neuropsychiatr. 2006; 18 :216–225. doi: 10.1111/j.1601-5215.2006.00145.x. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Buchanan AS, Docherty CL, Schrader J. Functional performance testing in participants with functional ankle instability and in a healthy control group. J Athl Train. 2008; 43 :342–346. doi: 10.4085/1062-6050-43.4.342. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Swanik CB, Covassin T, Stearne DJ, Schatz P. The relationship between neurocognitive function and noncontact anterior cruciate ligament injuries. Am J Sports Med. 2007; 35 :943–948. doi: 10.1177/0363546507299532. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Gabrys R, Kokoszka P. Portmanteau test of independence for functional observations. J Am Stat Assoc. 2007; 102 :1338–1348. doi: 10.1198/016214507000001111. [ CrossRef ] [ Google Scholar ]
• Manteiga WG, Vieu P. Statistics for functional data. Comput Stat Data Anal. 2007; 51 :4788–4792. doi: 10.1016/j.csda.2006.10.017. [ CrossRef ] [ Google Scholar ]
• Cardot H, Ferraty F, Mas A, Sarda P. Testing hypotheses in the functional linear model. Scand Stat Theory Appl. 2003; 30 :241–255. [ Google Scholar ]
• Dura JV, Belda JM, Poveda R, Page A, Laparra J, Das J, Prat J, Garcia AC. Comparison of functional regression and non-functional regression approaches to the study of the walking velocity effect in force platform measures. J Appl Biomech. 2010; 26 :234–239. [ PubMed ] [ Google Scholar ]
• Crane EA, Cassidy RB, Rothman ED, Gerstner GE. Effect of registration on cyclical kinematic data. J Biomech. 2010; 43 :2444–2447. doi: 10.1016/j.jbiomech.2010.04.024. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Zhu HT, Styner M, Tang NS, Liu ZX, Lin WL, Gilmore JH. FRATS: Functional regression analysis of DTI tract statistics. IEEE Trans Med Imaging. 2010; 29 :1039–1049. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Wu P, Müller H. Functional embedding for the classification of gene expression profiles. Bioinformatics. 2010; 26 :509–517. doi: 10.1093/bioinformatics/btp711. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Kim SB, Rattakorn P, Peng YB. An effective clustering procedure of neuronal response profiles in graded thermal stimulation. Expert Syst Appl. 2010; 37 :5818–5826. doi: 10.1016/j.eswa.2010.02.025. [ CrossRef ] [ Google Scholar ]
• Hyndman RJ, Shang HL. Rainbow plots, bagplots, and boxplots for functional Data. J Comput Graph Stat. 2010; 19 :29–45. doi: 10.1198/jcgs.2009.08158. [ CrossRef ] [ Google Scholar ]
• Torres JM, Garcia Nieto PJ, Alejano L, Reyes AN. Detection of outliers in gas emissions from urban areas using functional data analysis. J Hazard Mater. 2010; 186 :144–149. [ PubMed ] [ Google Scholar ]
• Maslova I, Kokoszka P, Sojka J, Zhu L. Statistical significance testing for the association of magnetometer records at high-, mid- and low latitudes during substorm days. Planet Space Sci. 2010; 58 :437–445. doi: 10.1016/j.pss.2009.11.004. [ CrossRef ] [ Google Scholar ]
• Hermanussen M. Auxology: An update. Horm Res Paediatr. 2010; 74 :153–164. doi: 10.1159/000317440. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Erbas B, Akram M, Gertig DM, English D, Hopper JL, Kavanagh AM, Hyndman RJ. Using functional data analysis models to estimate future time trends in age-specific breast cancer mortality for the United States and England-Wales. J Epidemiol. 2010; 20 :159–165. doi: 10.2188/jea.JE20090072. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Ogden RT, Greene E. Wavelet modeling of functional random effects with application to human vision data. J Stat Plan Inference. 2010; 140 :3797–3808. doi: 10.1016/j.jspi.2010.04.044. [ CrossRef ] [ Google Scholar ]
• Muller HG, Wu S, Diamantidis AD, Papadopoulos NT, Carey JR. Reproduction is adapted to survival characteristics across geographically isolated medfly populations. Proc Biol Sci. 2009; 276 :4409–4416. doi: 10.1098/rspb.2009.1461. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Roislien J, Skare O, Gustavsen M, Broch NL, Rennie L, Opheim A. Simultaneous estimation of effects of gender, age and walking speed on kinematic gait data. Gait Posture. 2009; 30 :441–445. doi: 10.1016/j.gaitpost.2009.07.002. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Gouttard S, Prastawa M, Bullitt E, Lin W, Gerig G. Constrained data decomposition and regression for analyzing healthy aging from fiber tract diffusion properties. Med Image Comput Comput Assist Interv. 2009; 12 :321–328. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Illian JB, Prosser JI, Baker KL, Rangel-Castro JL. Functional principal component data analysis: a new method for analysing microbial community fingerprints. J Microbiol Methods. 2009; 79 :89–95. doi: 10.1016/j.mimet.2009.08.010. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Parker BJ, Wen J. Predicting microRNA targets in time-series microarray experiments via functional data analysis. BMC Bioinforma. 2009; 10 :S32. doi: 10.1186/1471-2105-10-S1-S32. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Goodlett CB, Fletcher PT, Gilmore GH, Gerig G. Group statistics of DTI fiber bundles using spatial functions of tensor measures. Med Image Comput Comput Assist Interv. 2008; 11 :1068–1075. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Jiang CR, Aston JA, Wang JL. Smoothing dynamic positron emission tomography time courses using functional principal components. NeuroImage. 2009; 47 :184–193. doi: 10.1016/j.neuroimage.2009.03.051. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Lee KL, Meyer RJ, Bradlow ET. Analyzing risk response dynamics on the web: the case of Hurricane Katrina. Risk Anal. 2009; 29 :1779–1792. doi: 10.1111/j.1539-6924.2009.01304.x. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Oa D, Harrison AJ, Coffey N, Hayes K. Functional data analysis of running kinematics in chronic Achilles tendon injury. Med Sci Sports Exerc. 2008; 40 :1323–1335. doi: 10.1249/MSS.0b013e31816c4807. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Baladandayuthapani V, Mallick BK, Young HM, Lupton JR, Turner ND, Carroll RJ. Bayesian hierarchical spatially correlated functional data analysis with application to colon carcinogenesis. Biometrics. 2008; 64 :64–73. doi: 10.1111/j.1541-0420.2007.00846.x. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Muller H, Chiou J, Leng X. Inferring gene expression dynamics via functional regression analysis. BMC Bioinforma. 2008; 9 :60. doi: 10.1186/1471-2105-9-60. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Park C, Koo J, Kim S, Sohn I, Lee JW. Classification of gene functions using support vector machine for time-course gene expression data. Comput Stat Data Anal. 2008; 52 :2578–2587. doi: 10.1016/j.csda.2007.09.002. [ CrossRef ] [ Google Scholar ]
• Song JJ, Deng W, Lee H, Kwon D. Optimal classification for time-course gene expression data using functional data analysis. Comp Bio Chem. 2008; 32 (6):426–432. doi: 10.1016/j.compbiolchem.2008.07.007. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Hyndman R, Booth H. Stochastic population forecasts using functional data models for mortality, fertility and migration. Int J Forecast. 2008; 24 :323–342. doi: 10.1016/j.ijforecast.2008.02.009. [ CrossRef ] [ Google Scholar ]
• Ikeda T, Dowd M, Martin JL. Application of functional data analysis to investigate seasonal progression with interannual variability in plankton abundance in the bay of Fundy, Canada. Estuar Coast Shelf Sci. 2008; 78 :445–455. doi: 10.1016/j.ecss.2007.12.011. [ CrossRef ] [ Google Scholar ]
• Guo W. Functional data analysis in longitudinal settings using smoothing splines. Stat Methods Med Res. 2004; 13 :49–62. doi: 10.1191/0962280204sm352ra. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Laukaitis A. Functional data analysis for cash flow and transactions intensity continuous-time prediction using Hilbert-valued autoregressive processes. Eur J Oper Res. 2008; 185 :1607–1614. doi: 10.1016/j.ejor.2006.08.030. [ CrossRef ] [ Google Scholar ]
• Bapna R, Jank W, Shmueli G. Price formation and its dynamics in online auctions. Decis Support Syst. 2008; 44 :641–656. doi: 10.1016/j.dss.2007.09.004. [ CrossRef ] [ Google Scholar ]
• Koenig LL, Lucero JC, Perlman E. Speech production variability in fricatives of children and adults: results of functional data analysis. J Acoust Soc Am. 2008; 124 :3158–3170. doi: 10.1121/1.2981639. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Harezlak J, Wu MC, Wang M, Schwartzman A, Christiani DC, Lin X. Biomarker discovery for arsenic exposure using functional data. Analysis and feature learning of mass spectrometry proteomic data. J Proteome Res. 2008; 7 :217–224. doi: 10.1021/pr070491n. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Chapados C, Levitin DJ. Cross-modal interactions in the experience of musical performances: physiological correlates. Cognition. 2008; 108 :639–651. doi: 10.1016/j.cognition.2008.05.008. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Song JJ, Lee H, Morris JS, Kang S. Clustering of time-course gene expression data using functional data analysis. Comp Bio Chem. 2007; 31 :265–274. doi: 10.1016/j.compbiolchem.2007.05.006. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Vakorin VA, Borowsky R, Sarty GE. Characterizing the functional MRI response using Tikhonov regularization. Stat Med. 2007; 26 :3830–3844. doi: 10.1002/sim.2981. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Dabo-niang S, Vieu P. On the using of modal curves for radar waveforms classification. Comput Statist Data Anal. 2007; 51 :48–78. [ Google Scholar ]
• Gao H. Day of week effects on diurnal ozone/NOx cycles and transportation emissions in Southern California. Transp Res Part D Transp Envr. 2007; 12 :292–305. doi: 10.1016/j.trd.2007.03.004. [ CrossRef ] [ Google Scholar ]
• Meiring W. Oscillations and time trends in stratospheric ozone levels. J Am Stat Assoc. 2007; 102 :788–802. doi: 10.1198/016214506000000825. [ CrossRef ] [ Google Scholar ]
• Meyer PM, Zeger SL, Harlow SD, Sowers M, Crawford S, Luborsky JL, Janssen I, McConnell DS, Randolph JF, Weiss G. Characterizing daily urinary hormone profiles for women at midlife using functional data analysis. Am J Epidemiol. 2007; 165 :936–945. doi: 10.1093/aje/kwk090. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• West RM, Harris K, Gilthorpe MS, Tolman C, Will EJ. Functional data analysis applied to a randomized controlled clinical trial in hemodialysis patients describes the variability of patient responses in the control of renal anemia. J Am Soc Nephrol. 2007; 18 :2371–2376. doi: 10.1681/ASN.2006050436. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Harrison A, Ryan W, Hayes K. Functional data analysis of joint coordination in the development of vertical jump performance. Sports Biomech. 2007; 6 :199–214. doi: 10.1080/14763140701323042. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Leng X, Muller H. Classification using functional data analysis for temporal gene expression data. Bioinformatics. 2006; 22 :68–76. doi: 10.1093/bioinformatics/bti742. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Zhang Y, Muller HG, Carey JR, Papadopoulos NT. Behavioral trajectories as predictors in event history analysis: male calling behavior forecasts medfly longevity. Mech Ageing Dev. 2006; 127 :680–686. doi: 10.1016/j.mad.2006.04.001. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Duhamel A, Devos P, Bourriez JL, Preda C, Defebvre L, Beuscart R. Functional data analysis for gait curves study in Parkinson’s disease. Stud Health Technol Inform. 2006; 124 :569–574. [ PubMed ] [ Google Scholar ]
• Ryan W, Harrison A, Hayes K. Functional data analysis of knee joint kinematics in the vertical jump. Sports Biomech. 2006; 5 :121–138. doi: 10.1080/14763141.2006.9628228. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Page A, Ayala G, Leon MT, Peydro MF, Prat JM. Normalizing temporal patterns to analyze sit-to-stand movements by using registration of functional data. J Biomech. 2006; 39 :2526–2534. doi: 10.1016/j.jbiomech.2005.07.032. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Henderson B. Exploring between site differences in water quality trends: a functional data analysis approach. Environmetrics. 2006; 17 :65–80. doi: 10.1002/env.750. [ CrossRef ] [ Google Scholar ]
• Stewart K, Darcy D, Daniel S. Opportunities and challenges applying functional data analysis to the study of open source software evolution. Stat Sci. 2006; 21 :167–178. doi: 10.1214/088342306000000141. [ CrossRef ] [ Google Scholar ]
• Lee S, Byrd D, KrivokapicÌ J. Functional data analysis of prosodic effects on articulatory timing. J Acoust Soc Am. 2006; 119 :1666–1671. doi: 10.1121/1.2161436. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Newell J, McMillan K, Grant S, McCabe G. Using functional data analysis to summarise and interpret lactate curves. Comput Biol Med. 2006; 36 :262–275. doi: 10.1016/j.compbiomed.2004.11.006. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Vines BW, Krumhansl CL, Wanderley MM, Levitin DJ. Cross-modal interactions in the perception of musical performance. Cognition. 2006; 101 :80–113. doi: 10.1016/j.cognition.2005.09.003. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Bensmail H, Aruna B, Semmes OJ, Haoudi A. Functional clustering algorithm for high-dimensional proteomics data. J Biomed Biotechno. 2005; 2 :80–86. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Ormoneit D, Black MJ, Hastie T, Sidenbladh H. Representing cyclic human motion using functional analysis. Image Vision Comput. 2005; 23 :1264–1276. doi: 10.1016/j.imavis.2005.09.004. [ CrossRef ] [ Google Scholar ]
• Viviani R, Gron G, Spitzer M. Functional principal component analysis of fMRI data. Hum Brain Mapp. 2005; 24 :109–129. doi: 10.1002/hbm.20074. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Long CJ, Brown EN, Triantafyllou C, Aharon I, Wald LL, Solo V. Nonstationary noise estimation in functional MRI. NeuroImage. 2005; 28 :890–903. doi: 10.1016/j.neuroimage.2005.06.043. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Manté C, Durbec JP, Dauvin JC. A functional data-analytic approach to the classification of species according to their spatial dispersion. Application to a marine macrobenthic community from the Bay of Morlaix (western English Channel) J Appl Stat. 2005; 32 :831–840. doi: 10.1080/02664760500080124. [ CrossRef ] [ Google Scholar ]
• Rupp A. Quantifying subpopulation differences for a lack of invariance using complex examinee profiles: an exploratory multigroup approach using functional data analysis. Edu Res Eval. 2005; 11 :71–97. doi: 10.1080/13803610500110430. [ CrossRef ] [ Google Scholar ]
• Laukaitis A. Functional data analysis for clients segmentation tasks. Eur J Oper Res. 2005; 163 :210–216. doi: 10.1016/j.ejor.2004.01.010. [ CrossRef ] [ Google Scholar ]
• Lucero JC. Comparison of measures of variability of speech movement trajectories using synthetic records. J Speech Lang Hear Res. 2005; 48 :336–344. doi: 10.1044/1092-4388(2005/023). [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Vines BW, Nuzzo RL, Levitin DJ. Analysing temporal dynamics in music. Music Percept. 2005; 23 :137–152. doi: 10.1525/mp.2005.23.2.137. [ CrossRef ] [ Google Scholar ]
• Hutchinson RA, McLellan PJ, Ramsay JO, Sulieman H, Bacon DW. Investigating the impact of operating parameters on molecular weight distributions using functional regression. Macromol Symp. 2004; 206 :495–508. doi: 10.1002/masy.200450238. [ CrossRef ] [ Google Scholar ]
• Stier AW, Stein HJ, Schwaiger M, Heidecke CD. Modeling of esophageal bolus flow by functional data analysis of scintigrams. Dis Esophagus. 2004; 17 :51–57. doi: 10.1111/j.1442-2050.2004.00373.x. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Liggett WS, Barker PE, Semmes OJ, Cazares LH. Measurement reproducibility in the early stages of biomarker development. Dis Markers. 2004; 20 :295–307. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Buckner RL, Head D, Parker J, Fotenos AF, Marcus D, Morris JC, Snyder AZ. A unified approach for morphometric and functional data analysis in young, old, and demented adults using automated atlas-based head size normalization: reliability and validation against manual measurement of total intracranial volume. NeuroImage. 2004; 23 :724–738. doi: 10.1016/j.neuroimage.2004.06.018. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• McAdams S. Influences of large-scale form on continuous ratings in response to a contemporary piece in a live concert setting. Music Percept. 2004; 22 :297–350. doi: 10.1525/mp.2004.22.2.297. [ CrossRef ] [ Google Scholar ]
• Spitzner DJ, Marron JS, Essick GK. Mixed-model functional ANOVA for studying human tactile perception. J Am Stat Assoc. 2003; 98 :263–272. doi: 10.1198/016214503000035. [ CrossRef ] [ Google Scholar ]
• Yao F, Muller HG, Clifford AJ, Dueker SR, Follett J, Lin Y, Buchholz BA, Vogel JS. Shrinkage estimation for functional principal component scores with application to the population kinetics of plasma folate. Biometrics. 2003; 59 :676–685. doi: 10.1111/1541-0420.00078. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Ogden RT, Miller CE, Takezawa K, Ninomiya S. Functional regression in crop lodging assessment with digital images. J Agric Biol Environ Stat. 2002; 7 :389–402. doi: 10.1198/108571102339. [ CrossRef ] [ Google Scholar ]
• Clarysse P, Han M, Croisille P, Magnin IE. Exploratory analysis of the spatio-temporal deformation of the myocardium during systole from tagged MRI. IEEE Trans Biomed Eng. 2002; 49 :1328–1339. doi: 10.1109/TBME.2002.804587. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Ramsay JO, Ramsey BJ. Functional data analysis of the dynamics of the monthly index of nondurable goods production. J Econ. 2002; 107 :327–344. [ Google Scholar ]
• Hall P, Poskitt D, Presnell B. A functional data-analytic approach to signal discrimination. Technometrics. 2001; 43 :1–24. doi: 10.1198/00401700152404273. [ CrossRef ] [ Google Scholar ]
• Lucero JC, Koenig LL. Time normalization of voice signals using functional data analysis. J Acoust Soc Am. 2000; 108 :1408–1420. doi: 10.1121/1.1289206. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Besse PC, Cardot H, Stephenson DB. Autoregressive forecasting of some functional climatic variations. Scand Stat Theory Appl. 2000; 27 :673–687. [ Google Scholar ]
• Lucero JC. Computation of the harmonics-to-noise ratio of a voice signal using a functional data analysis algorithm. J Sound Vibrat. 1999; 222 :512–520. doi: 10.1006/jsvi.1998.2072. [ CrossRef ] [ Google Scholar ]
• Bjornstad O, Chr SN, Saitoh T, Lingjaerde OC. Mapping the regional transition to cyclicity in clethrionomys rufocanus: spectral densities and functional data analysis. Res Pop Ecol. 1998; 40 :77–84. doi: 10.1007/BF02765223. [ CrossRef ] [ Google Scholar ]
• Ramsay JO, Munhall KG, Gracco VL, Ostry DJ. Functional data analyses of lip motion. J Acoust Soc Am. 1996; 99 :3718–3727. doi: 10.1121/1.414986. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Ramsay JO, Wang X. A functional data analysis of the pinch force of human fingers. App Stat. 1995; 44 :17–30. doi: 10.2307/2986192. [ CrossRef ] [ Google Scholar ]
• Grambsch PM, Randall BL, Bostick RM, Potter JD, Louis TA. Modeling the labeling index distribution: An application of functional data analysis. J Am Stat Assoc. 1995; 90 :813–821. doi: 10.1080/01621459.1995.10476579. [ CrossRef ] [ Google Scholar ]
• Eubank RL. Nonparametric regression and spline smoothing. New York: Marcel Dekker; 1999. [ Google Scholar ]
• Marx BD, Eilers PHC. Direct generalized additive modeling with penalized likelihood. Comput Stat Data Anal. 1998; 28 :193–209. doi: 10.1016/S0167-9473(98)00033-4. [ CrossRef ] [ Google Scholar ]
• Wood SN. Modelling and smoothing parameter estimation with multiple quadratic penalties. J R Stat Soc Series B Stat Methodol. 2000; 62 :413–428. doi: 10.1111/1467-9868.00240. [ CrossRef ] [ Google Scholar ]
• Gao H, Niemeier D. Using functional data analysis of diurnal ozone and NOx cycles to inform transportation emissions control. Transp Res Part D Transp Envr. 2008; 13 :221–238. doi: 10.1016/j.trd.2008.02.003. [ CrossRef ] [ Google Scholar ]
• Ruppert D, Wand MP, Carroll RJ. Semiparametric regression. Cambridge: New York; 2003. [ PMC free article ] [ PubMed ] [ Google Scholar ]
• Simonoff JS. Smoothing methods in statistics. New York: Springer; 1996. [ Google Scholar ]
• Croux C, RuizGazen A. High breakdown estimators for principal components: The project-pursuit approach revisited. J Multivar Anal. 2005; 95 :206–226. doi: 10.1016/j.jmva.2004.08.002. [ CrossRef ] [ Google Scholar ]
• Ferraty F, Vieu P. Nonparametric functional data analysis. New York: Springer; 2006. [ Google Scholar ]
• Kneip A, Utikal KJ. Inference for density families using functional principal component analysis. J Am Stat Assoc. 2001; 94 :519–533. [ Google Scholar ]
• James GM, Hastie TJ, Sugar CA. Principal component models for sparse functional data. Biometrika. 2001; 87 :587–602. [ Google Scholar ]
• Wang H, Neill J, Miller F. Nonparametric clustering of functional data. Stat Interface. 2008; 1 :47–62. [ Google Scholar ]
• Dudoit S, Fridlyand J, Speed TP. Comparison of discrimination methods for the classification of tumors using gene expression data. J Am Stat Assoc. 2002; 97 :77–87. doi: 10.1198/016214502753479248. [ CrossRef ] [ Google Scholar ]
• Lee JW, Lee JB, Park M, Song SH. An extensive comparison of recent classification tools applied to microarray data. Comput Statist Data Anal. 2005; 48 :869–885. doi: 10.1016/j.csda.2004.03.017. [ CrossRef ] [ Google Scholar ]
• Denoeux T. A K-nearest neighbor classification rule based on Dempster-Shafer theory. IEEE Trans Syst Man Cybernet. 1995; 25 :804–813. doi: 10.1109/21.376493. [ CrossRef ] [ Google Scholar ]
• Keller JM, Gray MR, Givens JA. A fuzzy k-nearest neighbours algorithm. IEEE Trans. Syst. ManCybern. 1985; 15 :580–585. [ Google Scholar ]
• Cortes C, Vapnik V. Support-vector networks. Mach Learn. 1995; 20 :273–297. [ Google Scholar ]
• Fraley C, Raftery AE. Model-based clustering, discriminant analysis, and density estimation. J Am Stat Assoc. 2002; 97 :611–631. doi: 10.1198/016214502760047131. [ CrossRef ] [ Google Scholar ]
• Hyndman RJ, Koehler AB, Snyder RD, Grose S. A state space framework for automatic forecasting using exponential smoothing methods. Int J Forecast. 2002; 18 :439–454. doi: 10.1016/S0169-2070(01)00110-8. [ CrossRef ] [ Google Scholar ]
• Santen G, van Zwet E, Danhof M, Pasqua OD. Heterogeneity in patient response in depression: The relevance of functional data analysis. Universiteit Leiden, Netherland: PhD Thesis; 2008. [ Google Scholar ]
• Finch CF, Hayen A. Governmental health agencies need to assume leadership in injury prevention. Inj Prev. 2006; 12 :2–3. doi: 10.1136/ip.2005.010587. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Lee RD, Carter LR. Modeling and forecasting U.S. mortality. J Am Stat Assoc. 1992; 87 :659–675. [ Google Scholar ]
• Lee RD, Miller T. Evaluating the performance of the Lee-Carter method for forecasting mortality. Demography. 2001; 38 :537–549. doi: 10.1353/dem.2001.0036. [ PubMed ] [ CrossRef ] [ Google Scholar ]

2022 Theses Doctoral

Advances in Machine Learning for Complex Structured Functional Data

Tang, Chengliang

Functional data analysis (FDA) refers to a broad collection of statistical and machine learning methods that deal with the data in the form of random functions. In general, functional data are assumed to lie in a constrained functional space, e.g., images, and smooth curves, rather than the conventional Euclidean space, e.g., scalar vectors. The explosion of massive data and high-performance computational resources brings exciting opportunities as well as new challenges to this field. On one hand, the rich information from modern functional data enables an investigation into the underlying data patterns at an unprecedented scale and resolution. On the other hand, the inherent complex structures and huge data sizes of modern functional data pose additional practical challenges to model building, model training, and model interpretation under various circumstances. This dissertation discusses recent advances in machine learning for analyzing complex structured functional data. Chapter 1 begins with a general introduction to examples of modern functional data and related data analysis challenges. Chapter 2 introduces a novel machine learning framework, artificial perceptual learning (APL), to tackle the problem of weakly supervised learning in functional remote sensing data. Chapter 3 develops a flexible function-on-scalar regression framework, Wasserstein distributional learning (WDL), to address the challenge of modeling density functional outputs. Chapter 4 concludes the dissertation and discusses future directions.

• Machine learning--Statistical methods
• Statistical functionals

More About This Work

• DOI Copy DOI to clipboard

•   DigitalGeorgetown Home
• Georgetown University Institutional Repository
• Georgetown University Medical Center
• Biomedical Graduate Education
• Department of Biostatistics, Bioinformatics & Biomathematics
• Graduate Theses and Dissertations - Biostatistics, Bioinformatics & Biomathematics

Functional Data Analysis and its Application in Biomedical Research

Description

Permanent link, date published, collections, related items.

Showing items related by title, author, creator and subject.

Demo: Functional Analysis of Protein Data Part 3 

University of Minnesota

Digital conservancy.

•   University Digital Conservancy Home
• University of Minnesota Twin Cities
• Dissertations and Theses
• Dissertations

View/ Download file

Persistent link to this item, appears in collections, description, suggested citation, udc services.

• About the UDC
• How to Deposit
• Policies and Terms of Use

Related Services

• University Archives
• U of M Web Archive
• UMedia Archive
• Copyright Services
• Digital Library Services
• News & Events
• Staff Directory
• Subject Librarians
• Vision, Mission, & Goals

Novel methods for functional data analysis with applications to neuroimaging studies

• Original file (PDF) 3.3 MB
• Cover image (JPG) 17.2 KB
• Full text (TXT) 525.3 KB
• MODS (XML) 11 KB
• Dublin Core (XML) 5.3 KB
• NDLTD (XML) 5.6 KB
• Report accessibility issue

• Cookies & Privacy
• GETTING STARTED
• Introduction
• FUNDAMENTALS

Getting to the main article

Choosing your route

Setting research questions/ hypotheses

Assessment point

Building the theoretical case

Setting your research strategy

Data collection

Data analysis

Data analysis techniques

In STAGE NINE: Data analysis , we discuss the data you will have collected during STAGE EIGHT: Data collection . However, before you collect your data, having followed the research strategy you set out in this STAGE SIX , it is useful to think about the data analysis techniques you may apply to your data when it is collected.

The statistical tests that are appropriate for your dissertation will depend on (a) the research questions/hypotheses you have set, (b) the research design you are using, and (c) the nature of your data. You should already been clear about your research questions/hypotheses from STAGE THREE: Setting research questions and/or hypotheses , as well as knowing the goal of your research design from STEP TWO: Research design in this STAGE SIX: Setting your research strategy . These two pieces of information - your research questions/hypotheses and research design - will let you know, in principle , the statistical tests that may be appropriate to run on your data in order to answer your research questions.

We highlight the words in principle and may because the most appropriate statistical test to run on your data not only depend on your research questions/hypotheses and research design, but also the nature of your data . As you should have identified in STEP THREE: Research methods , and in the article, Types of variables , in the Fundamentals part of Lærd Dissertation, (a) not all data is the same, and (b) not all variables are measured in the same way (i.e., variables can be dichotomous, ordinal or continuous). In addition, not all data is normal , nor is the data when comparing groups necessarily equal , terms we explain in the Data Analysis section in the Fundamentals part of Lærd Dissertation. As a result, you might think that running a particular statistical test is correct at this point of setting your research strategy (e.g., a statistical test called a dependent t-test ), based on the research questions/hypotheses you have set, but when you collect your data (i.e., during STAGE EIGHT: Data collection ), the data may fail certain assumptions that are important to such a statistical test (i.e., normality and homogeneity of variance ). As a result, you have to run another statistical test (e.g., a Wilcoxon signed-rank test instead of a dependent t-test ).

At this stage in the dissertation process, it is important, or at the very least, useful to think about the data analysis techniques you may apply to your data when it is collected. We suggest that you do this for two reasons:

REASON A Supervisors sometimes expect you to know what statistical analysis you will perform at this stage of the dissertation process

This is not always the case, but if you have had to write a Dissertation Proposal or Ethics Proposal , there is sometimes an expectation that you explain the type of data analysis that you plan to carry out. An understanding of the data analysis that you will carry out on your data can also be an expected component of the Research Strategy chapter of your dissertation write-up (i.e., usually Chapter Three: Research Strategy ). Therefore, it is a good time to think about the data analysis process if you plan to start writing up this chapter at this stage.

REASON B It takes time to get your head around data analysis

When you come to analyse your data in STAGE NINE: Data analysis , you will need to think about (a) selecting the correct statistical tests to perform on your data, (b) running these tests on your data using a statistics package such as SPSS, and (c) learning how to interpret the output from such statistical tests so that you can answer your research questions or hypotheses. Whilst we show you how to do this for a wide range of scenarios in the in the Data Analysis section in the Fundamentals part of Lærd Dissertation, it can be a time consuming process. Unless you took an advanced statistics module/option as part of your degree (i.e., not just an introductory course to statistics, which are often taught in undergraduate and master?s degrees), it can take time to get your head around data analysis. Starting this process at this stage (i.e., STAGE SIX: Research strategy ), rather than waiting until you finish collecting your data (i.e., STAGE EIGHT: Data collection ) is a sensible approach.

Final thoughts...

Setting the research strategy for your dissertation required you to describe, explain and justify the research paradigm, quantitative research design, research method(s), sampling strategy, and approach towards research ethics and data analysis that you plan to follow, as well as determine how you will ensure the research quality of your findings so that you can effectively answer your research questions/hypotheses. However, from a practical perspective, just remember that the main goal of STAGE SIX: Research strategy is to have a clear research strategy that you can implement (i.e., operationalize ). After all, if you are unable to clearly follow your plan and carry out your research in the field, you will struggle to answer your research questions/hypotheses. Once you are sure that you have a clear plan, it is a good idea to take a step back, speak with your supervisor, and assess where you are before moving on to collect data. Therefore, when you are ready, proceed to STAGE SEVEN: Assessment point .

• ODSC EUROPE
• AI+ Training
• Speak at ODSC

• Data Analytics
• Data Engineering
• Data Visualization
• Deep Learning
• Generative AI
• Machine Learning
• NLP and LLMs
• Business & Use Cases
• Career Advice
• Write for us
• ODSC Community Slack Channel
• Upcoming Webinars

17 Compelling Machine Learning Ph.D. Dissertations

Machine Learning Modeling Research posted by Daniel Gutierrez, ODSC August 12, 2021 Daniel Gutierrez, ODSC

Working in the field of data science, I’m always seeking ways to keep current in the field and there are a number of important resources available for this purpose: new book titles, blog articles, conference sessions, Meetups, webinars/podcasts, not to mention the gems floating around in social media. But to dig even deeper, I routinely look at what’s coming out of the world’s research labs. And one great way to keep a pulse for what the research community is working on is to monitor the flow of new machine learning Ph.D. dissertations. Admittedly, many such theses are laser-focused and narrow, but from previous experience reading these documents, you can learn an awful lot about new ways to solve difficult problems over a vast range of problem domains. 

In this article, I present a number of hand-picked machine learning dissertations that I found compelling in terms of my own areas of interest and aligned with problems that I’m working on. I hope you’ll find a number of them that match your own interests. Each dissertation may be challenging to consume but the process will result in hours of satisfying summer reading. Enjoy!

Please check out my previous data science dissertation round-up article . 

1. Fitting Convex Sets to Data: Algorithms and Applications

This machine learning dissertation concerns the geometric problem of finding a convex set that best fits a given data set. The overarching question serves as an abstraction for data-analytical tasks arising in a range of scientific and engineering applications with a focus on two specific instances: (i) a key challenge that arises in solving inverse problems is ill-posedness due to a lack of measurements. A prominent family of methods for addressing such issues is based on augmenting optimization-based approaches with a convex penalty function so as to induce a desired structure in the solution. These functions are typically chosen using prior knowledge about the data. The thesis also studies the problem of learning convex penalty functions directly from data for settings in which we lack the domain expertise to choose a penalty function. The solution relies on suitably transforming the problem of learning a penalty function into a fitting task; and (ii) the problem of fitting tractably-described convex sets given the optimal value of linear functionals evaluated in different directions.

2. Structured Tensors and the Geometry of Data

This machine learning dissertation analyzes data to build a quantitative understanding of the world. Linear algebra is the foundation of algorithms, dating back one hundred years, for extracting structure from data. Modern technologies provide an abundance of multi-dimensional data, in which multiple variables or factors can be compared simultaneously. To organize and analyze such data sets we can use a tensor , the higher-order analogue of a matrix. However, many theoretical and practical challenges arise in extending linear algebra to the setting of tensors. The first part of the thesis studies and develops the algebraic theory of tensors. The second part of the thesis presents three algorithms for tensor data. The algorithms use algebraic and geometric structure to give guarantees of optimality.

3. Statistical approaches for spatial prediction and anomaly detection

This machine learning dissertation is primarily a description of three projects. It starts with a method for spatial prediction and parameter estimation for irregularly spaced, and non-Gaussian data. It is shown that by judiciously replacing the likelihood with an empirical likelihood in the Bayesian hierarchical model, approximate posterior distributions for the mean and covariance parameters can be obtained. Due to the complex nature of the hierarchical model, standard Markov chain Monte Carlo methods cannot be applied to sample from the posterior distributions. To overcome this issue, a generalized sequential Monte Carlo algorithm is used. Finally, this method is applied to iron concentrations in California. The second project focuses on anomaly detection for functional data; specifically for functional data where the observed functions may lie over different domains. By approximating each function as a low-rank sum of spline basis functions the coefficients will be compared for each basis across each function. The idea being, if two functions are similar then their respective coefficients should not be significantly different. This project concludes with an application of the proposed method to detect anomalous behavior of users of a supercomputer at NREL. The final project is an extension of the second project to two-dimensional data. This project aims to detect location and temporal anomalies from ground motion data from a fiber-optic cable using distributed acoustic sensing (DAS). 

4. Sampling for Streaming Data

Advances in data acquisition technology pose challenges in analyzing large volumes of streaming data. Sampling is a natural yet powerful tool for analyzing such data sets due to their competent estimation accuracy and low computational cost. Unfortunately, sampling methods and their statistical properties for streaming data, especially streaming time series data, are not well studied in the literature. Meanwhile, estimating the dependence structure of multidimensional streaming time-series data in real-time is challenging. With large volumes of streaming data, the problem becomes more difficult when the multidimensional data are collected asynchronously across distributed nodes, which motivates us to sample representative data points from streams. This machine learning dissertation proposes a series of leverage score-based sampling methods for streaming time series data. The simulation studies and real data analysis are conducted to validate the proposed methods. The theoretical analysis of the asymptotic behaviors of the least-squares estimator is developed based on the subsamples.

5.  Statistical Machine Learning Methods for Complex, Heterogeneous Data

This machine learning dissertation develops statistical machine learning methodology for three distinct tasks. Each method blends classical statistical approaches with machine learning methods to provide principled solutions to problems with complex, heterogeneous data sets. The first framework proposes two methods for high-dimensional shape-constrained regression and classification. These methods reshape pre-trained prediction rules to satisfy shape constraints like monotonicity and convexity. The second method provides a nonparametric approach to the econometric analysis of discrete choice. This method provides a scalable algorithm for estimating utility functions with random forests, and combines this with random effects to properly model preference heterogeneity. The final method draws inspiration from early work in statistical machine translation to construct embeddings for variable-length objects like mathematical equations

6. Topics in Multivariate Statistics with Dependent Data

This machine learning dissertation comprises four chapters. The first is an introduction to the topics of the dissertation and the remaining chapters contain the main results. Chapter 2 gives new results for consistency of maximum likelihood estimators with a focus on multivariate mixed models. The presented theory builds on the idea of using subsets of the full data to establish consistency of estimators based on the full data. The theory is applied to two multivariate mixed models for which it was unknown whether maximum likelihood estimators are consistent. In Chapter 3 an algorithm is proposed for maximum likelihood estimation of a covariance matrix when the corresponding correlation matrix can be written as the Kronecker product of two lower-dimensional correlation matrices. The proposed method is fully likelihood-based. Some desirable properties of separable correlation in comparison to separable covariance are also discussed. Chapter 4 is concerned with Bayesian vector auto-regressions (VARs). A collapsed Gibbs sampler is proposed for Bayesian VARs with predictors and the convergence properties of the algorithm are studied. 

7.  Model Selection and Estimation for High-dimensional Data Analysis

In the era of big data, uncovering useful information and hidden patterns in the data is prevalent in different fields. However, it is challenging to effectively select input variables in data and estimate their effects. The goal of this machine learning dissertation is to develop reproducible statistical approaches that provide mechanistic explanations of the phenomenon observed in big data analysis. The research contains two parts: variable selection and model estimation. The first part investigates how to measure and interpret the usefulness of an input variable using an approach called “variable importance learning” and builds tools (methodology and software) that can be widely applied. Two variable importance measures are proposed, a parametric measure SOIL and a non-parametric measure CVIL, using the idea of a model combining and cross-validation respectively. The SOIL method is theoretically shown to have the inclusion/exclusion property: When the model weights are properly around the true model, the SOIL importance can well separate the variables in the true model from the rest. The CVIL method possesses desirable theoretical properties and enhances the interpretability of many mysterious but effective machine learning methods. The second part focuses on how to estimate the effect of a useful input variable in the case where the interaction of two input variables exists. Investigated is the minimax rate of convergence for regression estimation in high-dimensional sparse linear models with two-way interactions, and construct an adaptive estimator that achieves the minimax rate of convergence regardless of the true heredity condition and the sparsity indices.

8.  High-Dimensional Structured Regression Using Convex Optimization

While the term “Big Data” can have multiple meanings, this dissertation considers the type of data in which the number of features can be much greater than the number of observations (also known as high-dimensional data). High-dimensional data is abundant in contemporary scientific research due to the rapid advances in new data-measurement technologies and computing power. Recent advances in statistics have witnessed great development in the field of high-dimensional data analysis. This machine learning dissertation proposes three methods that study three different components of a general framework of the high-dimensional structured regression problem. A general theme of the proposed methods is that they cast a certain structured regression as a convex optimization problem. In so doing, the theoretical properties of each method can be well studied, and efficient computation is facilitated. Each method is accompanied by a thorough theoretical analysis of its performance, and also by an R package containing its practical implementation. It is shown that the proposed methods perform favorably (both theoretically and practically) compared with pre-existing methods.

9. Asymptotics and Interpretability of Decision Trees and Decision Tree Ensembles

Decision trees and decision tree ensembles are widely used nonparametric statistical models. A decision tree is a binary tree that recursively segments the covariate space along the coordinate directions to create hyper rectangles as basic prediction units for fitting constant values within each of them. A decision tree ensemble combines multiple decision trees, either in parallel or in sequence, in order to increase model flexibility and accuracy, as well as to reduce prediction variance. Despite the fact that tree models have been extensively used in practice, results on their asymptotic behaviors are scarce. This machine learning dissertation presents analyses on tree asymptotics in the perspectives of tree terminal nodes, tree ensembles, and models incorporating tree ensembles respectively. The study introduces a few new tree-related learning frameworks which provides provable statistical guarantees and interpretations. A study on the Gini index used in the greedy tree building algorithm reveals its limiting distribution, leading to the development of a test of better splitting that helps to measure the uncertain optimality of a decision tree split. This test is combined with the concept of decision tree distillation, which implements a decision tree to mimic the behavior of a block box model, to generate stable interpretations by guaranteeing a unique distillation tree structure as long as there are sufficiently many random sample points. Also applied is mild modification and regularization to the standard tree boosting to create a new boosting framework named Boulevard. Also included is an integration of two new mechanisms: honest trees , which isolate the tree terminal values from the tree structure, and adaptive shrinkage , which scales the boosting history to create an equally weighted ensemble. This theoretical development provides the prerequisite for the practice of statistical inference with boosted trees. Lastly, the thesis investigates the feasibility of incorporating existing semi-parametric models with tree boosting. 

10. Bayesian Models for Imputing Missing Data and Editing Erroneous Responses in Surveys

This dissertation develops Bayesian methods for handling unit nonresponse, item nonresponse, and erroneous responses in large-scale surveys and censuses containing categorical data. The focus is on applications of nested household data where individuals are nested within households and certain combinations of the variables are not allowed, such as the U.S. Decennial Census, as well as surveys subject to both unit and item nonresponse, such as the Current Population Survey.

11. Localized Variable Selection with Random Forest  

Due to recent advances in computer technology, the cost of collecting and storing data has dropped drastically. This makes it feasible to collect large amounts of information for each data point. This increasing trend in feature dimensionality justifies the need for research on variable selection. Random forest (RF) has demonstrated the ability to select important variables and model complex data. However, simulations confirm that it fails in detecting less influential features in presence of variables with large impacts in some cases. This dissertation proposes two algorithms for localized variable selection: clustering-based feature selection (CBFS) and locally adjusted feature importance (LAFI). Both methods aim to find regions where the effects of weaker features can be isolated and measured. CBFS combines RF variable selection with a two-stage clustering method to detect variables where their effect can be detected only in certain regions. LAFI, on the other hand, uses a binary tree approach to split data into bins based on response variable rankings, and implements RF to find important variables in each bin. Larger LAFI is assigned to variables that get selected in more bins. Simulations and real data sets are used to evaluate these variable selection methods. 

12. Functional Principal Component Analysis and Sparse Functional Regression

The focus of this dissertation is on functional data which are sparsely and irregularly observed. Such data require special consideration, as classical functional data methods and theory were developed for densely observed data. As is the case in much of functional data analysis, the functional principal components (FPCs) play a key role in current sparse functional data methods via the Karhunen-Loéve expansion. Thus, after a review of relevant background material, this dissertation is divided roughly into two parts, the first focusing specifically on theoretical properties of FPCs, and the second on regression for sparsely observed functional data.

13. Essays In Causal Inference: Addressing Bias In Observational And Randomized Studies Through Analysis And Design

In observational studies, identifying assumptions may fail, often quietly and without notice, leading to biased causal estimates. Although less of a concern in randomized trials where treatment is assigned at random, bias may still enter the equation through other means. This dissertation has three parts, each developing new methods to address a particular pattern or source of bias in the setting being studied. The first part extends the conventional sensitivity analysis methods for observational studies to better address patterns of heterogeneous confounding in matched-pair designs. The second part develops a modified difference-in-difference design for comparative interrupted time-series studies. The method permits partial identification of causal effects when the parallel trends assumption is violated by an interaction between group and history. The method is applied to a study of the repeal of Missouri’s permit-to-purchase handgun law and its effect on firearm homicide rates. The final part presents a study design to identify vaccine efficacy in randomized control trials when there is no gold standard case definition. The approach augments a two-arm randomized trial with natural variation of a genetic trait to produce a factorial experiment. 

14. Bayesian Shrinkage: Computation, Methods, and Theory

Sparsity is a standard structural assumption that is made while modeling high-dimensional statistical parameters. This assumption essentially entails a lower-dimensional embedding of the high-dimensional parameter thus enabling sound statistical inference. Apart from this obvious statistical motivation, in many modern applications of statistics such as Genomics, Neuroscience, etc. parameters of interest are indeed of this nature. For over almost two decades, spike and slab type priors have been the Bayesian gold standard for modeling of sparsity. However, due to their computational bottlenecks, shrinkage priors have emerged as a powerful alternative. This family of priors can almost exclusively be represented as a scale mixture of Gaussian distribution and posterior Markov chain Monte Carlo (MCMC) updates of related parameters are then relatively easy to design. Although shrinkage priors were tipped as having computational scalability in high-dimensions, when the number of parameters is in thousands or more, they do come with their own computational challenges. Standard MCMC algorithms implementing shrinkage priors generally scale cubic in the dimension of the parameter making real-life application of these priors severely limited. 

The first chapter of this dissertation addresses this computational issue and proposes an alternative exact posterior sampling algorithm complexity of which that linearly in the ambient dimension. The algorithm developed in the first chapter is specifically designed for regression problems. The second chapter develops a Bayesian method based on shrinkage priors for high-dimensional multiple response regression. Chapter three chooses a specific member of the shrinkage family known as the horseshoe prior and studies its convergence rates in several high-dimensional models. 

15.  Topics in Measurement Error Analysis and High-Dimensional Binary Classification

This dissertation proposes novel methods to tackle two problems: the misspecified model with measurement error and high-dimensional binary classification, both have a crucial impact on applications in public health. The first problem exists in the epidemiology practice. Epidemiologists often categorize a continuous risk predictor since categorization is thought to be more robust and interpretable, even when the true risk model is not a categorical one. Thus, their goal is to fit the categorical model and interpret the categorical parameters. The second project considers the problem of high-dimensional classification between the two groups with unequal covariance matrices. Rather than estimating the full quadratic discriminant rule, it is proposed to perform simultaneous variable selection and linear dimension reduction on original data, with the subsequent application of quadratic discriminant analysis on the reduced space. Further, in order to support the proposed methodology, two R packages were developed, CCP and DAP, along with two vignettes as long-format illustrations for their usage.

16. Model-Based Penalized Regression

This dissertation contains three chapters that consider penalized regression from a model-based perspective, interpreting penalties as assumed prior distributions for unknown regression coefficients. The first chapter shows that treating a lasso penalty as a prior can facilitate the choice of tuning parameters when standard methods for choosing the tuning parameters are not available, and when it is necessary to choose multiple tuning parameters simultaneously. The second chapter considers a possible drawback of treating penalties as models, specifically possible misspecification. The third chapter introduces structured shrinkage priors for dependent regression coefficients which generalize popular independent shrinkage priors. These can be useful in various applied settings where many regression coefficients are not only expected to be nearly or exactly equal to zero, but also structured.

17. Topics on Least Squares Estimation

This dissertation revisits and makes progress on some old but challenging problems concerning least squares estimation, the work-horse of supervised machine learning. Two major problems are addressed: (i) least squares estimation with heavy-tailed errors, and (ii) least squares estimation in non-Donsker classes. For (i), this problem is studied both from a worst-case perspective, and a more refined envelope perspective. For (ii), two case studies are performed in the context of (a) estimation involving sets and (b) estimation of multivariate isotonic functions. Understanding these particular aspects of least squares estimation problems requires several new tools in the empirical process theory, including a sharp multiplier inequality controlling the size of the multiplier empirical process, and matching upper and lower bounds for empirical processes indexed by non-Donsker classes.

How to Learn More about Machine Learning

At our upcoming event this November 16th-18th in San Francisco,  ODSC West 2021  will feature a plethora of talks, workshops, and training sessions on machine learning and machine learning research. You can  register now for 50% off all ticket types  before the discount drops to 40% in a few weeks. Some  highlighted sessions on machine learning  include:

• Towards More Energy-Efficient Neural Networks? Use Your Brain!: Olaf de Leeuw | Data Scientist | Dataworkz
• Practical MLOps: Automation Journey: Evgenii Vinogradov, PhD | Head of DHW Development | YooMoney
• Applications of Modern Survival Modeling with Python: Brian Kent, PhD | Data Scientist | Founder The Crosstab Kite
• Using Change Detection Algorithms for Detecting Anomalous Behavior in Large Systems: Veena Mendiratta, PhD | Adjunct Faculty, Network Reliability and Analytics Researcher | Northwestern University

Sessions on MLOps:

• Tuning Hyperparameters with Reproducible Experiments: Milecia McGregor | Senior Software Engineer | Iterative
• MLOps… From Model to Production: Filipa Peleja, PhD | Lead Data Scientist | Levi Strauss & Co
• Operationalization of Models Developed and Deployed in Heterogeneous Platforms: Sourav Mazumder | Data Scientist, Thought Leader, AI & ML Operationalization Leader | IBM
• Develop and Deploy a Machine Learning Pipeline in 45 Minutes with Ploomber: Eduardo Blancas | Data Scientist | Fidelity Investments

Sessions on Deep Learning:

• GANs: Theory and Practice, Image Synthesis With GANs Using TensorFlow: Ajay Baranwal | Center Director | Center for Deep Learning in Electronic Manufacturing, Inc
• Machine Learning With Graphs: Going Beyond Tabular Data: Dr. Clair J. Sullivan | Data Science Advocate | Neo4j
• Deep Dive into Reinforcement Learning with PPO using TF-Agents & TensorFlow 2.0: Oliver Zeigermann | Software Developer | embarc Software Consulting GmbH
• Get Started with Time-Series Forecasting using the Google Cloud AI Platform: Karl Weinmeister | Developer Relations Engineering Manager | Google

Daniel Gutierrez, ODSC

Daniel D. Gutierrez is a practicing data scientist who’s been working with data long before the field came in vogue. As a technology journalist, he enjoys keeping a pulse on this fast-paced industry. Daniel is also an educator having taught data science, machine learning and R classes at the university level. He has authored four computer industry books on database and data science technology, including his most recent title, “Machine Learning and Data Science: An Introduction to Statistical Learning Methods with R.” Daniel holds a BS in Mathematics and Computer Science from UCLA.

ODSC’s AI Weekly Recap: Week of April 19th

AI and Data Science News posted by ODSC Team Apr 19, 2024 Every week, the ODSC team researches the latest advancements in AI. We review a selection of...

Meta AI Has Introduced Llama 3

AI and Data Science News posted by ODSC Team Apr 18, 2024 Meta AI has introduced Llama 3 to the world today through a new blog post by...

New Stanford University Report Shows Rapid Progress of AI and Expanding Costs

AI and Data Science News posted by ODSC Team Apr 18, 2024 A recent report from Stanford University’s Institute for Human-Centered Artificial Intelligence reveals that AI systems now...

File(s) under embargo

until file(s) become available

Deep Learning for Ordinary Differential Equations and Predictive Uncertainty

Deep neural networks (DNNs) have demonstrated outstanding performance in numerous tasks such as image recognition and natural language processing. However, in dynamic systems modeling, the tasks of estimating and uncovering the potentially nonlinear structure of systems represented by ordinary differential equations (ODEs) pose a significant challenge. In this dissertation, we employ DNNs to enable precise and efficient parameter estimation of dynamic systems. In addition, we introduce a highly flexible neural ODE model to capture both nonlinear and sparse dependent relations among multiple functional processes. Nonetheless, DNNs are susceptible to overfitting and often struggle to accurately assess predictive uncertainty despite their widespread success across various AI domains. The challenge of defining meaningful priors for DNN weights and characterizing predictive uncertainty persists. In this dissertation, we present a novel neural adaptive empirical Bayes framework with a new class of prior distributions to address weight uncertainty.

In the first part, we propose a precise and efficient approach utilizing DNNs for estimation and inference of ODEs given noisy data. The DNNs are employed directly as a nonparametric proxy for the true solution of the ODEs, eliminating the need for numerical integration and resulting in significant computational time savings. We develop a gradient descent algorithm to estimate both the DNNs solution and the parameters of the ODEs by optimizing a fidelity-penalized likelihood loss function. This ensures that the derivatives of the DNNs estimator conform to the system of ODEs. Our method is particularly effective in scenarios where only a set of variables transformed from the system components by a given function are observed. We establish the convergence rate of the DNNs estimator and demonstrate that the derivatives of the DNNs solution asymptotically satisfy the ODEs determined by the inferred parameters. Simulations and real data analysis of COVID-19 daily cases are conducted to show the superior performance of our method in terms of accuracy of parameter estimates and system recovery, and computational speed.

In the second part, we present a novel sparse neural ODE model to characterize flexible relations among multiple functional processes. This model represents the latent states of the functions using a set of ODEs and models the dynamic changes of these states utilizing a DNN with a specially designed architecture and sparsity-inducing regularization. Our new model is able to capture both nonlinear and sparse dependent relations among multivariate functions. We develop an efficient optimization algorithm to estimate the unknown weights for the DNN under the sparsity constraint. Furthermore, we establish both algorithmic convergence and selection consistency, providing theoretical guarantees for the proposed method. We illustrate the efficacy of the method through simulation studies and a gene regulatory network example.

In the third part, we introduce a class of implicit generative priors to facilitate Bayesian modeling and inference. These priors are derived through a nonlinear transformation of a known low-dimensional distribution, allowing us to handle complex data distributions and capture the underlying manifold structure effectively. Our framework combines variational inference with a gradient ascent algorithm, which serves to select the hyperparameters and approximate the posterior distribution. Theoretical justification is established through both the posterior and classification consistency. We demonstrate the practical applications of our framework through extensive simulation examples and real-world datasets. Our experimental results highlight the superiority of our proposed framework over existing methods, such as sparse variational Bayesian and generative models, in terms of prediction accuracy and uncertainty quantification.

Degree Type

• Doctor of Philosophy

Campus location

• West Lafayette

Advisor/Supervisor/Committee Chair

Additional committee member 2, additional committee member 3, additional committee member 4, usage metrics.

• Statistics not elsewhere classified
• Deep learning