Data Analytics - EDA

1 Pipeline

Data Engineering (upstream) -> Data Analytics (downstream)

where the engineering part is where most work happens on, including data ingestion/storing (e.g. DBMS, distributed file system); and the analysis part is where EDA is in. High quality data acquisition is hard.

Generally, Data Analytics = EDA + CDA + etc., where EDA = Data Exploration + Data Analysis(part of).

These concepts might overlap!

We say, Data Science = Data Engineering + Data Analytics; AI > Machine Learning > Deep Learning. The overlap between AI and Data Science is (the intersection of ML and DA).

Roger Peng’s evidence based data analysis from plots: we can see(data itself), we can read(read labels), we can compare(analyze relations).

Notes of p-hacking (pre-registration).

2 EDA Workflow

Use modern tidyverse. A common workflow is split-apply-combine: Group the samples/rows (by classes), add new columns (means, sd, etc.), then form a new table.

(Most important: Outliers/Errors/Missing Values.)

General

• Summary/Scan: find global outlier

Detailed

• Between-Group: ANOVA

• Within-Group: find in-group outlier

The restructure (e.g. merge 0-1 hot to categorial variables, impute n/a's, log + standardize) is seen as preprocessing. Though generally we can all see them as EDA.

Adding new features (e.g. means/sd) is seen as feature engineering.

Exploratory Factor Analysis/EFA: A workflow to capture latent common variables. Often apply PCA.

3 Tidy The Data

After accessing the data (e.g. API's, XML, etc.), tidy and clean data formats (e.g. delete comma etc.).

This is part of EDA (capture the problem) and preprocessing (handle the problem).

Data formats: cube data(*multiple dimensions selecting a single value), long-format data(like csv, a case per row). Long data isn't good for sparse data (and not flexible for adding features/insert rows etc).

Grammar of Graphics allow us to create all graphs using same grammar (ggplot).

4 Regex and OCR

(pronounced redjex) note there are \ \ and \ differences.

[], (), +, *, $, ?

OCR is a simple neural net. Pre-process: Greyscale, line detection, word detection, deskewing, size adjustments / Post-process: spell checking, grammar checking

5 Imputation

Generally: Structural missing (no need and can't impute) / MCAR, MAR, MNAR (should impute).

We mainly handle three types of missing: MCAR (missing completely at random)/MAR (missing at random)/MNAR (missing not at random). We assmue MAR at most times, unless we have evidence to use MNAR. Though sometimes we count MCAR is a MAR.

For the MAR imputation part, core is MICE + Rubin's Rule. We oftenly have 3 steps: impute (add randomness to predictions from known values), modeling (apply exact same model), pooling (estimates take the average, or law of total variance/rubin's rule).

6 LM and Random Forest

Adding multiplication terms of features is also considered LM methods.

Quantile Regression. (use asymmetric absolute error, \(\tau=0.5\) is median regression)

Bagging is a word from bootstraping + aggregating. For random forests, every split choose random features from all features.

• For random forests, without using pruning trees (very small minimum node size/use a very small complexity parameter), too many variables/features per tree, too many samples/observations per tree, will cause overfitting. We don't want trees to be similar.

Regression Tree oftenly uses MSE. Continuous -> take mean; discrete -> take majority vote.

7 kNN and LASSO

kNN global trends(high k/smoothness)/local trends(low k/fit to data) balanced.

Ripley's K function: calcs how many points are inside a circle (take mean of every point). For unif, should be propto \(\pi r^2\).

GLM and LASSO plots - \(\log(\lambda)\) represents log of penalty term. The top numbers are the number of parameters used in that penalty level. (We look at the remaining features to decide if it's a LASSO or Ridge.)

glmnet = glm + net (penalize, 1 is lasso, 0 is ridge).

• KDE (Kernel) is a kind of mixing normal distributions (bandsize). Every sample point relates to a kernel dist(normal). (generalized histogram)

• LOESS (Local Estimate) is a kind of smoothing but only neighbors (span).

8 Model Selection

CV - kfold. On average, it has lower error when k grows (but still higher than taking whole as training set). Prevent overfitting.

AIC/BIC/adjusted R2: balance model fit (high lik) vs. parsimony/simpleness/resistance to overfitting/explainability.

p-value small -> statistically significant.

Akeike -> \(-2\ell+2k\)

Bayesian -> adds sample size \(-2\ell+k\ln n\).

Stepwise regression(better)/best subsets regression(too slow) uses some unimputed missing data, so we can't use AIC to select models (different datasets/observations!).

When we do clustering, we focus on similarity within groups vs. number of groups. How do we check clustering - use human assistance.

For hierarhical clustering, we ideally choose clusters only significantly differs from others. (the height decides the distance)

Clustering methods are always bad (but can) for high dimensions. Note the difference from graphs between clustering and trend.

Note in R, factor can be used for unordered variables.

MCMC -> CI's. (Stationary/Detailed Balance)

Minimum Effects Test: more strict than Hypothesis tests (fall into an interval, not just null). If doesn't contain 0, it has a meaning but not significant.

9 Dimension Reduction

• PCA: use SVD as basic. Singular values are a type of generalized eigenvalue. We only use empirical 90% var. Note the remaining variance is defined by the uninterprated part. Multicollineary wouldn't be a problem (orthogonal).

• t-SNE: Use a high-dim Normal dist. and low-dim t-dist. with KL.

10 Psychology

• Cronbach's Alpha measure: how much a set of questions are measuring the same thing/how consistent is the questionaire.

• Likert's Scale: oftenly used in questionaires, mapping a response to a value/how to give a criteria.

• ICC (item characteristic curve): x-axis -> ability, y-axis -> prob. An ICC is only for one ques. Guess is left, mean is middle. (after Item Response Theory/IRT estimate all the parameters)/how hard is one question.

---

Q1: What are the three things you need to run an optimization?

Q2: What are some examples of methods we have used in Stat 847 that use optimization, and what is being minimized or maximized?

Q3: Linear regression fits the conditional mean, what is (linear) quantile regression fitting?

Q4: What are some of the preprocessing steps that OCR uses to prepare images for conversion to text?

Q5: What is the fundamental difference between cube data and the more standard long-format data?

Q6: What format does API output commonly arrive in?

Q7: What is the main purpose of using cross-validation to create your model?

Q8: What two things are criteria like AIC, BIC, and adjusted R-squared balancing?

Q9: What does Cronbach's Alpha measure?

Q10: What are two steps that MICE uses to reflect the additional uncertainty in imputed responses compared to measured ones?

---

Answers

Q1: What are the three things you need to run an optimization?
A: Loss function, parameter set, initial values

Q2: What are some examples of methods we have used in Stat 847 that use optimization, and what is being minimized or maximized?
A: Linear regression, the sum (or mean) square of the residuals.
A: LASSO, the sum square of the residuals, plus a penalty term.
A: Stepwise regression, the AIC, BIC, or adjusted R-squared.

Q3: Linear regression fits the conditional mean, what is (linear) quantile regression fitting?
A: The median (or other quantile) of the response variable, conditional on the explanatory variables.

Q4: What are some of the preprocessing steps that OCR uses to prepare images for conversion to text?
A: deskewing, greyscaling, line detection.

Q5: What is the fundamental difference between cube data and the more standard long-format data?
A: Cube data contains one value per row, long format contains one case per row with many values.

Q6: What format does API output commonly arrive in?
A: JSON

Q7: What is the main purpose of using cross-validation to create your model?
A: To prevent overfitting specific to the data that you have right now.

Q8: What two things are criteria like AIC, BIC, and adjusted R-squared balancing?
A: Model fit and model complexity.

Q9: What does Cronbach's Alpha measure?
A: How much a set of questions are measuring the same thing.

Q10: What are two steps that MICE uses to reflect the additional uncertainty in imputed responses compared to measured ones?
A: Adds random noise to its predictions, and makes multiple copies of each imputed value.