Interpreting Regression
1
Preamble
1.1
Caution
1.2
Purpose of the book
1.3
A focus on Interpretation
An outcome on its own
2
Probability: When an Outcome is Unknown
2.1
Probability Distributions
2.1.1
Probability
2.1.2
Probability Distributions
2.1.3
Examples of Probability Distributions
2.2
Continuous random variables (10 min)
2.3
Density Functions (20 min)
2.3.1
Example: “Low Purity Octane”
2.3.2
Example: Monthly Expenses
2.4
Summary and take-aways
3
Distribution Properties: Quantities we can Interpret
3.1
Probabilistic Quantities
3.2
Measures of central tendency and uncertainty
3.2.1
Mode and Entropy
3.2.2
Mean and Variance
3.3
What is the mean, anyway?
3.4
Quantiles
3.5
Continuous Distribution Properties
3.5.1
Mean, Variance, Mode, and Entropy (5 min)
3.5.2
Median (5 min)
3.5.3
Quantiles (5 min)
3.5.4
Prediction Intervals (5 min)
3.5.5
Skewness (5 min)
3.5.6
Examples
3.6
Heavy-Tailed Distributions
3.6.1
Sensitivity of the mean to extremes
3.6.2
Heavy-tailed Distributions
3.6.3
Heavy-tailed distribution families
3.6.4
Extreme Value Analysis
3.6.5
Multivariate Student’s
t
distributions
4
Explaining an uncertain outcome: interpretable quantities
4.0.1
Cumulative Density Functions (cdf’s) / Distribution Functions
4.0.2
Survival Function (2 min)
4.0.3
Quantile Function (5 min)
4.0.4
Other ways of depicting a distribution (Optional) (1 min)
5
Simulation: When calculations are difficult
5.0.1
Learning Objectives
5.0.2
Review Activity (15 min)
5.0.3
Random Samples: Terminology (5 min)
5.0.4
Seeds (5 min)
5.0.5
Generating Random Samples: Code
5.0.6
Running Simulations
5.0.7
Multi-Step Simulations (10 min)
5.0.8
Generating Continuous Data
6
Sampling Distributions: How Good are the Numbers?
6.1
Estimation of Probabilistic Quantities
7
Parametric Families of Distributions
7.1
Concepts
7.1.1
Binomial Distribution
7.1.2
Families vs. distributions
7.1.3
Parameters
7.1.4
Parameterization
7.1.5
Distribution Families in Practice
7.2
Common Parametric Families
7.2.1
Geometric
7.2.2
Negative Binomial
7.2.3
Poisson
7.2.4
Bernoulli
7.2.5
Uniform
(3 min)
7.2.6
Gaussian / Normal
(4 min)
7.2.7
Log-Normal
Family
7.2.8
Exponential
Family
7.2.9
Weibull
Family
7.2.10
Beta
Family
7.2.11
Activity
7.3
Relevant R functions (8 min)
7.4
Analyses under a Distributional Assumption
7.4.1
Maximum Likelihood Estimation
7.4.2
Usefulness in Practice
Prediction: harnessing the signal
8
Reducing uncertainty of the outcome: including predictors
8.1
Variable terminology
8.1.1
Variable types
8.2
Irreducible Error
8.3
In-class Exercises: Irreducible Error
8.3.1
Oracle regression
8.3.2
Oracle classification
8.3.3
(BONUS) Random prediction
8.3.4
(BONUS) A more non-standard regression
8.3.5
(BONUS) Oracle MSE
8.4
Learning Objectives
8.5
Conditional Distributions (15 min)
8.6
Joint Distributions (25 min)
8.6.1
Example: Length of Stay vs. Gang Demand
8.6.2
Marginal Distributions
8.6.3
Calculating Marginals from the Joint
8.6.4
Conditioning on one Variable
8.6.5
Law of Total Probability/Expectation
8.6.6
Exercises (10 min)
8.7
Multivariate Densities/pdf’s
8.7.1
Conditional Distributions, revisited (15 min)
8.8
Dependence concepts
8.8.1
Independence (5 min)
8.8.2
Measures of dependence (15 min)
8.8.3
Dependence as separate from the marginals
8.8.4
Dependence as giving us more information
8.9
Harvesting Dependence (20 min)
8.9.1
Example: River Flow
8.9.2
Direction of Dependence
8.10
Marginal Distributions (20 min)
8.10.1
Marginal Distribution from Conditional
8.10.2
Marginal Mean from Conditional
8.10.3
Marginal Quantiles from Conditional
8.10.4
Activity
8.11
Comparing Probabilities
9
The signal: model functions
9.1
Linear Quantile Regression
9.1.1
Exercise
9.1.2
Problem: Crossing quantiles
9.1.3
Problem: Upper quantiles
9.2
Concepts
10
The Model-Fitting Paradigm in R
10.1
broom
package
11
Estimating parametric model functions
11.1
Writing the sample mean as an optimization problem
11.2
Evaluating Model Goodness: Quantiles
11.3
Simple Linear Regression
11.3.1
Model Specification
11.4
Linear models in general
11.5
reference-treatment parameterization
11.5.1
More than one category (Lab 2)
11.6
Concepts
12
Estimating assumption-free: the world of supervised learning techniques
12.1
What machine learning is
12.2
Types of Supervised Learning
12.3
Local Regression
12.3.1
kNN
12.3.2
loess
12.3.3
In-Class Exercises
12.3.4
Hyperparameters and the bias/variance tradeoff
12.3.5
Extensions to kNN and loess
12.3.6
Model assumptions and the bias/variance tradeoff
12.4
Splines and Loess Regression
12.4.1
Loess
13
Overfitting: The problem with adding too many parameters
13.1
Classification Exercise: Do Together
13.2
Training Error vs. Generalization Error
13.3
Model complexity
13.3.1
Activity
13.4
Reducible Error
13.4.1
What is it?
13.4.2
Example
13.4.3
Bias and Variance
13.4.4
Reducing reducible error
13.4.5
Error decomposition
13.5
Model Selection
13.5.1
Exercise: CV
13.5.2
Out-of-sample Error
13.5.3
Alternative measures of model goodness
13.5.4
Feature and model selection: setup
13.5.5
Model selection
13.5.6
Feature (predictor) selection
Describing Relationships
14
There’s meaning in parameters
14.1
The types of parametric assumptions
14.1.1
1. When defining a
model function
.
14.1.2
2. When defining
probability distributions
.
14.2
The value of making parametric assumptions
14.2.1
Value #1: Reduced Error
14.2.2
Value #2: Interpretation
14.3
ANOVA
15
The meaning of interaction
16
Scales and the restricted range problem
16.1
Problems
16.2
Solutions
16.2.1
Solution 1: Transformations
16.2.2
Solution 2: Link Functions
16.2.3
Solution 3: Scientifically-backed functions
16.3
GLM’s in R
16.3.1
broom::augment()
16.4
Options for Logistic Regression
16.4.1
Models
17
Improving estimation through distributional assumptions
18
When we only want interpretation on some predictors
18.1
Non-identifiability in GAMS
18.1.1
Non-identifiability
18.1.2
Question 1b
Special cases
19
Regression when data are censored: survival analysis
19.1
Data
19.2
Univariate Estimation
19.2.1
Non-parametric Estimates with Kaplan-Meier
19.2.2
Parametric Estimation
19.3
Regression with Survival Data
19.3.1
Proportional Hazards Model
19.3.2
Prediction
19.4
Concept list
20
Regression in the presence of outliers: robust regression
20.1
Robust Regression in R
20.1.1
Heavy Tailed Regression
21
Regression in the presence of extremes: extreme value regression
22
Regression when data are ordinal
22.1
Concept list
23
Regression when data are missing: multiple imputation
23.1
Mean Imputation
23.2
Multiple Imputation
23.2.1
Patterns
23.2.2
Multiple Imputation
23.2.3
Pooling
23.3
Step 0: What data are missing?
23.4
Step 1: Handling Missing Data
23.4.1
Any Ideas?
23.4.2
mice
23.5
Step 3: Pool results
23.6
Concepts
24
Regression under many groups: mixed effects models
24.1
Motivation for LME
24.1.1
Definition
24.1.2
R Tools for Fitting
24.2
Mixed Effects Models in R: tutorial
24.3
Concepts
25
Regression on an entire distribution: Probabilistic Forecasting
25.1
Probabilistic Forecasting: What it is
25.2
Review: Univariate distribution estimates
25.2.1
Continuous response
25.2.2
Discrete Response
25.3
Probabilistic Forecasts: subset-based learning methods
25.3.1
The techniques
25.3.2
Exercise
25.3.3
Bias-variance tradeoff
25.3.4
Evaluating Model Goodness
25.4
Discussion Points
25.5
When are they not useful?
26
Regression when order matters: time series and spatial analysis
26.1
Timeseries in (base) R
26.2
Spatial Example
26.3
A Model for River Rock Size
26.3.1
1. Average rock size:
26.3.2
2. Mean rock size:
26.3.3
3. Downstream fining curve:
26.4
Statistical Objectives
26.4.1
Preliminaries: Variance and Correlation
26.5
Three Concepts
26.5.1
Error Variance
\(\sigma_{E}^{2}\left(x\right)\)
26.5.2
Mean Variance
\(\sigma_{M}^{2}\)
26.5.3
Mean Correlation
\(\rho\left(d\right)\)
26.6
Estimation
26.6.1
Constant Error Variance
26.6.2
Non-Constant Error Variance
26.7
Statistical Objective 1: Downstream Fining Curve
26.7.1
Regression Form
26.8
Statistical Objective 2: River Profile
26.8.1
Simple Kriging
26.8.2
Universal Kriging
26.8.3
Kriging under Non-Constant Error Variance
26.9
Confidence Intervals of the River Profile
Published with bookdown
Interpreting Regression
Describing Relationships