Logistic Regression

Regression Analysis

Statistical Foundations of Binary Classification

Logistic regression models the probability of a binary outcome using the log-odds link function. Maximum likelihood estimation, Wald tests, and likelihood ratio tests provide rigorous statistical inference for classification problems.

Medical Diagnosis — Predict disease presence from patient characteristics
Credit Scoring — Estimate default probability for loan applications
Customer Analytics — Model churn likelihood from behavioral features

The sigmoid function maps any linear combination to a valid probability.

Why Not Linear Regression for Binary Data?

Linear regression is inappropriate for binary responses $Y \in \{0, 1\}$ because:

The errors $\varepsilon_i = Y_i - (\beta_0 + \beta_1 X_i)$ are not normally distributed (they are Bernoulli).
The predicted values $\hat{Y}$ can fall outside $[0, 1]$ , which is impossible for probabilities.
The variance is not constant: $\text{Var}(Y \mid X) = p(X)(1-p(X))$ , which depends on $X$ .

Logistic regression solves these problems by modeling the probability through the logistic (sigmoid) function.

The Logistic Model

DfLogistic Regression Model

Logistic regression models the probability of success as a function of predictors via the logit link function:

P(Y = 1 \mid \mathbf{X}) = \frac{\exp(\beta_0 + \beta_1 X_1 + \cdots + \beta_p X_p)}{1 + \exp(\beta_0 + \beta_1 X_1 + \cdots + \beta_p X_p)}

Equivalently, in terms of the log-odds (logit):

\log\left(\frac{p}{1-p}\right) = \beta_0 + \beta_1 X_1 + \cdots + \beta_p X_p

where $p = P(Y = 1 \mid \mathbf{X})$ .

Logistic (Sigmoid) Function

p(x) = \frac{e^{\beta_0 + \beta_1 x}}{1 + e^{\beta_0 + \beta_1 x}} = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x)}}

Here,

$p(x)$ =Probability of success given x
$\beta_0$ =Intercept (log-odds when x = 0)
$\beta_1$ =Coefficient for x

Odds and Odds Ratios

DfOdds

The odds of an event with probability $p$ is:

\text{Odds} = \frac{p}{1 - p}

Odds ranges from 0 to $\infty$ . When $p = 0.5$ , odds = 1. When $p = 0.75$ , odds = 3 (3 to 1).

ThInterpretation of Coefficients via Odds Ratios

Exponentiating a coefficient gives the odds ratio (OR):

\text{OR} = e^{\beta_j} = \frac{\text{odds at } X_j + 1}{\text{odds at } X_j}

$\beta_j > 0 \Rightarrow \text{OR} > 1$ : increasing $X_j$ increases the odds of success
$\beta_j < 0 \Rightarrow \text{OR} < 1$ : increasing $X_j$ decreases the odds of success
$\beta_j = 0 \Rightarrow \text{OR} = 1$ : $X_j$ has no effect on the odds

Goodness of Fit

| Metric | Definition | Interpretation |

|--------|-----------|----------------|

| McFadden's $R^2$ | $1 - \ell(\hat{\beta})/\ell(\hat{\beta}_0)$ | 0.2–0.4 is considered good |

| AIC | $-2\ell(\hat{\beta}) + 2p$ | Lower is better; penalizes complexity |

| BIC | $-2\ell(\hat{\beta}) + p \ln(n)$ | Lower is better; stronger penalty than AIC |

| Hosmer–Lemeshow | Compares observed vs. predicted frequencies in deciles | Non-significant $p$ indicates good fit |

Key Takeaways

Summary: Logistic Regression

Logistic regression outputs probabilities via the sigmoid function — never below 0 or above 1
Coefficients are on the log-odds scale — exponentiate to get odds ratios
MLE, not OLS, is used to fit logistic regression — there is no closed-form solution
The Likelihood Ratio Test is more powerful than the Wald test for hypothesis testing
AUC–ROC is a better performance metric than accuracy for imbalanced classes
The logit link ensures the linear predictor maps to a valid probability
Odds ratios provide intuitive multiplicative interpretation of effects

Logistic Regression — Binary Classification with Statistics