Parameter Space and Data Space

All Models

All models are of the form $f(\underline x, \underline \theta)$. $f$ defines the functional form of the model, $\pmb x$ is a list of $p$ characteristics, and $\pmb \theta$ is a list of $k$ parameters that are needed to define the model.

For example we could have a model $f_1(\underline x_1, \underline \theta_1) = \theta_1 x_1 + \dots + \theta_p x_p$ (in this case $p=k$), or we could have a model like this $f_2(\underline x_2, \underline \theta_2) = \theta_0 + \theta_1 x_1 + \dots + \theta_p x_p$, in this case $p + 1 = k$.

Both $f_1$ and $f_2$ fit under the class of linear models but linear models need not be linear. We could have a model $f_3(\underline x_3, \underline \theta_3) = \frac{\theta_0}{\theta_0^2 + \theta_1 x_1}$. This model has $p=1$ and $k=2$ and it’s a nonlinear model.

Therefore, we say that all models can be written in the form $f(\underline x, \underline \theta)$.

Choose a Model

The first problem we need to considre when modeling is choosing $f$. That is to say, deciding what functional form the data generating process is taking. There are many choices for $f$. Every neural network architecture is a different $f$, various linear models can have $k=p$ yet use different characteristics, some models like decision trees are non-parametric and produce discontinuous outputs; in short, there are many choices for $f$ and different $f$‘s can be useful for different types of a data analysis. So the first step, is to decide which $f$ to consider.

To reiterate, choosing $f$ doesn’t simply mean picking some class of models, it includes deciding which $x_i$‘s to consider and what the relationship is among the $\theta_i$‘s. This step should be taken seriously when considering interpretability of the model. In fact, it can’t be over emphasized.

Fit / Test the Model

Warning on statements of inference

Data Space

Parameter Space

Statistical Significance