McNulty // w5d4

Winter 2015

Planned schedule and activities

9:00 am: Bonjour

9:15 am: Poisson Distribution

9:45 am: Short break

9:50 am: Poisson GLM and Model Evaluation in GLMs

10:30 am: Short break

10:45 am: Stochastic Gradient Descent

11:15 am: GLM Challenges

12:00pm: Bon App├ętit

1:30pm: GLM Challenges & Work on McNulty

5:00pm: OPTIONAL Speaker -- Aaron Schumacher

5:30pm: Au revoir

Lecture Notes

w5d4_Stochastic_Gradient_Descent.pdf (1.3 MB) ### Poisson GLM Challenges

Data for the Challenges

We are going to fit a model to count data. The first one we will use is about the total count of damage incidents to ships. You can get it here:

We are stealing this dataset from STATA, so it's in STATA format. Pandas has a reader for that format to make your life easy:

from import StataReader
reader = StataReader('ships.dta')
dataframe =

Here are some details on this dataset:

The file has 34 rows corresponding to the observed combinations of type of ship, year of construction and period of operation. Each row has information on five variables as follows:

Note that there no ships of type E built in 1960-64, and that ships built in 1970-74 could not have operated in 1960-74. These combinations are omitted from the data file.

Challenge 1

Model the damage incident counts with a Poisson Regression.
> Hint: You can look at the previous ipython notebook with the logistic GLM example to see how you can do GLM with statsmodels

Remember that you will have to create dummy variables for categorical variables, and if you have time bins (like "1960-1964"), you have the option of either a) treating each bin as a category (and create dummies for each bin), or b) treat it as a continuous variable and take the mid-value (1962). Also remember to add a constant (to model the intercept).

Take a look at the statsmodels summary table, the goodness of fit indicators (Deviance, Pearson's chi square statistic) and the coefficients. Is this a good model?

Challenge 2

The months of service provides the time interval in which a ship has chances to acquire damages. It can be thought of "exposure", and this column can be used as an offset. You can check these two resources for a quick idea on exposure and using an offset:
Offset in Wikipedia
When to use an offset in CrossValidated (StackOverflow for statistics)

Try your model with months of service as the offset. Does it perform better?

Challenge 3

Now separate your data (even though it's only 14 rows) into a training and test set (your test will only be 4 or 5 rows), and check if you predict well (you can look at mean absolute error or mean squared error using sklearn.metrics).

Challenge 4

Deviance. Compute the difference in Deviance statistics for your model and the null model. This is called the null deviance. You can do this in one of 2 ways:

  1. We need the deviance for the null model (a model where none of the explanatory variables are used; it's just a model with a mean guess). To do that, fit a poisson regression with only a constant. Get the deviance for this null model. Take the difference of deviances between your model and this null model.
  2. Use statsmodels.genmod.generalized_linear_model.GLMResults

Check if this difference is extreme enough that we can reject the null hypothesis. If we can't reject the null hypothesis, we cannot say that this model tells us more than that trivial, null model. To calculate the p-value (prob. of getting a deviance difference at least as extreme as this under the null hypothesis), we need to do a hypothesis test.

You can import the chi square test from scipy for this:

from scipy.stats import chisqprob

Is your model better than the null model?

Challenge 5

Now, instead of a poisson regression, do an ordinary least squares regression with log Y. Compare the models. Are the coefficients close? Do they perform similarly?

Challenge 6

Now, let's do this on another dataset: Smoking and Cancer.

You can get it here: This dataset has information on lung cancer deaths by age and smoking status. It has these columns:

That population looks a lot like an offset!

Fit poisson and OLS models and compare them.

Bonus: try all these things in R! Your numerical answers should all be the same.


Poisson Distribution in Wikipedia
Poisson Distribution Examples
Poisson Distribution Math

Poisson Regression
Comparison of Least Squares and Poisson Regression Math

GLM in statsmodels
Deviance (Evaluation metrics)

Stochastic Gradient Descent in Wikipedia
Stochastic Gradient Descent Math and Tricks for Success
Stochastic Gradient Descent in scikit.learn