Loading and validation
Read the variables I selected from the GSS dataset. You can look up these variables at https://gssdataexplorer.norc.org/variables/vfilter
Most variables use special codes to indicate missing data. We have to be careful not to use these codes as numerical data; one way to manage that is to replace them with NaN
, which Pandas recognizes as a missing value.
Here are summary statistics for the variables I have validated and cleaned.
Exercise
Look through the column headings to find a few variables that look interesting. Look them up on the GSS data explorer.
Use
value_counts
to see what values appear in the dataset, and compare the results with the counts in the code book.Identify special values that indicate missing data and replace them with
NaN
.Use
describe
to compute summary statistics. What do you notice?
Visualize distributions
Let's visualize the distributions of the variables we've selected.
Here's a Hist of the values in educ
:
Hist
as defined in thinkstats2
is different from hist
as defined in Matplotlib. The difference is that Hist
keeps all unique values and does not put them in bins. Also, hist
does not handle NaN
.
One of the hazards of using hist
is that the shape of the result depends on the bin size.
Exercise:
Run the following cell and compare the result to the
Hist
above.Add the keyword argument
bins=11
toplt.hist
and see how it changes the results.Experiment with other numbers of bins.
However, a drawback of Hist
and Pmf
is that they basically don't work when the number of unique values is large, as in this example:
Exercise:
Make and plot a
Hist
ofage
.Make and plot a
Pmf
ofeduc
.What fraction of people have 12, 14, and 16 years of education?
Exercise:
Make and plot a
Cdf
ofeduc
.What fraction of people have more than 12 years of education?
Exercise:
Make and plot a
Cdf
ofage
.What is the median age? What is the inter-quartile range (IQR)?
Exercise:
Find another numerical variable, plot a histogram, PMF, and CDF, and compute any statistics of interest.
Exercise:
Compute the CDF of
realinc
for male and female respondents, and plot both CDFs on the same axes.What is the difference in median income between the two groups?
Exercise:
Use a variable to break the dataset into groups and plot multiple CDFs to compare distribution of something within groups.
Note: Try to find something interesting, but be cautious about overinterpreting the results. Between any two groups, there are often many differences, with many possible causes.
Save the cleaned data
Now that we have the data in good shape, we'll save it in a binary format (HDF5), which will make it faster to load later.
Also, we have to do some resampling to make the results representative. We'll talk about this in class.
Save the file.
Load it and see how fast it is!