A small data science example 🔢

Exploring the world in data

One of the challenges in real data science is getting data from different sources in many different formats. In this notebook, we will explore some facts about the world using data taken from a TSV file, an Excel spreadsheet, and a database query.

Life expectancy

First, we'll read in a TSV file containing the most recent CIA World Factbook data using the meta-csv library.

Expanding the results in the data viewer tells us that there are some nil values in columns of interest, and that our TSV importer was thrown off by this fact and so didn't convert the numerical columns to number types.

We're going to post process this table a bit with ordinary Clojure sequence functions to filter out rows that have nils for our columns of interest, select those rows, convert strings to numbers, and — because we're Clojurists — convert keys to keywords.

Things look pretty good in the data structure browser, but it would be easier to get an overview in tabular form. Luckily, Clerk's built in table viewer is able to infer how to handle all of the most common configurations of rows and columns automatically.

We can also graph the data to see if there are any visible correlation between our two variables of interest, GDP per capita and life expectancy.

Unsurprisingly, it seems that living in an extremely poor country has negative consequences for life expectancy. On the other hand, it looks like things start to flatten out once GDP/capita goes above $10-15k/year. Some other interesting patterns also emerge: Singapore and Japan have similar life expectancies, despite the former's GDP being twice the latter's, and Qatar — the richest nation in the dataset by GDP/capita — has similar average life expectancy as the Dominican Republic.

Inequality

Now, let's try the same experiment using information from a spreadsheet containing the GINI coefficient — a widely used measure of income inequality — for each country. We're going to use a library called Docjure that provides access to Microsoft Office file formats.

Docjure's API is a bit low-level and doesn't make the obvious tasks easy, so we're going to use this helper function to make the code below clearer. Check out the line-by-line comments to see how this function works.

Now we're going to use a few lines of code to:

1. Load the spreadsheet data.
2. Use clojure.set's join function to combine our freshly loaded GINI spreadsheet with our previously prepared life expectancy data, which works because they are both sequences of maps that have a :country key.
3. Assoc a :gini key in each map to the World Bank's number, but falling back to the CIA's estimate. (These kinds of small programmatic tasks are a constant feature of data wrangling.)

Expanding the Clojure data structures makes it look like this will work for our comparisons. Let's plot the data to see a list of countries from most to least equal:

And now to have a look at whether inequality and life expectancy are correlated:

It seems like the mass of long lived countries are also in the lower two thirds of the inequality distribution. A little filtering shows is that the only really long-lived countries above a GINI coefficient of ~50 is Hong Kong.

Happiness

Let's look at happiness! This time, we'll use jdbc.next to perform a SQL query on a Sqlite data containing a table of countries and their relative happiness ratings. Note that we're changing the column name :country_or_region to :country using clojure.set's rename-keys function so that this table will be easy to join with our others.

Looking at the happiness data, it appears that all the usual suspects — Nordics, Western Europeans, Canadians, and Kiwis — are living pretty good lives by their own estimation. Looking closer, we see that although the top twenty countries all relatively prosperous, it's clear that GDP is not strongly correlated with happiness within that cohort.

Next, we're computing a linear regression for this dataset using kixi.stats.

We'll use this linear regression to augment out dataset so each datapoint also gets a :regression value.

Let's graph the relationship between happiness and GDP to get a bird's eye view on the situation over our entire dataset. You can mouse over individual data points to get more info:

It looks, as we might have expected, like richer countries are happier than poor ones in general, though with variations and outliers. For example, Finland is in first place but has a similar GDP/capita as number 58, Japan. Perhaps even more striking, Qatar has the highest GDP/capita in the dataset, but Qataris are on average about as happy as people in El Salvador. Likewise, Botswana has five times the GDP/capita of Malawi, but its people are no happier for it. If I were forced to guess why, I might theorize that a properous country with all of its wealth concentrated in very few hands can still be a fairly wretched place to live for the average person to live.

One way to investigate this possibility is to plot the correlation between equality and happiness in the rich world. We'll use join again, but we'll first use clojure.set's project (named by analogy to SQL projection) to pluck just the :country and :score from the happiness dataset, then sort by the GDP and take the top 20 countries.

This does, at least at first glance, support the notion that the happiest people — just like the longest lived ones — tend to inhabit countries in the more equal part of the GINI distribution.

I hope this example gives you some ideas about things you'd like to investigate.