cs-05

Case Study 5: I can clean your data

Background

The Scientific American argues that humans have been getting taller over the years. As the data scientists that we are becoming, we would like to find data that validates this concept. Our challenge is to show different male heights across the centuries.

This project is not as severe as the two quotes below, but it will give you a taste of pulling various data and file formats together into “tidy” data for visualization and analysis. You will not need to search for data as all the files are listed here

1. “Classroom data are like teddy bears and real data are like a grizzly bear with salmon blood dripping out its mouth.” - Jenny Bryan
2. “Up to 80% of data analysis is spent on the process of cleaning and preparing data” - Hadley Wickham

• Back or Course Website

Tasks

• Use the correct Polars functions from to load the 5 data sets listed here
• Clean the Worldwide estimates .xlsx file
  • Make sure the file is in long format with year as a column. See here for an example of the final format.
  • Use appropriate functions to create a decade column.
• Import the other five datasets into R and combine them into one clean dataset.
  • This dataset should have the following columns - birth_year, height.cm, height.in, and study_id
  • The BLS wage data does not have birth information. Let’s assume it is mid-twentieth century and use 1950.
  • See the reading of Task 8 for how to read in dbf files.
• Save the two tidy datasets to your repository - The world country estimates and the row-combined individual measurements.
• Make a plot with decade on the x-axis and height in inches on the y-axis with the points from Germany highlighted based on the data from the .xlsx file.
• Make a small-multiples plot of the five studies to examine the question of height distribution across centuries
• Create a marimo notebook with 1-2 paragraphs summarizing your graphics and how those graphics answer the driving question
• Export your notebook to .html and have both the notebook and HTML into your git repository
• Find two other student’s compiled files in their repository and provide feedback using the issues feature in GitHub (If they already have three issues find a different student to critique)
• Address 1-2 of the issues posted on your project and push the updates to GitHub
  
  Use the function in this code snippet to read in .dta files

import pandas as pd
import polars as pl
from pathlib import Path

def read_dta_polars(file_path: str | Path) -> pl.DataFrame:
    file_path = Path(file_path)

    if not file_path.exists():
        raise FileNotFoundError(f"File not found: {file_path}")

    try:
        # Load into pandas first (pandas supports .dta)
        pdf = pd.read_stata(file_path)

        # Convert to Polars DataFrame
        return pl.from_pandas(pdf)

    except ValueError as e:
        raise ValueError(f"Error reading .dta file: {e}")
    except Exception as e:
        raise RuntimeError(f"Unexpected error: {e}")