3

My question is similar to this one about whether or not tidy data vary by context. My initial reading of Hadley's article on the concept (here) led me to believe that all data can be reduced to a tidy data format (i.e., the quote about happy and unhappy families), and that the format is consistent regardless of the specific analysis to be conducted. But now I'm not so sure.

My confusion started while reading Modern Data Science with R by Ben Baumer and colleagues, in which they state that the following dataset is tidy because each case is an individual ballot (the rows)

|Precinct |First           |Second      |Third      |Ward |
|---------|----------------|------------|-----------|-----|
|P-10     |BETSY HODGES    |undervote   |undervote  |W-7  |
|P-06     |BOB FINE        |MARK ANDREW |undervote  |W-10 |
|P-09     |KURTIS W. HANNA |BOB FINE    |MIKE GOULD |W-10 |
|P-05     |BETSY HODGES    |DON SAMUELS |undervote  |W-13 |
|P-01     |DON SAMUELS     |undervote   |undervote  |W-5  |
|P-04     |undervote       |undervote   |undervote  |W-6  |

But this doesn't look tidy to me. Rather, I would expect it to look like this:

| Ballot  |Precinct |Ward |Choice |Candidate    |
|---------|---------|-----|-------|-------------|
|        1|P-10     |W-7  |First  |BETSY HODGES |
|        1|P-10     |W-7  |Second |undervote    |
|        1|P-10     |W-7  |Third  |undervote    |
|        2|P-06     |W-10 |First  |BOB FINE     |
|        2|P-06     |W-10 |Second |MARK ANDREW  |
|        2|P-06     |W-10 |Third  |undervote    |

But in this version a case would represent a choice within a ballot.

Are both formats tidy depending on the unit of analysis (i.e., what defines a case)?

The latter generally seems preferable to me. For example, I can pretty easily use the latter format to produce a figure like the below (subset to show only candidates with 2000 or more total votes), which seems at the very least very difficult to produce with the former format.

votes_by_choice

So, to reiterate, can a dataset have multiple tidy forms, depending on the level of analysis? Or is the second form above not tidy?

(I realize at a certain point it doesn't matter, so long as you can get it to do what you want, but I'd like to be clear on terminology)

Postscript: Code

library(mdsr)
library(tidyverse)

minn2 <- Minneapolis2013 %>% 
    mutate(ballot = row_number()) %>% 
    gather(choice, candidate, First:Third) %>% 
    arrange(ballot)

pd <- minn2 %>% 
    group_by(candidate) %>% 
    mutate(n = n()) %>% 
    filter(n > 2000) %>% 
    group_by(choice, candidate) %>% 
    tally(wt = NULL)  

ggplot(pd, aes(candidate, nn, fill = choice)) +
    geom_bar(stat = "identity") +
    xlab("Candidate") +
    ylab("Number of votes") +
    theme(axis.text.x = element_text(angle = 45, hjust = 1))
Daniel Anderson
  • 133

1 Answers1

2

Hmm...I see your point. In our example, the unit of analysis is a ballot, which contains three votes. In your example, the unit of analysis is a vote.

From @hadley's paper, he defines tidy data such that:

This is Codd’s 3rd normal form (Codd 1990), but with the constraints framed in statistical language, and the focus put on a single dataset rather than the many connected datasets common in relational databases.

The question then is whether these examples are in 3NF (or 2NF).

My reading is this (but while I have lots of SQL experience, I have no formal training in relational algebras or database normalization):

  • Your example is not in 3NF, because the Ward and Precinct information is redundant based on the Ballot.
  • Our example is in 3NF, and is therefore tidy. This is true because none of the information in the columns is redundant. That is, you can't reconstruct someone's ballot if any of the columns were missing. However, we omitted the Ballot id variable! If that variable is present, then our example fails to be 3NF by the same rationale as yours. In this case, the 3NF way to store these data would be with two tables:

-- Votes, which looks like this:

| Ballot |Choice |Candidate | |---------|-------|-------------| | 1|First |BETSY HODGES | | 1|Second |undervote | | 1|Third |undervote | | 2|First |BOB FINE | | 2|Second |MARK ANDREW | | 2|Third |undervote |

-- Ballots, which looks like this:

| Ballot |Precinct |Ward | |---------|---------|-----| | 1|P-10 |W-7 | | 2|P-06 |W-10 | | 3|P-09 |W-10 | | 4|P-05 |W-13 |

So my suspicion is that 3NF is unique, given the set of columns. I would love for someone else to weigh in and clarify all of this.

beanumber
  • 36
  • 1