---
title: "The great library heist"
output: html_document
editor_options: 
  chunk_output_type: inline
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(cache = FALSE, echo = TRUE, message = FALSE, warning = FALSE)
```


Suppose a vandal has broken into your study and torn apart four of your books ([RDS](books.rds)):

>* *Great Expectations* by Charles Dickens
>* *The War of the Worlds* by H.G. Wells
>* *Twenty Thousand Leagues Under the Sea* by Jules Verne
>* *Pride and Prejudice* by Jane Austen

This vandal has torn the books into individual chapters, and left them in one large pile. How can we restore these disorganized chapters to their original books? 

This is a challenging problem since the individual chapters are **unlabeled**: we don't know what words might distinguish them into groups. 

Use topic modeling to discover how chapters cluster into distinct topics, each of them (presumably) representing one of the books.

## Download books

```{r}
library(tidyverse)
library(tidytext)
library(topicmodels)
# library(gutenbergr)
# 
# titles <- c("Twenty Thousand Leagues under the Sea", 
#             "The War of the Worlds",
#             "Pride and Prejudice", 
#             "Great Expectations")
# 
# 
# books <- gutenberg_works(title %in% titles) %>%
#   gutenberg_download(meta_fields = "title")

#write_rds(books, "books.rds")
books = read_rds("books.rds")
```


# Find topics with LDA

```{r}
# divide into documents, each representing one chapter
by_chapter <- books %>%
  group_by(title) %>%
  mutate(chapter = cumsum(str_detect(text, regex("^chapter ", ignore_case = TRUE)))) %>%
  ungroup() %>%
  filter(chapter > 0) %>%
  unite(document, title, chapter)

# split into words
by_chapter_word <- by_chapter %>%
  unnest_tokens(word, text)

# find document-word counts
word_counts <- by_chapter_word %>%
  anti_join(stop_words) %>%
  count(document, word, sort = TRUE) %>%
  ungroup()

word_counts

# cast into DTM
chapters_dtm <- word_counts %>%
  cast_dtm(document, word, n)

chapters_dtm

# create 4 topics with LDA
chapters_lda <- LDA(chapters_dtm, k = 4, control = list(seed = 1234))
chapters_lda
```

## Analyse probabilities


```{r}
# per-topic-per-word probabilities
chapter_topics <- tidy(chapters_lda, matrix = "beta")
chapter_topics

# top 5 terms within each topic
top_terms <- chapter_topics %>%
  group_by(topic) %>%
  top_n(5, beta) %>%
  ungroup() %>%
  arrange(topic, -beta)

top_terms


# visualize top terms
top_terms %>%
  mutate(term = reorder_within(term, beta, topic)) %>%
  ggplot(aes(term, beta, fill = factor(topic))) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~ topic, scales = "free") +
  coord_flip() +
  scale_x_reordered()

# per-document-per-topic probabilities
chapters_gamma <- tidy(chapters_lda, matrix = "gamma")
chapters_gamma

# separate title and chapter
chapters_gamma <- chapters_gamma %>%
  separate(document, c("title", "chapter"), sep = "_", convert = TRUE)

chapters_gamma

# visualize the per-document-per-topic probability
chapters_gamma %>%
  mutate(title = reorder(title, gamma * topic)) %>%
  ggplot(aes(factor(topic), gamma)) +
  geom_boxplot() +
  facet_wrap(~ title)

```

We notice that almost all of the chapters from Pride and Prejudice, War of the Worlds, and Twenty Thousand Leagues Under the Sea were uniquely identified as a single topic each. It does look like some chapters from Great Expectations (which should be topic 4) were somewhat associated with other topics.


## Miss-classification

Are there any cases where the topic most associated with a chapter belonged to another book? 

```{r}
chapter_classifications <- chapters_gamma %>%
  group_by(title, chapter) %>%
  top_n(1, gamma) %>%
  ungroup()

chapter_classifications

# get consensus topic for each book (the most common topic among its chapters)
book_topics <- chapter_classifications %>%
  count(title, topic) %>%
  group_by(title) %>%
  top_n(1, n) %>%
  ungroup() %>%
  transmute(consensus = title, topic)

chapter_classifications %>%
  inner_join(book_topics, by = "topic") %>%
  filter(title != consensus)

chapters_gamma %>% 
  filter(title == "Great Expectations", chapter == 23)

chapters_gamma %>% 
  filter(title == "Great Expectations", chapter == 54)

```

We see that only two chapters from Great Expectations were misclassified, as LDA described one as coming from the Pride and Prejudice topic (topic 1) and one from The War of the Worlds (topic 3). That’s not bad for unsupervised clustering!
