---
title: "Sentiment analysis"
author: "Massimo Franceschet"
output:
  ioslides_presentation:
    css: ../style.css
    incremental: yes
editor_options: 
  chunk_output_type: inline
---

```{r setup, include=FALSE}
knitr::opts_chunk$set(cache = FALSE, message = FALSE, warning = FALSE)
```



## Sentiment analysis

* let's address the topic of opinion mining or **sentiment analysis**
* when human readers approach a text, we use our understanding of the emotional intent of words to infer whether a section of text is positive or negative
* or perhaps characterized by some other more nuanced emotion like surprise or disgust
* sentiment analysis is the automatic extraction of sentiment or emotion from an arbitrary text

## Dictionary-based methods

> Dictionary-based methods find the total sentiment of a piece of text by adding up the individual sentiment scores for each word in the text.

The tidytext package contains three **general-purpose lexicons**:

>* [bing](https://www.cs.uic.edu/~liub/FBS/sentiment-analysis.html), with binary associations (yes/no) for positive/negative sentiment;
>* [AFINN](http://www2.imm.dtu.dk/pubdb/views/publication_details.php?id=6010), with numeric scores (between -5 and +5) for positive/negative sentiment;
>* [nrc](http://saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm), with binary associations with eight basic emotions (anger, fear, anticipation, trust, surprise, sadness, joy, and disgust) and two sentiment poles (negative and positive)


```{r}
library(tidyverse)
library(tidytext)
library(janeaustenr)

# lexicons
get_sentiments("bing")
get_sentiments("afinn")
get_sentiments("nrc")
```

---

<div style="text-align:center"><img width = "800" height = "316" src ="img/tidyflow-ch-2.png"/></div>

## Dictionary-based methods

* not every English word is in the lexicons because many English words are pretty neutral 
* these methods do not take into account qualifiers before a word, such as in “no good” or “not true”; a lexicon-based method like this is based on **unigrams** only 
* one last caveat is that the size of the chunk of text that we use to add up unigram sentiment scores can have an effect on an analysis 
* a text the size of many paragraphs can often have positive and negative sentiment averaged out to about zero, while sentence-sized or paragraph-sized text often works better

## Sentiment analysis with inner join

* with data in a tidy format, sentiment analysis can be done as an **inner join**
* this is another of the great successes of viewing text mining as a tidy data analysis task 
* much as removing stop words or stemming is an antijoin operation, performing sentiment analysis is an inner join operation

## What are the most common joy words in *Emma*? 

```{r}
tidy_books <- austen_books() %>%
  group_by(book) %>%
  mutate(linenumber = row_number(),
         chapter = cumsum(str_detect(text, regex("^chapter [\\divxlc]", 
                                                 ignore_case = TRUE)))) %>%
  ungroup() %>%
  unnest_tokens(word, text)

nrc_joy <- get_sentiments("nrc") %>% 
  filter(sentiment == "joy")

tidy_books %>%
  filter(book == "Emma") %>%
  inner_join(nrc_joy) %>%
  count(word, sort = TRUE)

```

## How sentiment changes throughout each novel?

```{r}
jane_austen_sentiment <- tidy_books %>%
  inner_join(get_sentiments("bing")) %>%
  count(book, index = linenumber %/% 80, sentiment) %>%
  spread(sentiment, n, fill = 0) %>%
  mutate(sentiment = positive - negative)

jane_austen_sentiment

ggplot(jane_austen_sentiment, aes(index, sentiment, fill = book)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~book, ncol = 2, scales = "free_x")

```


## Which words contribute to each sentiment?

```{r}
bing_word_counts <- tidy_books %>%
  inner_join(get_sentiments("bing")) %>%
  count(word, sentiment, sort = TRUE) %>%
  ungroup()

bing_word_counts

bing_word_counts %>%
  group_by(sentiment) %>%
  top_n(10) %>%
  ungroup() %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(word, n, fill = sentiment)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~sentiment, scales = "free_y") +
  labs(y = "Contribution to sentiment",
       x = NULL) +
  coord_flip()
```

Notice an anomaly in the sentiment analysis; the word "miss" is coded as negative but it is used as a title for young, unmarried women in Jane Austen's works. If it were appropriate for our purposes, we could easily add "miss" to a custom stop-words.

## What are the most negative chapters in each of Jane Austen's novels?

```{r}
bingnegative <- get_sentiments("bing") %>% 
  filter(sentiment == "negative")

wordcounts <- tidy_books %>%
  group_by(book, chapter) %>%
  summarize(words = n())

tidy_books %>%
  semi_join(bingnegative) %>%
  group_by(book, chapter) %>%
  summarize(negativewords = n()) %>%
  left_join(wordcounts, by = c("book", "chapter")) %>%
  mutate(ratio = negativewords/words) %>%
  filter(chapter != 0) %>%
  group_by(book) %>% 
  top_n(1, ratio) %>%
  ungroup()

```

What is happening in these chapters?

* in Chapter 43 of *Sense and Sensibility* Marianne is seriously ill, near death
* in Chapter 34 of *Pride and Prejudice* Mr. Darcy proposes for the first time (so badly!) 
* chapter 46 of *Mansfield Park* is almost the end, when everyone learns of Henry's scandalous adultery 
* Chapter 15 of *Emma* is when horrifying Mr. Elton proposes 
* in Chapter 21 of *Northanger Abbey* Catherine is deep in her Gothic faux fantasy of murder
* Chapter 4 of *Persuasion* is when the reader gets the full flashback of Anne refusing Captain Wentworth and how sad she was and what a terrible mistake she realized it to be

## Play

1. download from Project Gutenberg [White Nights and Other Stories](http://www.gutenberg.org/ebooks/36034) by Fyodor Dostoyevsky ([RDS](nights.rds))
1. find most positive and negative words
1. notice that the novel is divided in 4 nights (include the last morning into the fourth night). Consider these as chapters. Compare positive and negative sentiments in each chapter



```{r echo=FALSE, eval=FALSE}
#library(gutenbergr)
#nights_down = gutenberg_download(36034)
#write_rds(nights, "nights.rds")
nights = read_rds("nights.rds")

# nights =
#   nights_down %>% 
#   mutate(linenumber = row_number()) %>% 
#   filter(linenumber >= 59, linenumber <= 2154) %>% 
#   mutate(linenumber = linenumber - 58,
#          chapter = cumsum(
#            str_detect(text, "^(.*) NIGHT")))

tidy_nights <- nights %>%
  unnest_tokens(word, text) %>%
  anti_join(stop_words)

tidy_nights %>%
  count(word, sort = TRUE)

```


```{r echo=FALSE, eval=FALSE}
bing_word_counts <- tidy_nights %>%
  inner_join(get_sentiments("bing")) %>%
  count(word, sentiment, sort = TRUE) %>%
  ungroup()

bing_word_counts

bing_word_counts %>%
  group_by(sentiment) %>%
  top_n(10) %>%
  ungroup() %>%
  mutate(word = reorder(word, n)) %>%
  ggplot(aes(word, n, fill = sentiment)) +
  geom_col(show.legend = FALSE) +
  facet_wrap(~sentiment, scales = "free_y") +
  labs(y = "Contribution to sentiment",
       x = NULL) +
  coord_flip()

```



```{r echo=FALSE, eval=FALSE}
bingnegative <- get_sentiments("bing") %>% 
  filter(sentiment == "negative")

wordcounts <- tidy_nights %>%
  group_by(chapter) %>%
  summarize(words = n())

negativeCount = 
  tidy_nights %>%
  semi_join(bingnegative) %>%
  group_by(chapter) %>%
  summarize(negativeWords = n()) %>%
  left_join(wordcounts, by = "chapter") %>%
  mutate(negativeRatio = negativeWords/words)

bingpositive <- get_sentiments("bing") %>% 
  filter(sentiment == "positive")

positiveCount = 
  tidy_nights %>%
  semi_join(bingpositive) %>%
  group_by(chapter) %>%
  summarize(positiveWords = n()) %>%
  left_join(wordcounts, by = "chapter") %>%
  mutate(positiveRatio = positiveWords/words)

positiveCount
negativeCount

sentimentCount = 
  inner_join(positiveCount, negativeCount) %>% 
  mutate(sentimentBalance = (positiveWords - negativeWords) / words)
sentimentCount

```










