Unsupervised ML: Topic Models for Classification

We want to know what specific issues areas, or TOPICS, our judgments cover. Topics, here, can mean different things. Think about any text: depending on the level of abstraction a text can have different “topics”.

A judicial decision, depending on the level of abstraction, can be about aboriginal rights, the consideration of aboriginal concerns in specific content such as sentencing, or the factual circumstances of the case.

We can use these levels of abstraction purposefully. When we are interested in abstract content, we can choose fewer topics that allow us to classify documents in content groups: this is a decision about aboriginal rights, this is a decision about immigration, etc.

When we are interested in more specific content, we can choose a higher number topics allow us to know what issues a document talks about: 60% of case A concerns the facts relating to burglaries, 20% relates to the criminal code and 20% relate to sentencing. Hence we can use topic models purposefully for different content analysis tasks.

Here, we focus on the first task: classification. All our cases have already been classified manually. We now want to know whether a topic modelling algorithm can guess these topics correctly. 

We begin by installing and loading the package topicmodels.

# Load package.

library(topicmodels) 


library(plyr)

Topic models are unsupervised machine learning algorithms. The only information we have to give to the computer is the number of topics we suspect to be in a set of documents. Based on the distribution of words in our documents, the computer then makes a statistically informed “guess” what these topics are.

The algorithm will not return a label for each topic, but it will return a list of words most associated with each topic. By reading this list of words, we can assign labels to each topic (and check whether the grouping is sensible).

In light of the above considerations, a LOW number of topics helps create sensible categories, while a HIGH number of topics will provide more specific categories.

Let’s first try to classify our decisions in 3 baskets.

# So we set the number of topics to 3.

k <- 3

# We can run our topic model on our dtm with 3 topics.

topic_model <- LDA(dtm,k)

# We create a new object terms with the top words for each topic as input.

terms <- terms(topic_model, 10)

terms

 ##

# Consult the most distinct 10 words for each topic. Based on these words, assign [1] aboriginal, [2] health, [3] immigration to the corresponding topic. # !!! IMPORTANT: ADAPT THAT ORDERING TO YOUR RESULTS!!!

topic_label <- c("health","immigration","aboriginal") 

colnames(terms) <- topic_label

terms

 ##

We can now use that classification to determine which is the most prominent topic per case.

Since we already have the correct, human-assigned label, we will combine the assignment with our classification prediction to compare results.

# We create a new object topics with the topic for each document as input and create a list that compares actual with predicted classifications.

topics <- topics(topic_model)

topics <- mapvalues(topics, from=1:3, to=topic_label)

topics <- as.data.frame(cbind(as.character(cases$issue),topics))

topics <- cbind(cases$case,topics)

colnames(topics) <- c("case","issue", "prediction")

head(topics, 5)

 ##

# We determine the probability of whether a topic is in a given document.

topics$prob_topic <- posterior(topic_model)$topics

head(topics,5)

 ##

# Here we check how many times the preassigned "issue" label is identical to our prediction. If the prediction is correct, we count it as a hit. We start the hit count with 0 and add one every time the assignment was correct. We can then divide the hits by the total number of guesses.

hits <- 0 

for (row in 1:nrow(topics)) {
if (topics$issue[row] ==topics$prediction[row] ) {
hits <- hits+1
}
}

correctness <- hits/length(topics$issue)

correctness

 ##
[1] 0.8983051