Neural Networks

Since the terms "I", "this", and "movie" appear with equal frequency across both labels, their weights $w_1​$, $w_4​$, and $w_5​$ are neutralized. On the other hand, the terms "love" and "hate" appear only with the positive and negative labels, respectively. Therefore, while the weight $w_1​$ for "love" ($x_1​$) contributes positively to the label $y=1$, the weight $w_2$ for "hate" ($x_2​$) has a negative impact on the label $y=1$. Furthermore, as positive and negative sentiment labels are equally presented in this corpus, the bias $b$ is also set to 0.

Given the weight vector and the bias, we have $\mathrm{x}_1 \cdot \mathrm{w}^T + b = 1.5$ and $\mathrm{x}_2 \cdot \mathrm{w}^T + b = -1.5$, resulting the following probabilities:

As the probability of $x_1​$ being $y=1$ exceeds $0.5$ (50%), the model predicts the first sentence to convey a positive sentiment. Conversely, the model predicts the second sentence to convey a negative sentiment as its probability of being $y=1$ is below 50%.

Under what circumstances would the bias $b$ be negative in the above example? Additionally, when might neutral terms such as "this" or "movie" exhibit non-neutral weights?

Softmax regression, aka. multinomial logistic regression, is an extension of logistic regression to handle classification problems with more than two classes. Given an input vector $\mathrm{x} \in \mathbb{R}^{1 \times n}$ and its output lable $y \in \{0, \ldots, m-1\}$, the model uses the softmax function to estimates the probability that $\mathrm{x}$ belongs to each class separately:

The weight vector $\mathrm{w}_k​$ assigns weights to $\mathrm{x}$ for the label $y=k$, while $b_k​$ represents the bias associated with the label $y=k$.

Then, the input vectors $\mathrm{x}_1$, $\mathrm{x}_2$, and $\mathrm{x}_3$ for the sentences can be created using the bag-of-words model:

Let $y_1 =1$, $y_2 = 0$, and $y=2$ be the output labels of $\mathrm{x}_1$, $\mathrm{x}_2$, and $\mathrm{x}_3$, representing postive, negative, and neutral sentiments of the input sentences, respectively. Then, weight vectors $\mathrm{w}_1$, $\mathrm{w}_2$, and $\mathrm{w}_3$ can be trained using softmax regression as follows:

Unlike the case of logistic regression where all weights are oriented to $y = 1$ (both $w_1$ and $w_2$ giving positive and negative weights to $y = 1$ respectively, but not $y=0$), the values in each weigh vector are oriented to each corresponding label.

Given the weight vectors and the biases, we can estimate the following probabilities for $\mathrm{x}_1$:

Since the probabiilty of $y=1$ is the highest among all labels, the model predicts the first sentence to convey a positive sentiment. For $\mathrm{x}_3$, the following probabilities can be estimated:

Since the probabiilty of $y=2$ is the highest among all labels, the model predicts the first sentence to convey a neutral sentiment.

Softmax regression always predicts $m$ values so that it is represented by an output vector $\mathrm{y} \in \mathbb{R}^{1 \times m}$, wherein the $i$'th value in $\mathrm{y}$ contains the probability of the input belonging to the $i$'th class. Similarly, the weight vectors for all labels can be stacked into a weight matrix $\mathrm{W} \in \mathbb{R}^{m \times n}$, where the $i$'th row represents the weight vector for the $i$'th label.

With this new formulation, softmax regression can be defined as $\mathrm{y} = \mathrm{softmax}(\mathrm{x} \cdot \mathrm{W}^T)$, and the optimal prediction can be achieved as $\mathrm{argmax}(\mathrm{y})$, which returns a set of labels with the highest probabilities.

A multilayer perceptron (MLP) is a type of Feedforward Neural Networks consisting of multiple layers of neurons, where all neurons from one layer are fully connected to all neurons in its adjecent layers. Given an input vector $\mathrm{x} \in \mathbb{R}^{1 \times n}$ and an output vector $\mathrm{y} \in \mathbb{R}^{1 \times m}$, the model allows zero to many hidden layers to generate intermediate representations of the input.

Let $\mathrm{h} \in \mathbb{R}^{1 \times d}$ be a hidden layer between $\mathrm{x}$ and $\mathrm{y}$. To connect $\mathrm{x}$ and $\mathrm{h}$, we need a weight matrix $\mathrm{W}_x \in \mathbb{R}^{n \times d}$ such that $\mathrm{h} = \mathrm{activation}(\mathrm{x} \cdot \mathrm{W}_x)$, where $\mathrm{activation}()$ is an activation function applied to the output of each neuron; it introduces non-linearity into the network, allowing it to learn complex patterns and relationships in the data. Activation functions determine whether a neuron should be activated or not, implying whether or not the neuron's output should be passed on to the next layer.

Similarly, to connect $\mathrm{h}$ and $\mathrm{y}$, we need a weight matrix $\mathrm{W}_h \in \mathbb{R}^{m \times d}$ such that $\mathrm{y} = \mathrm{softmax}(\mathrm{h} \cdot \mathrm{W}_h^T)$. Thus, a multilayer perceptron with one hidden layer can be represented as:

Consider a corpus comprising the following five sentences the corresponding labels ($\Rightarrow$):

D1: I love this movie $\Rightarrow$ postive

D2: I hate this movie $\Rightarrow$ negative

D3: I watched this movie $\Rightarrow$ neutral

D4: I truly love this movie $\Rightarrow$ very positive

D5: I truly hate this movie $\Rightarrow$ very negative

The input vectors $\mathrm{x}_{1..5}$ can be created using the bag-of-words model:

The first weight matrix $\mathrm{W}_x \in \mathbb{R}^{7 \times 5}$ can be trained by an MLP as follows:

Given the values in $\mathrm{W}_x$, we can infer that the first, second, and third columns represent "love", 'hate", and "watch", while the fourth and fifth columns learn combined features such as {"truly", "love"} and {"truly", "hate"}, respectively.

Each of $\mathrm{x}_{1..5}$ is multiplied by $\mathrm{W}_x$ to achieve the hiddner layer $\mathrm{h}_{1..5}$, respectively, where the activation function is designed as follow:

The second weight matrix $\mathrm{W}_h \in \mathbb{R}^{5 \times 5}$ can also be trained by an MLP as follows:

By applying the softmax function to each $\mathrm{h}_i \cdot \mathrm{W}^T_h$, we achieve the corresponding output vector $\mathrm{y}_i$:

The prediction can be made by taking the argmax of each $\mathrm{y}_i$.