Review On Sentiment Analysis English Language Essay

Published: Last Edited:

This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.


Recent years have seen an ever increasing popularity in online opinion resources where people can actively and easily seek out the opinions of others in a variety of different forms. (Kim and Hovy, 2007, Pang and Lee, 2008) Online resources such as news articles, web logs or blogs, product review sites such as and online discussion groups such as are all perfect examples of places where people share their opinions, often with complete strangers on the internet. (Kim and Hovy, 2006a) An important part of how people make decisions has always involved the opinions of others, which is why opinions expressed by users is viewed as one of the most important types of information on the internet today. (Ding et al., 2008, Pang and Lee, 2008) People are eager to express their opinions and find out other people's views and experiences on a wide variety of subjects and a growing number of people are using the internet as a means to do this, even if they have never met before. (Bruce and Wiebe, 1999, Kim and Hovy, 2007, Pang and Lee, 2008, Kim and Hovy, 2006a, Yi et al., 2003)

The internet has made it easier for people to access information which in turn has increased the demand for information extraction systems that organise documents according to their relevant categories in order to make it easier for users to search for relevant content. (Pang et al., 2002, Eguchi and Lavrenko, 2006) Traditional information extraction systems seek to classify a document or a piece of text by the topic or subject matter to which it belongs. Eg sport. (Pang et al., 2002, Pang and Lee, 2008, Wiebe, 2000, Hatzivassiloglou and Wiebe, 2000) However, the subject matter of a document may not be the only criteria for sorting its relevancy to a particular user. The relevancy of a document or a particular piece of text, to a user or task, is also influenced by additional components such as the reasons behind the current state of the topic in question and the attitudes in favour or against a particular entity. (Wiebe, 2000, Hatzivassiloglou and Wiebe, 2000) In other words, the opinion of the author regarding the topic in question can be just as important to the user or task as the subject matter itself. In recent years, the increase in user generated content on the internet, which more often than not express some form of opinion, has seen an increasing amount of interest in methods that automatically detect and extract opinions and sentiment in text. (Choi et al., 2005, Wiebe and Riloff, 2005, Pang et al., 2002) Automatic mining and extraction of opinions and sentiments in text from different sources of information can provide a number of similar benefits and opportunities across a variety of applications and to a wide variety of user groups such as consumers, businesses, organisations and governments. (Pang et al., 2002, Turney, 2002, Bruce and Wiebe, 1999) (Nasukawa and Yi, 2003)

Topic Background and Definitions

The terms opinion mining, sentiment analysis and subjectivity analysis are all used interchangeably in the body of work of this area. Essentially, these terms all refer to the same problem. i.e. the treatment of opinion, sentiment and subjectivity in text using computers and algorithms. (Pang and Lee, 2004, Pang and Lee, 2008) The term "opinion mining" is used by Dave et al (2003) when they propose a tool that automatically distinguishes between positive and negative product reviews. An opinion mining tool should process a set of results for a particular item that is being searched for, generate a list of the product's features and classify the opinions about each of them as being "poor" "mixed" or "good". (Dave et al., 2003) Ku et al (2006) develops this idea further by identifying three important techniques for the automatic mining and organisation of opinions which are; "Opinion extraction, opinion summarisation and opinion tracking." Research on opinion extraction and opinion mining from articles is generally carried out at three levels; word level, sentence level and document level. (Bruce and Wiebe, 1999, Kim and Hovy, 2006a) Ku et al. (2006) agree with Dave et al (2003) that an aggregation or summarisation of opinions should take place which classifies the "sentiment polarities" into categories (positive, negative or neutral) along with the "degree" (strength) of each polarity.

The term sentiment analysis has emerged as the term that encompasses the entire field of study and is broadly described by Pang and Lee (2008) as "the computational treatment of opinion, sentiment and subjectivity in text." The word "treatment" would suggest that this includes all aspects of dealing with non-objective language in documents such as identifying subjective language, extracting opinions, analysing and classifying opinions in terms of polarity and degrees of polarity and finally the summarisation of these opinions. Also the use of the word "text" implies that the term covers every instance of text at all levels including, news articles, online reviews, web blogs and forums at word sentence and document levels. Sentiment analysis and opinion mining in their broadest terms describe the same field of study. (Pang and Lee, 2008) Other researchers in the area describe sentiment in more specific terms. Sentiment analysis is often desribed as the process of identifying expressions of sentiment, emotions and evaluations in text and analysing whether these expressions convey positive, negative or neutral opinions towards an entity. (Nasukawa and Yi, 2003, Wilson et al., 2005) The word "entity" here may describe a person, place, object or topic. Pang and Lee (2004) believe that sentiment analysis "seeks to identify the view point underlying a text span." Perhaps this is a rather simplified definition that does not take into account the various tasks that the process entails.

Applications of sentiment analysis

Generally speaking, any application that is intended to record information could benefit from being able to recognise and separate fact from opinion in text. (Riloff and Wiebe, 2003, Riloff et al., 2003) Because text documents can contain both factual and opinion sentences, an application that is able to detect factual (objective) and opinionated (subjective) sentences would help us decide what information we want to extract and and how we go about organising and presenting this information for people or tasks to use. (Bethard et al., 2004, Yu and Hatzivassiloglou, 2003) Furthermore, by labeling a document such as an online review, in terms of its sentiment, i.e. whether it is positively or negatively orientated, websites would be able to provide readers with summaries of users' experiences and opinions. (Pang et al., 2002, Pang and Lee, 2004) This type of sentiment analysis is particularly useful when a user wants to quickly find out how many positive and negative reviews exist respectfully on an online review website for a particular product or service. A system that summarises documents and articles for users, would also benefit enormously by using sentiment analysis to separate objective sentences from subjective sentences as some readers may only be interested in the hard facts rather than opinions, plus many summaries tend to include only facts. (Wiebe, 2000, Hatzivassiloglou and Wiebe, 2000) Governments, companies, political parties and organisations can all benefit from the automatic identification and extraction of opinions in text and it is in their best interest to avail of such tools in order to automatically track attitudes and feelings towards them in the news, online forums and blogs. (Wiebe and Riloff, 2005) As well as allowing the tracking of sentiment towards them, a system that detects favourable and unfavourable opinions would provide organisations with powerful tools for competitive analysis, marketing analysis and also allow them to react in real time to unfavourable rumours. (Nasukawa and Yi, 2003)

As well as giving them a competitive edge in the market and real-time updates on their public perception, organisations can utilise sentiment analysis as an inexpensive means of gaining feedback from their customers as opposed to spending huge amounts of time and money on creating and analysing customer satisfaction surveys. Detecting and analysing the public's feelings online in the form of discussion forums, blogs and news articles is also much less limiting in terms of the sample size and the difficulties that come with making an effective questionnaire. (Nasukawa and Yi, 2003, Hatzivassiloglou et al., 2001)

Sentiment Classification

Sentiment classification is a very broad term that exists within the umbrella of sentiment analysis. The problem of classification exists in many applications of sentiment analysis as some of the most fundamental issues within this area of research fall under the term of 'classification.' Problems such as identifying a document's polarity, (how positive or negative it is) the degree or strength of it's polarity, subjectivity detection, opinion identification and semantic role labeling (identifying the roles of words in a sentence) can all be labelled as classification problems. (Pang and Lee, 2008) Pang and Lee (2008) also include regression, ranking and extraction problems under the term of 'classification'. I will now go through the main classification issues that exist within the area of sentiment analysis.

Sentiment Polarity Classification

Pang and Lee (2008) describe the process of sentiment polarity classification as "the binary classification task of labeling an opinionated document as expressing an overall positive or negative opinion." The task is also referred to as 'polarity classification' in some articles. A related term is that of 'semantic orientation' which refers to the positive, negative or neutral assocation that belongs to a word or a phrase. Eg. The word 'good' would be said to have a positive semantic orientation. (Turney, 2002) Recent years have seen a huge growth in interest in the area of sentiment polarity classification. (Eguchi and Lavrenko, 2006, Pang et al., 2002) A lot of research in the area has involved the labeling of online reviews for products as either positive or negative. (Riloff and Wiebe, 2003, Hu and Liu, 2004) Others have investigated the usefulness of labeling movie reviews by their overall sentiment polarity with the overall purpose of providing readers with useful summaries on review websites such as and (Pang et al., 2002, Turney, 2002) Dave et al. (2003), Turney (2002) and Pang et al. (2002) aim to classify reviews at the document level, whereas Hu and Liu (2004) look at classifying reviews by identifying a product's attributes and features and extracting opinion sentences regarding those product features. Therefore Hu and Liu (2004) approach the sentiment polarity classification problem at the sentence level. For the purpose of this review I will look at document level and sentence level polarity classification separately in turn.

Document Level Sentiment Polarity Classification

Turney (2002) uses an unsupervised learning algorithm for classifying reviews as either "recommended" or "not recommended" by calculating the average semantic orientation of the phrases in reviews that contain adjectives or adverbs. As I explained above, a phrase has a positive semantic orientation when it suggests something postive, e.g. 'well directed' and a negative semantic orientation when it has a negative connotation, e.g. 'amateur acting'. (Turney, 2002) An algorithm takes the written review as input and the classification of the review is the final output. Pang et al. (2002) is the work that is closest in relation to Turney's (2002) paper. As opposed to using an unsupervised learning algorithm, Pang et al. (2002) experiment with three machine learning techniques (Naive Bayes classifier, Maximum entropy classifier and Support Vector Machines) and compares the results of all three techniques in their accuracy at classifying reviews as positive or neagtive that were taken from the internet movie database, They consciously left out reviews that they were classified as having a neutral opinion as they only wanted to concentrate on discriminating between reviews that are either positive or negative.

Similar to Turney's (2002) and Pang et al. (2002) work, Dave el al. (2003) aims at developing a classifier that automatically distinguishes between positive and negative reviews using information retrieval techniques for feature extraction and scoring. A tool that automatically sifts through reviews and aggregates them in terms of their opinion is very useful and is already in use on such review sites as (Riloff and Wiebe, 2003, Pang et al., 2002) Similar to the sentiment classification research conducted at document level above, Dave et al. (2003) uses input data of reviews taken from two review websites; C|net and Amazon and uses review that are taken from different caregories of products on each website in order to train their review classfier.

Sentence-level and Phrase-level Sentiment Polarity Classification

Although determining the overall semantic orientation of a document is useful for distinguishing between positive or negative reviews or when one is trying to distinguish between different types of articles, eg. Editorials vs. Factual news, a system that is capable of answering complex opinion questions requires sentiment analysis at the sentence-level or even phrase-level.(Wilson et al., 2005, Yu and Hatzivassiloglou, 2003, Breck et al., 2007)

Other researchers have approached the problem of review polarity classification at the sentence level, as opposed to the document level. (Popescu and Etzioni, 2005, Hu and Liu, 2004) A feature-based review classification system seeks to extract opinions about certain product attributes that people have expressed feelings towards in an online written review. (Hu and Liu, 2004, Popescu and Etzioni, 2005) Instead of trying to classify the polarity of the review as a whole document, Hu and Liu (2004) are only interested in the opinions being expressed about certain features belonging to the product in order to generate "feature-based summaries of customer reviews of products sold online." With the rapid expansion of e-commerce websites and also because people are writing more and more product reviews, making an informed decision about whether to buy a product can be an overwhelming process that involves sifting through a large number of online reviews. (Popescu and Etzioni, 2005) (Hu and Liu, 2004) A system that automatically identifies product features that people like or dislike, extracts those opinion sentences and then summarises them, would be much more beneficial for users than reading hundreds of product reviews. (Hu and Liu, 2004) Popescu and Etzioni (2005) go one step further than Hu and Liu (2004) in their review classifier. After they identify the product features, the opinions associated with those features and the polarity of those opinions, they also organise those opinions based on their strength.(Popescu and Etzioni, 2005)

Sentence-level sentiment classifiers and phrase-level sentiment classifiers have been explored in other research. (Wilson et al., 2005, Kim and Hovy, 2004) A related study to the ones presented above involves extracting the reasons behind the negative or positive opinions in online product reviews. (Kim and Hovy, 2006a) The principle behind this research is similar to exploring what features of a product or service people found good or bad. Knowing the exact reason behind a positive or negative review is much more informative and helpful to someone than simply a postive or negative rating. (Kim and Hovy, 2006a) A system that automatically identifies reasons in online reviews is trained by identifying pros and cons in the text and labeling the sentences that they appear in, with the assumption that pros and cons in the text are closely related to the reasons behind them. (Kim and Hovy, 2006a)

Approaches to sentence-level or phrase-level subjectivity detection (discussed later) and polarity classification often involve using a large group or lexicon of positive and negative words and tagging them on their own as having either a positive, negative or neutral polarity. (Wilson et al., 2005, Pang et al., 2002, Breck et al., 2007, Esuli and Sebastiani, 2006, Kim and Hovy, 2004) For example the word 'fantastic' has a positive polarity and the word 'brutal' has a negative polarity. Words can also have a neutral polarity eg. Mediocre. Wilson et al (2005) focus on the problem of automatically determining the polarity of phrases by tagging words in a lexicon with their "prior polarity" i.e. The polarity that the word has when it's on its own e.g. 'Great' has a positive polarity on its own, however, the phrase 'a great danger' has a negative contextual polarity. Wilson et al. (2005) argue that the "contextual polarity" of a phrase in which a word appears might be different to the word's' "prior polarity" and therefore knowing the word's prior polarity is of no advantage in determining the polarity of a phrase that contains it. Also, the placement of 'negation' words such as 'not' or 'no' close to words that have a positive polarity tends to result in giving the phrase or sentence the opposite polarity. (Wilson et al., 2005) Wilson et al. (2005) aim to automatically distinguish between prior and contextual polarity by using a lexicon of previously tagged words as clues to train a machine learning classifier. The system would be able to automatically identify the contextual polarity for a large subset of sentiment expressions.

Related research on sentence-level sentiment classifiers includes examining the liklihood of a sentence containing information with a specific sentiment polarity on a given topic, (Eguchi and Lavrenko, 2006) and extracting sentiment with positive or negative polarities for specific subjects. (Hatzivassiloglou et al., 2001, Nasukawa and Yi, 2003) In order to improve the accuracy of sentiment analysis it is important to properly associate sentiment expressions in sentences with a specific topic but this can only be done below the document level and be difficult to achieve. (Yi et al., 2003, Nasukawa and Yi, 2003) Yi et al. (2003) develop a classifier that firstly extracts topic-specific features similar to Hu and Liu's (2004) feature-based classifier, then extracts the sentiment associated with each of those features and finally establishes a relationship between the two. The identification of the relationship between the topic and sentiment phrases and words is an important issue because the sentiment polarity may be completely different depending on the relationship. (Nasukawa and Yi, 2003) For example, in the sentence 'A is better than B' the expression 'is better than' is positively orientated towards A but negatively orientated towards B.

Eguchi and Lavrenko's (2006) work is similar to the above research but they focus on sentiment retrieval of positive and negative views according to a given topic without focusing on the relationships that exist between the topic and sentiments involved.

Another related area of research is that of predicting the outcomes of an event like an election by analysing opinions retrieved from the web as input data. (Kim and Hovy, 2007) In order to do this a corpus is built from opinions retrieved from previous elections and the words and phrases obtained are used to train a classifier to predict the likely outcome of an election. (Kim and Hovy, 2007) The lexical patterns that people frequently use when they express their opinions about an election typically contain two types of opinion expressions; Judgement opinions (I like/dislike) and Predictive opinions (it is likely/unlikely to happen). (Kim and Hovy, 2007)

Subjectivity Detection and Opinion Identification

Much of the early research in subjectivity detection and opinion identification has been dominated by the work of Wiebe and colleagues (Wiebe et al., 1999, Wiebe, 2000, Wiebe et al., 2001, Hatzivassiloglou and Wiebe, 2000) detecting sentiment at document, sentence and phrase levels.

Subjectivity detection, also known as subjectivity tagging, refers to the process of separating sentences that convey beliefs or opinions from those that set out to present the facts.(Wiebe et al., 2001, Wiebe, 2000) A system that is capable of distinguishing between subjective (opinionated) sentences and objective (factual) sentences provides considerable benefits to users who wish to be presented with the beliefs or opinions of the author rather than just being presented with the facts. (Wiebe, 2000) It would also be useful when searching for documents in a search engine if a user could search for subjective opinions rather than on the subject matter alone. (Wiebe et al., 1999) Wiebe et al. (2001) identifies two different types of subjectivity; (i) Evaluation which is made up of evaluations, judgements and opinons and (ii) Speculation which includes words that evoke uncertainty such as 'maybe', 'probably'. In (Wiebe, 2000) the author argues that there are three types of subjectivity in text; Positive evaluation, negative evaluation and speculation.

The majority of the research that I have mentioned so far has been based on the assumption that subjective language exists or that the document being analysed contains mostly opinionated language. However, many natural language processing applications would greatly benefit just from being able to automatically detect opinions and distinguish them from fact, especially in news reporting and on internet forums where various people's opinions are being expressed and also for text categorisation, question answering, information extraction and summarisation problems. (Riloff and Wiebe, 2003, Wiebe, 2000, Riloff et al., 2003, Wiebe et al., 1999, Wiebe et al., 2004, Wiebe and Riloff, 2005, Wiebe et al., 2001, Breck et al., 2007, Hatzivassiloglou and Wiebe, 2000) Another useful application is the automatic detection and extraction of opinionated words in order to recognise opinion-bearing sentences (Kim and Hovy, 2005a) and the automatic detection of opinion expressions in text. (Breck et al., 2007) Subjectivity in text refers to an aspect of a sentence that expresses some form of perception, opinion, thought, evaluation or attitude that may be aimed towards an entity. (Wiebe, 1994, Wiebe et al., 2001, Wiebe et al., 2004, Wiebe, 2000)

An automatic system for performing subjectivity tagging was presented by Wiebe et al. (1999) that achieved an average accuracy of 21 percentage points higher than the baseline using simple features in a Naive Bayes classifier such as the presence in a sentence of a pronoun, an adjective, a cardinal number, modal verbs, adverbs, punctuation and sentence position. A later analysis of the system showed that the adjective feature was important for achieving significant improvements over the baseline system. A closely related study involved using a technique for clustering words according to distibutional similarity, the results of which were previously calculated in (Wiebe et al., 1999) and adding the effects of additional lexical semantic features of adjective polarity and gradability (Hatzivassiloglou and Wiebe, 2000) using the presence of adjectives in a sentence as a baseline for comparison.(Wiebe, 2000)

Effects of adjectives on sentence subjectivity

I had initially decided to dedicate a separate section to the research done on effects of adjectives on subjective words and phrases, but having noticed that most of the research conducted on phrase, sentence and document level subjectivity stems from the early work on adjectives, I decided to discuss it here.

Early work on this topic focuses on the issue of how constraints from conjunctions of adjectives affect their positive or negative semantic orientations. (Hatzivassiloglou and McKeown, 1997) Conjunctions between adjectives can provide indirect information about the orientation of adjectives, and it is this indirect information that Hatzivassiloglou and McKeown (1997) utilise to predict the semantic orientation of conjoined adjectives. Conjunctions such as 'and' placed between two adjectives usually means the adjectives are of the same polarity, whereas when the word 'but' appears between two adjectives this would suggest that they are of opposite polarity. (Hatzivassiloglou and McKeown, 1997) Constraints are collected from a large corpus, a log linear regression model determines if the two adjectives are of the same or different orientations, a clustering algorithm separates the adjectives into different groups of orientations and then the adjectives are labeled as positive or negative. (Hatzivassiloglou and McKeown, 1997)

Hativassiloglou and Wiebe (2000) investigate the effects of adjective polarity and gradability on sentence subjectivity with the objective of being able to predict whether a sentence is subjective or not judging from the adjectives that appear in it. The basis behind this research is that since the mere presence of adjectives in a sentence is useful in predicting whether or not a sentence is subjective, an investigation into additional lexical features of adjectives like polarity and gradability would be beneficial in improving the accuracy of subjectivity detection. Methods for automatically separating oriented adjectives into positive and negative classes (Hatzivassiloglou and McKeown, 1997) and a method for automatically learning and extracting gradable adjectives using a large corpus are presented, followed by an analysis of the effects that the polarity and gradabiltity of adjectives have on the ability to detect sentence level subjectivity. (Hatzivassiloglou and Wiebe, 2000) Gradability "characterises a word's ability to express a property in varying degrees." (Hatzivassiloglou and Wiebe, 2000) Gradable words are modifying expressions that act as intensifiers or diminishers. Examples of gradable words are; little, exceedingly, somewhat and very. An adjectives 'gradability' is its ability to accept such modifying expressions.(Hatzivassiloglou and Wiebe, 2000, Wiebe, 2000) The results show that the predictability of the automatically extracted sets of polarity and gradability is better than or at least as predictable as the manually determined sets.

Document-level Subjectivity Analysis

Subjectivity detection or tagging is also carried out at the document level which is useful when it is important to know whether a document contains mostly opinionated language or mostly objective langauge. E.g. when trying to distinguish between an editorial article and a regular news article. In order to recognise opinionated documents, it is important to learn good subjective clues from corpora to enable the development of effective natural language processors. (Wiebe et al., 2001) Wiebe et al. (2001) presents a method for "automatically identifying collocational clues of subjectivity in texts." The system identifies collocations made up of a fixed sequence of words called stems, which make up a subjective expression when they are put together. (Wiebe et al., 2001) This method is useful for identifying subjectivity at the document level especially for recognising editorials in news reports and for filtering results of search engine queries. (Wiebe et al., 2001)

Sentence-level and Phrase-level Subjectivity Detection

Detecting subjectivity at the document-level can be useful, for example, when trying to distinguish between subjective texts (e.g. reviews, editorials) and objective texts such as newspaper articles. However, in reality most documents on the internet and beyond are made up of both subjective and objective sentences and to label an entire document as being subjective or objective is an innaccurate assumption. (Riloff and Wiebe, 2003, Riloff et al., 2005) For example editorial articles usually contain statements of fact in order to back up their arguments and news articles will often contain statements of opinion from a source related to the topic. (Riloff and Wiebe, 2003, Riloff et al., 2005) Wiebe et al. (2001) found following an analysis of editorial and fact-based news articles, that 44% of sentences in the fact-based news articles were subjective while only 56% of sentences were objective. An important step towards building subjective information extraction systems is the task of identifying words and phrases that express opinions in text. (Breck et al., 2007) Much of the research on sentence-level subjectivity detection has focused on identifying, extracting and learning subjective clues such as adjectives from corpora. (Hatzivassiloglou and McKeown, 1997, Hatzivassiloglou and Wiebe, 2000, Wiebe et al., 2004, Wiebe et al., 1999, Wiebe, 2000) Other research conducted in this area involves creating subjectivity classifiers by learning subjective nouns by bootstrapping algorithms (Riloff et al., 2003) and using an extraction pattern learning algorithm to automatically generate patterns that represent subjective expressions in text. (Riloff and Wiebe, 2003) The goal is to be able to automatically distinguish between subjective and objective sentences in text. (Riloff and Wiebe, 2003, Riloff et al., 2003)

Related work in the field involves an approach for identifying opinion expressions that uses conditional random fields and is evaluated at the expression-level. (Breck et al., 2007) There are two types of opinion expressions; Direct subjective expressions are "spans of text that explicitly express an attitude or opinion." (Breck et al., 2007) E.g. 'Criticised' They are also referred to as individual expressions of subjectivity. (Wiebe, 2000) Expressive subjective elements are described as spans of text that evoke a degree of subjectivity on the part of the speaker. (Breck et al., 2007) Wiebe (2000) describes the presence of potential subjective elements that may be used to express subjectivity. The system uses a machine-learning technique for the identification of the two types of opinion expressions mentioned above using lexical, syntactic and dictionary-based (online databases of words) features.

In (Kim and Hovy, 2005a) a method for automatically detecting opinion bearing words and sentences is put forward. They come up with a method for obtaining opinion-bearing words and non opinion-bearing words and then use them to recognise opinion-bearing sentences. (Kim and Hovy, 2005a) Using only a small set of previously annotated data, they can find synonyms and antomyms (word opposites) of an opinion-bearing word through automatic expansion in WordNet (an online lexical database of words) and then use them as feature sets of a classifier. (Kim and Hovy, 2005a) Detection of opinion sentences is also explored in (Wiebe and Riloff, 2005) where they develop subjective classifiers using only unannotated data for training. Wiebe and Riloff (2005) are also concerned with learning new objective clues and creating better performing objective classifiers.

Developing Question Answering Systems

Recent research in the area has focused primarily on developing complex question answering systems by identifying opinion holders, the topic of the opinion and the actual sentiment being expressed towards the subject or topic. (Kim and Hovy, 2005a, Kim and Hovy, 2005b, Choi et al., 2006, Yu and Hatzivassiloglou, 2003, Bethard et al., 2004, Breck et al., 2007, Choi et al., 2005, Kim and Hovy, 2004) Automatically identifying the holder of a belief or an opinion is extremely important for complex question answering systems to be able to answer questions about the relationship between two entities. (Choi et al., 2005, Choi et al., 2006, Bethard et al., 2004, Kim and Hovy, 2004, Kim and Hovy, 2005b)

Various approaches have been put forward to identify opinion holders in text. Kim and Hovy (2004, 2005a) identify four important elements of an opinionated sentence; the opinion holder or source, the opinion or belief itself, the subject or target of the belief and finally the opinion valence ie. The semantic orientation of the opinion (positive, negative or neutral). For their analysis they leave out sentences that have a neutral valence as these sentences are usually not that opinionated. Eg. 'I believe that there was one hundred people there'. In order to simplify the task, Kim and Hovy (2005a) build a classifier that automatically identifies all the sentences that express a valence in a given piece of text. In order to help them achieve this, collections of verbs and adjectives are built that are strong indicators of opinion. A system for identifying the opinion holder and the sentiment polarity of the opinions expressed about a given topic is introduced using a set of seed words taken from Wordnet and sorting them according to their negative or positive polarity. (Kim and Hovy, 2004) Furthermore, a system for identifying the topic, opinion holder, and the opinion expressed is proposed using semantic role labeling which attaches an appropriate label to key elements of the sentence in order to identify the role (holder, topic, opinion) each element plays in a given sentence. (Kim and Hovy, 2006b) This is especially important with regard to news articles. Because so many different opinions are being expressed by many different entities knowing which opinion belongs to which entity is essential in understanding the relationships between people, organisations and even countries.

Building on earlier work, Kim and Hovy (2005b) purpose an automatic system for identifying opinion holders and the strengths of those opinions. From a given sentence taken from a corpus of over 10,000 sentences and from over 500 news documents Kim and Hovy (2005b) annotate the sentences by identifying the opinion holder or holders if there is more than one possibility, the expressive subjectivity i.e. the opinion being expressed and the opinion expression which is given a rank according to the strength of the term. Choi et al (2005, 2006) also propose a system for identifying opinion holders in documents with a view to developing question answering systems based on opinions. Choi et al (2005) focus on identifying direct and indirect sources of "private states" such as the authors' opinions and feelings etc. A system for extracting opinion entities and the relations between them is put forward by analysing entities that express opinions directly (opinion expressions) as well as the expressions of speech that introduce indirect opinions or "subjective propositions".(Choi et al., 2006) Similarly, Bethard et al (2004) develop a system that automatically extracts "propositional opinions" which are defined as an opinion that is contained within the propositional argument of a verb in a sentence. An example of a propositional opinion is included by Bethard et al.:

"I believe [you have to use the system to change it]"

The words contained in the brackets make up the propositional argument or the component opinion. The opinion of the author is found within the argument of the verb 'believe'.

The research is conducted by annotating over 5,000 sentences by labelling the opinion propositions and opinion holders in each sentence. A "NULL" label is given to an element that is neither a propositional opinion or an opinion holder.

Yu and Hatzivassiloglou (2003) develop a system that aims to integrate document-level subjectivity classification with subjectivity and sentiment polarity classification at the sentence-level. It is important at this stage to point out that the majority of the research mentioned so far has mainly dealt with subjectivity and polarity classification either at the document, sentence or phrase-levels but not a combination of these. The main objective is to develop a method of classifying documents as either factual or opinionated, identify opinion bearing sentences in both types of articles and working out whether the feelings expressed in the opinionated sentences have a positive or negative semantic orientation. A large number of news articles are analysed for their subjectivity, and over 400 sentences are analysed to determine their semantic orientation.

A Naive Bayes classifier is presented to separate opinionated from factual news articles that have pre-assigned opinion and fact labels at the document-level. The labels are only used to provide the correct classification labels during training and evaluation. For the problem of subjectivity classification at the sentence-level three different approaches are used; A similarity approach, a Naive Bayes classifier and multiple Naive Bayes classifiers. The similarity approach uses a similarity measuring and clustering tool called SIMFINDER (Hatzivassiloglou et al., 2001) that converts pieces of text into related text units that can be further reduced to measure sentence similarity based on shared words, phrases and WordNet data. (Hatzivassiloglou et al., 2001, Yu and Hatzivassiloglou, 2003) The hypothesis behind this approach is that given a topic "opinion sentences will be more similar to other opinion sentences than to factual sentences." (Yu and Hatzivassiloglou, 2003) The second approach trains a Naive Bayes classifier using the sentences in both the factual and opinion documents and includes words, bigrams, trigrams and parts of speech in sentences as features to indicate subjectivity as well as the presence of positively or negatively oriented words. (Yu and Hatzivassiloglou, 2003) Thirdly, the use of multiple Naive Bayes classifiers is then used to more accurately classify the opinionated sentences by reducing the training set to those that are more likely to be correctly labelled. Once the opinionated sentences are identified from the documents, the next process is to determine their polarity. The sentences are put into three classes; positive, negative and neutral, based on the number and strength of the opinionated words in a sentence. Next, the semantic orientation of the words needs to be measured by comparing them with words taken from a known seed set of semantically oriented words. (Yu and Hatzivassiloglou, 2003) Finally, the semantic orientation of the opinion sentences is established