Detecting Human Interaction Patterns Psychology Essay

Published: Last Edited:

This essay has been submitted by a student. This is not an example of the work written by our professional essay writers.

Mining Human Interaction in Meetings is useful to identify how a person reacts in different situations. Behavior represents the nature of the person and mining helps to analyze, how the person exhibits his/her opinion. Detecting semantic knowledge is significant. Meeting interactions are categorized as propose, comment, acknowledgement, request-information, ask-opinion, post-opinion and negative opinion. The sequence of human interactions is represented as a Tree. Tree structure is used to capture how the person interacts in Meetings and to discover the interaction flow often occurs in a Meetings and to reveal the relationship exist among the interactions. Tree pattern mining and sub tree pattern mining algorithms are automated to analyze the structure and to extract interaction flow patterns. The extracted patterns are interpreted from human interactions. The frequent patterns are used as an indexing tool to access a particular semantics. Frequent interaction flow helps to assume the probability of another type of interaction. The frequent interaction patterns are clustered and the behavior of the person is determined.

Index Terms - Human interaction, interaction flow, frequent interaction pattern, tree-based mining, clustering.


Human Interaction is a vital event to understand communicative information. Understanding human behavior is essential in applications including automated surveillance, video archival/retrieval, medical diagnosis, and human-computer interaction. The advent of smart meeting that automatically records a meeting and analyzes the generated audio-visual content for future viewing .While most of current smart meeting systems analyze the meeting content for understanding what conclusion was made, it is more interesting and important to know how a conclusion was made, for example, did all members agree on the outcome? Who did not give his opinion? Who spoke a little or a lot? etc., such kind of group social dynamics can be useful for determining whether meeting was well organized and whether the conclusion was rational. Human interaction plays an important role in understanding this communicative information and different from physical interactions (e.g. turn-taking and addressing), the human interactions here are defined as behaviors among meeting participants with respect to the current topic, such as proposing an idea, giving some comments, expressing positive opinion, and requesting information. When incorporated with semantics (i.e. user intention or attitude towards a topic), interactions are more meaningful in understanding conclusion drawing and meeting organization. The interaction issues including turn-taking, gaze behavior, influence and talkativeness and analyzing user interactions during poster presentation in an exhibition room are mainly focus on detecting physical interactions between participants without any relations with topics.

The context information is gathered through multiple sensors (e.g. video cameras, microphones, and motion sensors).The various interactions imply different user roles, attitudes, and intentions about a topic during a discussion. The interaction type definition naturally varies according to its usage. We create a set of human interactions that includes seven categories: propose, comment, acknowledgement, requestInfo, askOpinion, posOpinion, and negOpinion. The detailed meanings are described as: propose - a user proposes an idea with respect to a topic; comment - a user gives comments on a proposal; acknowledgement - a user confirms someone else's comment or explanation, e.g. "yeah", "uh huh" and "OK"; requestInfo - a user requests information about a proposal; askOpinion - a user asks someone else's opinion about a proposal; posOpinion - a user expresses positive opinion, i.e. follow a proposal; and negOpinion - a user expresses negative opinion, i.e. against a proposal.

The context used in our interaction detection includes head gesture, attention from others, speech tone, speaking time, interaction occasion, and information about previous interaction. Head gesture (e.g. nodding) is very common and used often in detection of human response (acknowledgement or agreement). For example, when a user is proposing some idea, he is usually being looked at by most of the participants. Attention from others can be treated as how many persons looking at the target user during the interaction. Thus the problem can be roughly turned into detection of face orientation. The face orientation is determined as the one whose vector makes the smallest angle. Speech tone refers to whether a statement is a question or a normal one. Speaking time is another important indicator in detection the type of human interaction. When a user puts forward a proposal, it usually takes relatively long time. But it takes short time when he gives an acknowledgement or asks a question. The interaction occasion has two values: spontaneous and reactive. The former means the interaction is initiated by the person spontaneously (e.g. proposing an idea or asking a question). The latter denotes the interaction is triggered as response to another interaction. It is intuitive that there are certain patterns or flows frequently appear in meeting discussion. For instance propose and request Info are usually followed by the interaction of comment.


There have been several works done in discovering Human behavior patterns by using stochastic techniques. Bakeman and Gottman [2] applied sequential analysis to observe and analyze human interactions. Magnusson [3] proposed a pattern detection method, called T-pattern to discover hidden time patterns in human behavior. T-pattern has been adopted in several applications such as interaction analysis and sports research .Although the purpose of these techniques is similar to our work, we conduct analysis on human interaction in meetings and address the problem of discovering interaction patterns from the perspective of data mining.

Casas-Garriga[3] proposed algorithms to mine unbounded episodes (those with unfixed window width or interval) from a sequence of events on a time line. The work is generally used to extract frequent episodes, i.e., collections of events occurring frequently together. Morita et al. [4] proposed a pattern mining method for the interpretation of human interactions in a poster exhibition. It extracts simultaneously occurring patterns of primitive actions such as gaze and speech. Sawamoto et al. [5] presented a method for extracting important interaction patterns in medical interviews (i.e., doctor-patient communication) using non-verbal information.


We were extracting appropriate scenes from the viewpoints of individual users by clustering events having spatial and temporal relationships. Scene videos were created in a linear time fashion using only one source of video at a time. Root denoting the person who organizes the interaction. From the Figure 1. after the Root, person 4 and Person 2 are initiating the new statement with propose. Based on the person4's comment person3, person1, person2 were exhibits their comment. Tree Hierarchy represents the flow in which the person represents their comments.

Figure1: Interaction Flow Hierarchy

The comment may be of any interaction type namely PRO, ACK, COM, POS, and NEG.

Figure2: Interaction Tree Hierarchy


Based on the interaction defined and recognized, we now describe the notion of interaction flow and its construction. An interaction flow is a list of all interactions in a discussion session with triggering relationship between them. We first give the definition of a session in a meeting discussion. A session is a unit of a meeting that begins with a spontaneous interaction and concludes with an interaction that is not followed by any reactive interactions.

Here, spontaneous interactions are those that are initiated by a person spontaneously and reactive interactions are triggered in response to another interaction. For instance,

propose and askOpinion are usually spontaneous interactions, while acknowledgement is always a reactive interaction. Whether an interaction is spontaneous or reactive is not determined by its type (e.g., propose, askOpinion, or acknowl-edgement ), but labeled by the annotator manually. Hence, a session contains at least one interaction (i.e., a spontaneous interaction). A meeting discussion consists of a sequence of sessions, in which participants discuss topics continuously.

The label represents interactions in the meeting. Labels in the interaction tree are not sorted, because the edges reflect temporal relationship between the siblings. Hence, sorting,e.g., alphabetically, would likely break this relationship.For efficient processing, we use a string coding method for the interaction trees.


With the representation model and annotated interaction flows, we generate a tree for each interaction flow and thus build a tree data set. Patterns are frequent trees or sub trees in the tree database. Support is with given a tree or sub tree T and a data set of trees TD.

Number of occurrences of T

Support =

Total No of Trees in TD

If the value of supp(T) is more than a threshold value Minimum support T is called a frequent tree or frequent sub tree. We have a data set of interaction trees TD. Given a minimum support σ, we would like to find all trees and sub trees that appear at least σ Ã- | TD | times in the data set.


Tree Database TD and Threshold value is given as an Input

Generate its full set of isomorphic trees ITD, from the Database TD

Count the number of occurrences for each tree t in the

Database ITD.

(3) Calculate the support of each tree

(4) Select the trees whose supports are larger than σ and

Detect isomorphic trees.

(4.1) if m trees are isomorphic select one of them

and discard the others.

(5) Output the frequent trees.



A tree database TD and a support threshold σ is given as a input.

(1) i := 0

(2) Calculate the support of each node from the database TD.

(3) Select the nodes whose supports are larger than σ to

Form F1

(4) i = i + 1

(5) for each tree ti in Fi, do

(5.1) for each node t1 in F1, do

(5.1.1) join ti and t1 to generate Ci+1

(6) calculate the support of each tree in Ci+1

(7) if there are any trees whose supports are larger than σ

(7.1) select them to form Fi+1 and return to Step (4)

(7.2) Else output the frequent sub trees whose supports

are larger than σ


(1) count:= 0

(2) supp(st):= 0

(3) for each tree t €TD do

(3.1) Create subtrees S of t with any item s €S, |s|=|st|

(3.1.1)flag:= false

(3.1.2) for each item s €S do

(7) generate isomorphic trees IS of s

(8) for each item is € IS do

(9) if tsc(is)=tsc(st) then

(9.1)count:= count +1

flag:= true


(10) if flag=true then


(11) supp(st)= count=/|TD|

(12) return supp(st)


Our work involves real meetings lasting 15 minutes on average. Video camera was used for capturing the meetings. Each meeting had four participants seated around a table. In order to use a correct data for mining, we tuned the interaction types manually after applying the recognition method. The goal of our wok is to discover frequent interaction trees and analyze the behavior of the algorithms on the data set, focusing on the effect of threshold.

Table 1: Sub tree Patterns

Rule No

Association rule















If we set the threshold value as 0.122, frequent sub trees cannot be identified, because threshold value is less than the support value. If we set the threshold value as 0.135, frequent sub trees is identified. Because threshold value is less then support value. If we set threshold value as 0.278, frequent sub tree cannot be displayed, because threshold value is greater than the support value. Frequent trees were identified which satisfies the threshold value. In our work all the sub trees






(1) (2)







Figure 3: Top three ranked frequent trees


In our work all the sub trees were found as frequent trees, because threshold value is greater than the support value. Each person interaction in the meeting is calculated as 1 for COM, ACK, POS, PRO, and NEG. The person's interaction ratio is shown in the following table,

Table 2: Interaction type measures

No.of Persons




Person 2

Person 3

Person 4


























In the above Table specifies the interaction measures of each person. Each Interaction type is taken as 1, if it arises in the meeting with regard of each person.

Figure 4: Interaction measure of distinct person

Using this evaluation we can able to find each person's interest level in participating the meetings. The Interaction measure of distinct persons explores how far one person involves in meetings in-depth. So that we can evaluate the percentage of interest level in propose, acknowledge, comment, positive opinion and negative opinion.

Persons are clustered based on their interaction percentage level using k-means clustering. For each person's COM, ACK, POS, PRO, NEG are evaluated and clustered. Each cluster specifies different characteristics of a people. From the cluster analysis behavior of the person is identified to some extent. If one person's PRO level is high in all meetings then that person will typically have passion in proposing new ideas in the enhancement of the organization. If one person's ACK level is high in all meetings then that person will typically have passion in exposing opinion about each person's comment and encouraging others. Likewise each person's interaction type percentage was analyzed.


We have proposed an Interaction based tree mining method for discovering frequent interaction. From the Interaction based clustering analysis we have evaluated the person's behavior. As future work, we have planned to integrate more contexts like lexical cues in the detection process in order to improve the recognition accuracy. We also plan to design a visualization system for reviewing the human interactions.


[1] Zhiyong Yu, Xing she Zhou, Christ ian Becker, and Yuichi Nakamura," Tree-Based M ining f or Discovering Patterns of Human Interaction in Meetings ",IEEE Transac tions on knowledge and data engineering, vol. 24, no. 4, april 2012.

[2] R. Bakeman and J.M. Gottman, "Observing Interaction: "An Introduction to Sequential Analysis", Cambridge Univ. Press, 1997.

[3] M.S. Magnusson, "Discovering Hidden Time Patterns in Behavior:T-Patterns and Their Detection," Behavior Research Methods, Instruments and Computers, vol. 32, no. 1, pp. 93-110, 2000.

[4] G. Casas-Garriga, "Discovering Unbounded Episodes in Sequential Data," Proc. European Conf. Principles and Practice of Knowledge Discovery in Databases (PKDD '03), pp. 83-94, 2003.

[5] T. Morita, Y. Hirano, Y. Sumi, S. Kajita, and K. Mase, "A Pattern Mining Method for Interpretation of Interaction," Proc. Int'l Conf.Multimodal Interfaces (ICMI '05), pp. 267-273, 2005.

[6] Y. Sawamoto, Y. Koyama, Y. Hirano, S. Kajita, K. Mase, K.Katsuyama, K. Yamauchi, "Extraction of Important Interactions in Medical Interviews Using Nonverbal Information," Proc. Int'l Conf. Multimodal Interfaces (ICMI '07), pp. 82-85, 2007.

[7] J. Yang, W. Wang, P. S. Yu, and J. Han. "Mining long sequential patterns in a noisy environment". In Proc. of SIGMOD conf., pages 406-417, 2002.

[8] Joseph N. Cappella,"The Biological Origins of Automated

Patterns of Human Interaction", University of Pennsylvania Year 1991.

[9] Palivela Hemant,Prashanth G ,Vijay Kumar S ,Kalpana Patil,"Discovering Patterns in Interactions between Humans and Animals by Using Tree Based Mining" International Journal of Engineering Research & Technology (IJERT),Vol. 1 Issue 6, August - 2012.

[10] R. C. Schank and R. P. Abelson." Scripts, plans, goals, and understanding : an inquiry into human knowledge structures". John Wiley and Sons Inc., Hillsdale, N.J., 1977.

[11 ]Z.W. Yu, Z.Y. Yu, Y. Ko, X. Zhou, and Y. Nakamura,

"Inferring Human Interactions in Meetings:A Multimodal Approach," Proc.Sixth Int'l Conf. Ubiquitous Intelligence and Computing (UIC '09),pp. 14-24, July 2009.