Quasi Framework: A New Student Disengagement Detection In Online Learning

Quasi Framework: A New Student Disengagement Detection In Online Learning

P. V. Praveen Sundar Research Scholar,

Hindusthan College of Arts &Science, Coimbatore-641028.

Abstract

Educational Data Mining (EDM) has emerged as independent research area in recent years. Moreover student learning environment also rapidly move towards online. Compare to traditional teaching method, online tutoring will attracts younger generations. However student engagement is an important aspect of effective learning. Most of the students performed well in their academic performance and spent more time with internet too. Thus measuring disengagement is likely to help poor performance students. In this paper we propose a new framework called Quasi Framework, which is trying to measure the significant relationship between disengagement level and their academic achievement.

Keywords: Disengagement Detection, Online Learning, EDM, Student Performance Prediction

1. INTRODUCTION

   Educational Data Mining (EDM) is a field that exploits statistical, machine-learning, and data-mining (DM) algorithms over the different types of educational data. Its main objective is to analyze these types of data in order to resolve educational research issues. EDM is concerned with developing methods to explore the unique types of data in educational settings and, using these methods, to better understand students and the settings in which they learn.

   EDM is concerned with developing methods to explore the unique types of data that come from an educational context and, using these methods, to better understand students and the settings in which they learn[16]. From a technical perspective, EDM is the application of data mining techniques to educational data, and so, its objective is to analyze this type of data in order to resolve educational research issues and understand the setting in which students learn [3], [4].

   Educational software strives to meet the learners needs and preferences in order to make learning more efficient; the complexity is considerable and many aspects are taken into consideration. However, most systems do not consider the learners motivation for tailoring teaching strategies and content, despite its great impact on learning being generally acknowledged. A lack of motivation is clearly correlated with learning rate decrease (e.g., [2]). A number of attempts have been undertaken to accommodate the learners motivational states, mostly by means of design. E- learning systems attempted to motivate students through an attractive design by using multimedia materials or including game features that have great potential [7] and have been proved successful in a number of cases (e.g., [6]).

   Several efforts to detect motivational aspects from learners actions are reported in the literature [2], [8],

   [5], [1], [14], [9], [19], [20]. However, all these efforts are concentrated on Intelligent Tutoring Systems or problem solving environments. As online content- delivery systems are increasingly used in formal education, there is a need to extend this research to encompass this type of systems as well. The interaction in these systems is less constrained and structured compared with problem-solving environments, posing several difficulties to an automatic analysis of learners activity.

   The learners actions preserved in log files have been relatively recently discovered as a valuable source of information and several approaches to motivation detection and intervention have used log-file analysis. An important advantage of log-file analysis over self- assessment approaches is the unobtrusiveness of the assessment process, similar to the classroom situation where a teacher observes that a learner is not motivated without interrupting his/her activities.

   To address this challenge, we restricted our research to one motivational aspect, disengagement, and looked at identifying the relevant information from learners actions to be used for its prediction. Being able to automatically detect disengaged learners would offer the opportunity to make online learning more efficient, enabling tutors and systems to target disengaged learners, to reengage them, and thus, to reduce attrition. The motivational based disengagement detection system is still in infancy stage with respect to assessing the learners attitude and characteristics. Thus we propose a new framework for disengagement detection to enhance the value of existing prediction system.

   Analyzing data from log-file is an efficient method for automatic analysis, whereas it has certain level of fuzziness in order to retrieve desired information in robust fashion. Thus, we introduce metadata as quasi assessment technique to obtain the result. Further, disengagement is correlated with academic achievement to ensure the quality of assessment. The remaining part of the paper is structured as follows: in Section 2, previous work related motivation and engagement prediction is presented. Section 3 briefly presents the present disengagement detection techniques and their

   pattern of predicting results. Section 4 describes our proposed framework and its implications referred with previous approaches. Section 5 discuss about the implementation feasibility, possible impact on educational space and conclude the paper.

2. RELATED RESEARCH

   This section describes review of related literatures with respect to motivational based engagement and disengagement concepts.

   Motivational research [13] makes uses of several concepts, besides motivation itself: engagement, interest, effort, focus of attention, self-efficacy, confidence, etc. The research presented in this paper focuses on engagement, or rather on disengagement, as an undesirable motivation state. For our purposes, a student is considered to be engaged if she/he is focused on the current learning activity and disengaged otherwise.

   A dynamic mixture model combining a hidden Markov model with Item Response Theory was proposed in [9]. The dynamic mixture model takes into account: student proficiency, motivation, evidence of motivation, and a students response to a problem. The motivation variable can have three values: 1) motivated; 2) unmotivated and exhausting all the hints in order to reach the final one that gives the correct answer: unmotivated-hint; and 3) unmotivated and quickly guessing answers to find the correct answer: unmotivated guess.

   A Bayesian Network has been developed [1] from log data in order to infer variables related to learning and attitudes toward the tutor and the system. The log data registered variables like problem-solving time, mistakes, and help requests.

   A latent response model [2] was proposed for identifying the students that game the system. Using a pretest-posttest approach, the gaming behavior was classified in two categories: 1) with no impact on learning and 2) with decrease in learning gain. The variables used in the model were: students actions and probabilistic information about the students prior skills.

   The same problem of gaming behavior was addressed in [19], an approach that combines classroom observations with logged actions in order to detect gaming behavior manifested by guessing and checking or hint/help abuse. In order to prevent this gaming behavior, two active interventions (one for each type of gaming behavior) and a passive strategy have been proposed [20]. When a student was detected to manifest one of the two gaming behaviors, a message was displayed to the student encouraging him/her to try harder, ask the teacher for help or pursue other suitale actions. The passive strategy had no triggering mechanism, but merely provided visual feedback on students actions and progress.

   Disengagement can be attained via two roads [10], [15], [17], [18]. The first pathway is the devaluation of the domain, so that the outcomes or feedback received are no longer viewed as relevant or important to how a person defines his/her self [17]. In the academic domain, it occurs when students decrease the importance of academic achievement to the point where they no longer view it as a self-relevant domain [15]. For example, students who devalue the academic domain could say that being good at school is not an important part of who they are or that it is not important for their future lives. Second, the individual may discount the validity of the feedback or evaluation and thus reject it as a true indicator of his/her competencies in the domain [10]. In the academic setting, this process of discounting is illustrated by the rejection of academic feedback, whereby students decrease the importance of grades received by considering them as biased indicators of their ability [15]. In this case, students could say that the grades they obtained did not provide a valid evaluation of their achievement level or did not correctly reflect their academic abilities. Discounting is considered to be a less radical path to disengagement, because it has the advantage of protecting self-esteem without reducing the value of socially important domains [12].

   As [12] emphasize, psychological disengagement may sometimes be activated situationally in response to the threat of evaluation in a specific situation. Whereas chronic disengagement could ultimately lead to drop-out

   from academic enrolment ([10], [15], [17]), situational disengagement may be useful in dampening the psychological severity of feedback in a particular situation without necessarily having detrimental effects on underlying motivation, and could even facilitate persistence in the activity [12]. Thus, in a specific context, it may be possible to continue to value achievement in the domain, while being relatively disengaged from a particular evaluation.

   Existing studies have emphasized that, in response to a specific threatening academic situation, such as a poor grade or negative feedback, students may temporarily disengage their self-esteem from performance feedback [10], [12]. Thus, when confronted with a specific academic evaluative situation, students are more likely to discount the feedback received than devalue the academic domain [12]. As [12] suggest, discounting is a situational response that enables persistence despite negative feedback. However, existing studies have mainly focused on the consequences of situational disengagement and little is known about the predictors of both discounting and devaluing in specific threatening situations.

   Online disengagement [11] detection investigates the extendibility of our approach to other systems by studying the relevance of these attributes for predicting disengagement in a different e-learning system. To this end, two validation studies were conducted indicating that the previously identified attributes are pertinent for disengagement prediction, and two new meta-attributes derived from log-data observations improve prediction and may potentially be used for automatic log-file annotation.

3. DISENGAGEMENT DETECTION

   Several concepts are used in motivational research, besides motivation itself: engagement, interest, effort, focus of attention, self-efficacy, confidence etc. For the results presented in this paper the focus of our research on motivation is on engagement. A student is engaged if he/she is focused on the learning activity. A number of concepts in motivational research such as interest, effort, focus of attention and motivation are related though not identical to engagement: 1) engagement can

   be influenced by interest, as people tend to be more engaged in activities they are interested in; thus, interest is a determinant of engagement; 2) effort is closely related to interest in the same way: more effort is invested if the person has interest in the activity; the relation between engagement and effort can be resumed by: engagement can be present with or without effort; if the activity is pleasant (and/or easy), engagement is possible without effort; in the case of more unpleasant (and/or difficult) activities, effort might be required to stay engaged; 3) the difference between engagement and focus of attention, as it is used in research is that focus of attention refers to attention through a specific sensorial channel (e.g. visual focus), while engagement refers to the entire mental activity (involving in the same time perception, attention, reasoning, volition and emotions); 4) in relation to motivation, engagement is just one aspect indicating that, for a reason or another, the person is motivated to do the activity he/she is engaged in, or the other way, if the person is disengaged, he/she may not motivated to do the activity; in other words, engagement is an indicator of motivation.

   Among different studies accomplished on disengagement detection, this paper is proposing an enhancement of Cocea et al. [11] study. She had conducted two validation studies on iHelp data indicate that the attributes identified in the studies on HTML- Tutor data are relevant for the new system as well. Paired t-tests were used to investigate the statistical significance of the differences in the distribution of accuracy and true positive rates across the eighth methods between the two studies on iHelp data, on the one hand, and between the second iHelp study and the HTML-Tutor study, on the other hand. The mean for each data set and the significance of the t-test are considered. All accuracy and True-Positive (TP) rates on all data sets were tested and proved to follow a normal distribution.

   When comparing the results of two iHelp studies, we can see that the difference is statistically significant with one exception, i.e., the difference between the accuracy distribution for the data sets with sequences of only 10

   minutes (DS1_S1 and DS1_S2). The amount of data and the new score attribute did not contribute to better predictions.

   To assess the contribution to prediction of the attributes in each system, three attribute evaluation methods with ranking as search method for attribute selection were used: chi-square, information gain, and OneR [21]. For HTMLTutor, according to chi-square and information gain ranking, the most valuable attribute is average time spent on pages, followed by the number of pages, number of tests, average time spent on tests, number of correctly answered tests, and number of incorrectly answered tests. OneR ranking differs only in the position of the last two attributes: number of incorrectly answered tests comes before number of correctly answered tests. The attribute ranking using information gain filter for iHelp attributes delivered the following ranking: NoPpP, NoPages, AvgTimeP, NoPpM, AvgTimeQ, Score, and NoQuestions. Chi-square evaluator produces the same ranking, except that the positions of the last two attributes are reversed, i.e., NoQuestions contributes a higher gain than Score. OneR evaluator produces a different ranking compared to the other two, even if the main trend is preserved (attributes related to reading come before the ones for quizzes): NoPpP, AvgTimeP, NoPages, NoPpM, NoQuestions, AvgTimeQ, and Score.

   The attribute ranking results show that for both HTMLTutor and iHelp, the attributes related to reading are more important than the ones related to tests. The iHelp score attribute and its two correspondent attributes from HTMLTutor (number of currently answered tests and number of incorrectly answered tests) are among the least important ones. This study suggested that despite the problem they may pose, knowledge about the two patterns of disengagement would be useful for a more targeted intervention and in further work; the possibilityto predict them will be investigated. The next section discuss about our new framework proposed for this study.

4. QUASI FRAMEWORK

   Existing proposal of disengagement detection has certain limitations with respect to the system considered

   for analysis. There are two different systems have been used to confirm the prediction accuracy known as HTML-Tutor and I-Help. Both systems are web based interactive learning environment, which is used to analyze the log file. There are five attributes used for analysis such as number of pages read, average time spent on reading, number of questions attended, average time spent on quizzes and total time of a sequence.

   The objective of this study has two folds; the first objective would like to prove amount of time spent on pages and their quiz performances are not correlated by most of the time. Second objective is to evaluate the prior history of student performance on reading and quiz.

   Cocea et al. [11] proposal will be very suitable for predicting generic disengagement prediction, whereas for disengagement to engagement model should consider continuous evaluation method. Hence predicting the personalized disengagement pattern helps to cluster the students.

   Consistency

   Student

   Internet

   Geographical Region

   Learning Attitude

   Performance Score

   Prior History of Performance

   Disengagement Guidelines

   Correlation

   Disengagement Detection

   Disengagement Decision

   Figure 1: Quasi Framework Disengagement Detection

   A generic student disengagement prediction will not suitable for all scenarios, since ability and characteristics of study may differ from one another. Some students have the ability to understand the concept within short period of time and perform well in their examinations and some of them may not follow the same structure.

   Fig. 1 depicts our students disengagement detection model. As per the general online learning environment student enroll it in a course. Further online tutoring option will be provided to learn the specific subject and this system generates log file, which observe all the required attributes such as time spent on specific concept, page, navigation pattern, number of times logged in to a system. Similarly another system contains only online examination towards the specific subject proposed for him. The entire script should be written in log-file. Quasi framework is suggested to use the meta- data format for student observations. Priory academic history will be separately maintained for effective assessment. In order to predict the disengagement, statistical correlation is suggested. Final student disengagement prediction can be obtained through geographical disengagement guidelines, correlation measures, and performance consistency score.

5. CONCLUSION

Disengagement detection is a supportive mechanism to keep the students engaged in their academic activities. So many online tutoring models have been proposed to maximize the chance of engagement. Since online tutoring models are in growing interest, student engagement and their performance should be maintained as better than of traditional teaching method. Quasi framework trying to predict the disengagement in an enhanced pattern compared to previous studies.

REFERENCES

1. I. Arroyo and B.P. Woolf, Inferring Learning and Attitudes from a Bayesian Network of Log File Data, Artificial Intelligence in Education, Supporting Learning through Intelligent and Socially Informed Technology, C.K. Looi et al., eds., pp. 33-34, IOS Press, 2005.

2. R. Baker, A. Corbett, and K. Koedinger, Detecting Student Misuse of Intelligent Tutoring Systems, Proc. Seventh Intl Conf. Intelligent Tutoring Systems, pp. 531-540, 2004.

3. Baker and Yacef, Proceedings of "Educational Data Mining" workshop, held in conjunction with the 8th International Conference on Intelligent Tutoring Systems, Jhongli, Taiwan, July 2006.

4. Barnes, T. CAREER: Educational Data Mining for Student Support in Interactive Learning Environments, NSF-ALT,

   $646,982, Jul 2009 Jun 2014.

5. J. Beck, Engagement Tracing: Using Response Times to Model Student Disengagement, Artificial Intelligence in Education: Supporting Learning through Intelligent and Socially Informed Technology, C. Looi et al., eds., pp. 88-95, IOS Press, 2005.

6. G.D. Chen, G.Y. Shen, K.L. Ou, and B. Liu, Promoting Motivation and Eliminating Disorientation for Web Based Courses by a Multi- User Game, Proc. World Conf. Educational Multimedia and Hypermedia and World Conf. Educational Telecomm, June 1998

7. T. Connolly and M. Stansfield, Using Games-Based eLearning Technologies in Overcoming Difficulties in Teaching Information Systems, J. Information Technology Education, vol. 5, pp. 459-476, 2006.

8. A. de Vicente and H. Pain, Informing the Detection of the Students Motivational State: An Empirical Study, Proc. Sixth Intl Conf. Intelligent Tutoring Systems, S.A. Cerri et al., eds., pp. 933-943, 2002.

9. J. Johns and B. Woolf, A Dynamic Mixture Model to Detect Student Motivation and Proficiency, Proc. 21st Natl Conf. Artificial Intelligence (AAAI-06), 2006.

10. Major & Schmader, Coping With Ethnic Stereotypes in the Academic Domain: Perceived Injustice and Psychological Disengagement, the Society for the Psychological Study of Social Issues, 1998.

11. Mihaela Cocea and Stephan Weibelzahl Disengagement Detection in Online Learning: Validation Studies and Perspectives, 2011.

12. Nussbaum & Steele, Situational disengagement and persistence in the face of adversity. Journal of Experimental Social Psychology, 43, 127134. 12.007 2007.

13. P.R. Pintrich and D.H. Schunk, Motivation in Education: Theory, Research and Applications. Prentice Hall, 2002.

14. L. Qu, N. Wang, and W.L. Johnson, Detecting the Learners Motivational States in an Interactive Learning Environment, Artificial Intelligence in Education, C.-K. Looi et al., eds., pp. 547- 554, IOS Press, 2005.

15. R'enger & Loose, Relationship of sociocultural factors and academic self-esteem to school grades and school disengagement in North African French adolescents, British Journal of Social Psychology, 45, 777797. 2006.

16. Romero, Cristobal, and Sebastian Ventura. "Educational data mining: A survey from 1995 to 2005." Expert Systems with Applications 33.1 (2007): 135-146.

17. Schmader et al, Coping with ethnic stereotypes in the academic domain: Perceived injustice and psychological disengagement. Journal of Social Issues, 57, 93111. 2001.

18. Tougas, F., Rinfret, N., Beaton, A. M., & De la Sablonniere, R. Policewomen acting in self defense: Can psychological disengagement protect self-esteem from the negative outcomes of relative deprivation, Journal of Personality and Social Psychology, 88, 790800. 2005.

19. J. Walonoski and N.T. Heffernan, Detection and Analysis of Off- Task Gaming Behavior in Intelligent Tutoring Systems, Proc. Eighth Intl Conf. Intelligent Tutoring Systems, M. Ikeda,

    K. Ashley, and T.-W. Chan, eds., pp. 382-391, 2006.

20. J. Walonoski and N.T. Heffernan, Prevention of Off-Task Gaming Behaviour within Intelligent Tutoring Systems, Proc. Eighth Intl Conf. Intelligent Tutoring Systems, M. Ikeda, K. Ashley, and T.-W. Chan, eds., pp. 722-724, 2006.

21. I.H. Witten and E. Frank, Data Mining. Practical Machine Learning Tools and Techniques, second ed. Morgan Kauffman/Elsevier, 2005.