Cognitive Task Analysis to Improve Student Competencies: a Pilot Study based on the Use of the Square of PErceived ACtion Model

Copyright © 2021, Journal Educational Verkenning, Under the license CC BY-SA 4.0 | 20 Cognitive Task Analysis to Improve Student Competencies: a Pilot Study based on the Use of the Square of PErceived ACtion Model Philippe Fauquet-Alekhine1, Patrick Martinez2 1SEBE-Lab, Department of Psychological & Behavioural Science, Houghton St., WC2A 2AE, London, United Kingdom 2Institution Saint-Louis, Saumur, France


Introduction
The SEBE-Lab at the London School of Economics & Political Science (UK) wanted to evaluate the capacity of the protocol based on the Square of PErcieved ACtion model (SPEAC model) to improve teaching at school. The protocol was developed to identify what constitutes competencies in adults in different work activities so as to improve vocational training (Fauquet-Alekhine, 2016); the results were positively conclusive, improving performance (Fauquet-Alekhine & Lahlou, 2017, 2019, Fauquet-Alekhine et al., 2019. The purpose of applying the SPEAC protocol at school was to broaden the applicability of the SPEAC-based method in a first pilot study in a new area. The study was conducted in secondary education at Saint-Louis School (Saint-Louis Institution, Saumur, France), that is with students 2 years before entering university.
To do this, a simple exercise in Economics & Social Science was chosen by the teacher. The way in which a skilled student completed the exercise was analysed using the SPEAC-based protocol to adjust the teaching of the exercise. The performances of two randomized samples of students before and after this adjustment were compared; one of the two samples benefited from the SPEAC protocol improvement, and the other not. This approach was integrated in a Cognitive Task Analysis approach (Wei & Salvendy, 2004, Tofel-Grehl & Feldon, 2013, applying a subjective tracing method structured by the SPEAC model (subjective video-recording of the skilled student in the chosen activity) followed by a reflective analysis.

Methods
The experimentation begun with the selection of a work activity (a 15-minute exercise) for which the teacher had identified an improvement needed regarding student performance and for which he wanted to modify the way he taught. The activity was first analysed during a phase involving the SPEAC protocol. This was followed by a phase of teaching. Evaluation of the student performance was undertaken before and after these phases (pre and post-tests).
It was a blinded experiment except for the student whose activity was analysed: students took lessons and tests as normal courses. There was a debriefing discussion between students and the teacher after the experiment.

Analysis Phase of the Activity
For the selected activity, the teacher chose a skilled volunteer student to perform it. The experiment was presented to the student and the parents, all of them signing an informed consent. Then the student was given the conditions necessary to carry out the activity, individually, wearing a miniature camera stuck on a pair of spectacles. This was undertaken during free time so as to preserve the blinded nature of the experiment (other students were not informed of it) and because the time allocated for teaching the class could not be devoted to analysing the activity. The realization of the activity was recorded from a first-person perspective; doing so, a subjective view of the action was recorded, and then analysed by the student and the researcher using the SPEAC protocol in the presence of the teacher (sequence also recorded): the student was questioned applying a goal-oriented subjective re-situ interview favouring reflexive analysis of the activity by the subject. It combines the techniques of self-confrontation (Von Cranack, 1989, Theureau, 2002, explicitation (Vermersh, 2006) and questioning structured by the SPEAC model (Fauquet-Alekhine, 2016).
The SPEAC model ( Fig. 1) articulates the effectiveness of competencies in action between 4 poles; (1) Having to act relates to the instruction given to the student by the teacher in the aim of performing the task, to the instruction written on the exercise paper with which the student is presented and to the fact that the student must complete the exercise as requested by the teacher. It relates to the motives; (2) Knowing to act relates to the knowledge, the know-how and the experience the student has acquired and developed regarding the activity; it is linked to the student' training and experience. Were this pole only to refer to know-how and experience, the wording of the pole would be: knowing how to act; (3) Being able to act relates to the means at the student's disposal, from a physical and a psychological standpoint: the student is given a pen and paper or a computer to perform the task, is seated in a quiet environment; the student is healthy, not psychologically disturbed by any concerns other than the test, not too stressed for example; (4) Wanting to act relates to the student's will to perform the task. It relates to the student's motivation.
The SPEAC model (Fig. 1)  and adequacy the four poles of the model are obtained. This is analysed through static and dynamic approaches.
The static approach examines whether the four poles are effective, which means both available and coherent. To be "available", a pole must have a content fully defined and this content must exist; this permits the subject to act. To be "coherent", any part of a pole must not counter another part of the same pole.
Example: the pole Having to act cannot be considered as available if a part of the required instruction to perform the exercise is missing and cannot be considered as coherent if the instructions are inconsistent with the teacher's requirements. The dynamic approach considers interpolar relationships in the model. Each of them must account for the symmetrical adequacy between the poles. A mismatch between at least two poles can lead to conflicts between these poles, generating difficulties in successfully putting competencies in action.
Example: the poles Having to act and Being able to act are not in adequacy if the results of the exercise must be written on a paper but there is no pen, no pencil. This dynamic approach refers to the property of "adequacy" and is combined with the properties of "availability" and "coherence" (static approach) to assess how competencies may be successfully put in action.  After the interview, a matrix of competencies was constructed by the researcher for the activity. This was achieved based on the interview data. Beforehand, the same protocol was applied with the teacher for the same purpose: for the selected activity, a matrix of competencies was constructed from the point of view of the teacher.

Teaching phase
The assumption was made that the student's matrix would be different from that of the teachers 'one. The comparison would make it possible to identify differences leading to teaching modifications and thus improving student performance. Previous studies have shown that tacit knowledge or know-how as well as non-technical skills are easily highlighted using the SPEAC protocol (Fauquet-Alekhine, 2016; Fauquet-Alekhine & Lahlou, 2017& Lahlou, , 2021Fauquet-Alekhine et al., 2020).

Phase of evaluation of the students' performance
In order to evaluate the effect of the modified teaching method, the teacher had the students pass written pre and post-tests using an exercise of the same type but of different content, this in the two classes before and after the phases of analysis of the activity and teaching. One class was associated with the control group and the other was associated with the test group. The test group was the student sample benefiting from the SPEAC analysis: between the two tests, the control group attended a typical lesson, i.e. the lesson did not benefit from the analysis of matrix of competencies, while the test group had a lesson benefiting from these contributions.
A predefined evaluation grid helped to mark the pre and post-tests, according to 22 criteria each scoring 1 point if successfully answered. The criteria, chosen by the teacher and checked by the researcher, were the exactitude of values to be calculated or arguments to be delivered in an answer. The marks were not disclosed to the student between the two tests which were carried out as a surprise test. Identical conditions were applied to the two classes during the tests: same day of the week, end of morning to avoid postprandial effect, same test duration and same type of classroom.
For each student, a score was therefore calculated for the pre-test and the-post test. For each group, an average score and its standard deviation was calculated for the pre-test and posttest. Inter-group and inter-test results were compared using a t-test.

Constitution of control and test groups
The size of the groups was smaller than that of the class size (30 students per class): both classes had similar overall levels (similar means of annual averaged score in Economics & Social Science (ESS) for each class) but the score distribution was significantly different as well as the average score for the pre-test (mean score about 11 for one and 15 for the other). A selection of 18 subjects per class (mean age: 17 years old) was thus undertaken on the basis of their annual mean score in ESS. Doing so, the means of annual averaged score was similar for both groups as well as the score distribution (Fig. 2).
A t-test confirmed the similitude between the means of annual averaged scores: • mean of averaged annual scores for the control group: 12.48 • mean of averaged annual scores for the test group: 12.76 • t(df= 34)=0.52 (p=0.60) A  2 test confirmed the similitude between the distribution of the averaged annual scores per group:  2 (df=4)=1.16 (p=0.44).
A t-test confirmed the similitude between the means of scores at the pre-test: • mean score for the control group: 12.77 ; variance  2 =14.50 • mean score for the test group : 13.33 ; variance  2 =52.0 • t(df= 34)=0.28 (p=0.77)

Teaching phase
The comparative analysis of the two matrices (that of the students versus that of the teacher) led to identify differences; therefore, the hypothesis has been verified. These differences led to teaching modification by identifying tacit knowledge or know-how and non-technical skills that were not included in teaching: for example i) analysis of the exercise data by deduction but also by analogy with previous exercises, ii) first doing what is easy to guarantee points but also to avoid the stress associated with getting blocked on a question. It also made it possible to identify points to be reinforced: for example, insisting on the need to combine two criteria to identify the optimal combination of productivity by clarifying the means to do so and the objectives to be known.

Evaluation and factors of performances
Post-test yielded data that is reported in Table 1 and compared to the results of the pre-test: the difference between mean scores at post-test between groups is significant, the difference between mean scores at pre and post-test for the control group has little significance, the difference between mean scores at pre and post-test for the test group is significant.
The SPEAC protocol contribution thus had a significant impact on the teaching of the exercise: the results show that the effect of a lesson and the repetition of an exercise increase performance (first line, Table 1) but they show that the SPEAC protocol allows a significantly higher increase in performance (second line, Table 1): teaching gains relevance and comprehensibility, and incorporates non-specific skills (i.e. skills common to other exercises or other disciplines) such as a strategy to help reduce stress. This also contributes, in the case studied, to reducing the variance and thus homogenizing the scores in the test group towards higher values. There were several factors that contributed towards the performance of the SPEAC protocol.
The first of these factors is the effect of the first-person perspective. Self-confrontation with the subjective recording of the activity rather than with a third person perspective video during the interview increases efficiency. This has also been observed by other researchers (Lahlou, 2006;Le Bellu, 2011;Lahlou et al. 2015). The first-person perspective helps the subject to activate the episodic memory which provides the context for a conscious re-experience of past experiences. Subjects watching the subjective video of their past activity are put back into the action: the auditory and visual stimuli help them to remember, to rebuild the perception and the understanding of the experienced situation and the related reasoning. Lahlou et al. (2015: 5) explained: "It seems that the more similar the context of memory retrieval is to the context of memory encoding, the better is the recall, and that having multimodal cues helps, especially when they are spatial or motor -see the enactment effect (Engelkamp & Cohen, 1991). In other words, re-living the situation from a first-person perspective would facilitate recalling one's own actions and mental states/processes. This considerably simplifies analysis and interpretation, as well as validating/falsifying research hypotheses." Second, the efficiency of the first-person perspective during the interview also takes its strength from the naturalistic context of the observed activity (Fauquet-Alekhine, 2016). An activity that is reconstructed would lose in spontaneity which is a criterion of spontaneous expression of the tacit competencies (Wasonga & Murphy, 2006). These tacit competencies are seized during the interview by the subjects themselves who discover a detail in their action that was unconscious and unnoticed until the analysis, or they become aware of them because of the researchers' questioning.
Third, the interview, structured according to the SPEAC model, allows the researchers to undertake an inter-comparison of answers per pole that favours the identification of what is conscious or what is not for the subjects. All that seems unconscious may relate to tacit competencies (Wasonga & Murphy, 2006). Through further questions, the researchers must then determine whether or not it must be understood as tacit or explicit.
Fourth, the self-confrontation provides easy access to non-technical skills provided that the researchers point it out whilst viewing the video because subjects usually focus on the technical gestures and objectives and forget the non-technical side of the activity.
Limitations of the SPEAC protocol Rieken et al. (2015) pointed out that "Just as the presence of a researcher can affect respondents' actions (Wickström and Bendix 2000), so might the presence of the recording device change the behaviour of recording and recorded individuals." In the present study, the student's behaviour during the analysis of the activity could be modified at least in two ways. Firstly, the behaviour could be impacted by the stress perceived by the student. However, previous studies (Fauquet-Alekhine, 2016; Fauquet-Alekhine & Lahlou, 2017) have shown that this bias is lessened early on once the subject gets involved in the nitty-gritty of the activity. Secondly, the subjects might force themselves to do better than they would in a daily context at school whilst performing the activity. However, this factor must not be considered as a bias when related the access to what makes students competencies, which is the objective of the work analysis. Indeed, the best way to obtain exhaustive results is to observe skilled students performing at their best. Therefore, combining positive and negative contributions, the firstperson perspective video induces a bias which is difficult to assess.
Another limitation is the time and the competencies needed to undertake the SPEAC-based method: do teachers have the time and the competencies for that, given their workload, the content of the programs to be taught and the time they are given to spend with students? Regarding the time (the present analysis took one hour for an activity lasting 15 minutes), it must be admitted that it is not easy to find a slot suitable for both teacher and student, especially due to the fact that it is carried out during free time. Regarding competencies, it is likely that the teachers will need a little training before implementing the method. In sum, applying the method requires an investment in time.

Conclusion
The present pilot study demonstrated that the SPEAC protocol could be successfully applied in the educational field, showing benefits for teaching (gaining relevance and comprehensibility, integrating tacit and non-specific skills). In that, it is consistent with the results obtained in the occupational field. The SPEAC-based method should not be confused with "flipped learning" or "inverted learning" (Starkey, 2020) which designates teaching methods that favour learning outside the classroom to promote student-teacher exchange in the classroom. The SPEAC protocol is not a teaching method but a means to improve teaching methods. It participates towards the shift from teacher-centred learning to student-centred learning (Marshall, 2007;Starkey, 2020). In the long term, the protocol will be applied again at the Institution Saint-Louis during the new academic year. Articles are currently being prepared for French national education publications to promote this method widely in the French educational system.

Data availability statement
The data that support the findings of this study are available from the corresponding author upon reasonable request.