05. Levels of Reflective Text

Haiyue12/28/23About 5 min

Implementation on google colab

Dataset

There are 6 columns with this dataset, which are module_id, student_id, Answers, Question, Group, Final like the picture below.

module_id: relative to question.
student_id: student id
Answers: The answer for the relative question, the data need to be classified.
Question: The question for answer.
Group: relative to question.
Final: represent the levels, from 1 to 4 correspond to [No-Reflection, Understanding, Simple Reflection, Critical Reflection]

Only Answers, Question and Final are related to our task.

Levels definition

No-Reflection: Professionals do not indicate any significant reflective thought processes and mainly mirror the words and phrases from the question or give a general comment.
Understanding: Professionals indicate an understanding of concepts or topics from demonstrating a practical application of theory, yet lack specific details of professional or real-life experiences.
Simple Reflection: Professionals indicate an understanding of the theory in unison with examples of practical application, yet only provide a vague explanation of future actions or outcomes. They do not provide concrete examples of action or application for the future.
Critical Reflection: Besides relating the theory with examples from real-life experiences, professionals indicate what they are likely to do in the future or what they need to work on. In some cases, professionals might also reflect a change in their perspective.

Tasks

Using openai to classify the answers to a specified level based on the level definitions and question.

Parameters for openai

model: gpt-4-0314
others: default values

Prompt & result

Prompt

The prompt should be defined as the content below:

System prompt

There are four levels definition for reflection content as below.

Level definitions:
No-Reflection: Professionals do not indicate any significant reflective thought processes and mainly mirror the words and phrases from the question or give a general comment.
Understanding: Professionals indicate an understanding of concepts or topics from demonstrating a practical application of theory, yet lack specific details of professional or real-life experiences.
Simple Reflection: Professionals indicate an understanding of the theory in unison with examples of practical application, yet only provide a vague explanation of future actions or outcomes. They do not provide concrete examples of action or application for the future.
Critical Reflection: Besides relating the theory with examples from real-life experiences, professionals indicate what they are likely to do in the future or what they need to work on. In some cases, professionals might also reflect a change in their perspective.

Question:
"""[question]"""

Please define the level based on the level definitions and question above. The result should be json format, like 
{"level": [The level result]}

User Prompt

[Answer text]

Result

The dataset has been processed with two times. All the result will be listed below.

The distribution for the final column in the original dataset.

The results for the distribution from the openai.

If we regard final column as a baseline. The difference is the result from openai minus baseline. If the result is 0, it means the result remains the same with baseline. If the result is a positive value, it means the openai gives a higher level than baseline; else if the result is a negative value, it means the openai gives a lower level than baseline.

According to the results above, the level from openai normally give a higher level than baseline. The difference is mostly 1.

Difference distribution on the final level

The following distribution descripe the openai result give a higher level for original 2 (Understanding) and 3 (Simple Reflection); the original level 1 and 2 (exist on relative lower o original level) will get 2 level higher from openai; the count of 3 level higher than baseline is quite small that could be ignored; 1 level lower than baseline focus on 2(Understanding) and 4 (Critical Reflection), it can be ignored because of the lower acount.

Distribution of difference based on (1st) — Distribution of difference based on `Final` (1st)

Distribution of difference based on (2nd) — Distribution of difference based on `Final` (2nd)

The different levels exist overlaps, the result will be different from the baseline even if a person judges sometimes, especially for the neighboring levels.

Based on the distributions of the results. So, I think it can work.

If want to get more accurate results need more investigation and fine-tuning.

Feedback from Abhinava

In terms of the descriptions of the four levels, it is very tricky. I will try to explain here as much as I can -

No reflection: This doesn't provide any kind of understanding or reflection. Usually, it comprises a simple one-line response.
Understanding: This provides an understanding of the situation. However, it does not reflect on past learning or professional experience. For example - I think having a strategy is very important. This course material on Strategy development is useful.
Simple reflection: This provides understanding as well as some sort of reflection. For example - I think having a strategy is very important. However, I have never used strategy will working in teams in the past.
Critical reflection: It reflects the past and provides prescriptions for the future. For example - I think having a strategy is very important. I have never used strategy will working in teams in the past. However, going forward, I will extensively use strategies to do bla bla bla.

Result Analysis

Example 1

Original

Question: Can you think of a time that you would have had a better experience or outcome if you had known about strategy? Is there an opportunity, by becoming more familiar with strategy, to align your efforts with it? Write 3-5 sentences on what you experienced and how it would go better if you knew the strategy framework you work within.
Reflection: At one point in my life I was a Supplier Diversity Program Coordinator and my main goal was attract new businesses to our chamber of commerce. I focused too much on attracting businesses and not enough on aligning the businesses with our strategy. Had I focused more on strategy, I believe I would have had more luck attracting the businesses.
Final: 1

Result from OpenAI:
Result: Simple Reflection - 3
Reason: The reflection provides an understanding of the situation and includes a personal experience, but doesn't provide prescriptions for the future.

My Thought
Info
Comparing the explanation from Abhinava and the definition of levels, I think the result from OpenAI is more reasonable.

Example 2

Original

Question: Reflect and internalize the attributes of strategic thinkers. Have you seen these attributes in others around you? Look back at your identification activity at the beginning of this lesson. Pick out one that you would like to improve and develop a personal development plan. What 2 steps will you take to improve on that attribute? Write a few sentences outlining that plan. After you write your reflection, share your choice in the poll below, and explain your choice in the discussion below.
Reflection: CreativitynI tend to look for opportunities and solutions within the scope of my experience and available tools.n2 stepsn1. I will focus on identifying at least one NEW idea when looking for solutions.n2. I will intentionally step back and divert my mind to other things before choosing a solution, with being mindful of the one new idea and giving myself the opportunity to come up with additional options.
Final: 1

Result from OpenAI:

Result: Simple Reflection - 3
Reason: The reflection provides understanding and some reflection on a selected attribute and a plan to improve it without mentioning past experiences or prescribing for the future.

My Thought
Info
Comparing the explanation from Abhinava and the definition of levels, I think the result from OpenAI is more reasonable.

Feedback from Srecko

Keep it that way for now.