A Novel Approach To Analyzing The Impact of AI Cha
A Novel Approach To Analyzing The Impact of AI Cha
A Novel Approach To Analyzing The Impact of AI Cha
Nahid Hasan, Johora Akter Polin, Md. Rayhan Ahmmed, Md. Mamun Sakib, Md. Farhan Jahin, Md.
Mahfuzur Rahman
Department of Computer Science and Engineering, Daffodil International University, Dhaka, Bangladesh
Corresponding Author:
Nahid Hasan
Department of Computer Science and Engineering, Daffodil International University
Daffodil Smart City, Birulia 1216, Dhaka, Bangladesh
Email: [email protected]
1. INTRODUCTION
Artificial intelligence (AI) in short AI is all around us in the current era. AI is now everywhere, from
virtual assistants like Amazon’s Alexa or Google’s Bard or OpenAI ChatGPT, or Microsoft’s Cortana to the
internet’s predictions of what we might want to do next or what we want AI to do for us. Another example of
the use of AI autonomous vehicles which will soon be revolutionizing the current world. Machines with AI
are able to comprehend and accomplish specific objectives. AI is characterized as “a machine-based system
that can make predictions, recommendations, or decisions influencing real or virtual environments” [1]. The
former speaks about machines that autonomously learn from data that is already available without human
assistance. The fast-developing subject of AI may be inaccessible to researchers and practitioners without a
background in computer science or engineering [2].
Technology is, after all, constantly evolving too. Therefore, in the future, AI will control all
technology. Learning management systems, electronic textbooks, and other forms of technology have
historically and largely appeared in the educational sector [3]. AI could theoretically help open up teaching
and learning opportunities that are otherwise challenging, that challenge current pedagogies, or that help
teachers be more effective [4]. An AI research team called OpenAI released a chatbot in November 2022
dubbed “ChatGPT” (generative pre-trained transformer) [5]. However, it offers performance on various
question kinds, accounting subjects, class levels, open/closed assessments, test bank questions, and once
again, discussion of implications for accounting research and teaching [6] and is used in the field of
psychology [7], utilized to offer medical advice and support, such as responding to medical inquiries or
supplying differentials for typical symptoms [8], can gather data, help with literature searches, and produce a
rough manuscript for the medical writer to continue working on [9]. However, the primary issue that is
becoming more and more of a concern is that students will eventually lose the ability to come up with
original ideas and will be unable to use strong arguments to support their claims [10]. For example, college
students utilize ChatGPT to potentially cheat on essay writing tasks when they are given assignments
something like “write a paragraph about how college students might use ChatGPT to write essays” [11].
The educational sector is on the verge of a paradigm shift, with AI-driven chatbots giving a dynamic
and interactive approach to learning. As we investigate the background of this research proposal, we discover
the difficulties and the solutions to those problems. One leading example, ChatGPT, not only disseminates
material but also engages students in tailored, real-time conversations that mimic the instruction of a human
tutor. The need to maximize the potential of these AI tools becomes increasingly important as demand for
remote and online education soars. Through our research, we hope to better understand the complex
dynamics and opportunities that come with the union of AI, specifically ChatGPT in education. However, as
the educational environment adopts AI, difficulties emerge. There is currently a lack of a thorough
knowledge of the effectiveness and best use of AI-driven chatbots in education. Significant challenges
include problems with scalability, potential biases, and flexibility to different learning styles. To ensure the
smooth and unbiased incorporation of AI into educational practices, a nuanced study of these difficulties is
clearly required.
The study used various machine learning models, including k-nearest neighbor (K-NN), random
forest (RF), and support vector machine (SVM), to investigate. Data from 2066 training points from the
university was collected through professor and student surveys and online resources. The results showed that
SVM model had a 98% accuracy (ACC) rate compared to other models.
2. LITERATURE REVIEW
Scholars have researched OpenAI’s ChatGPT, which was unveiled on November 22 a supervised
and reinforcement learning chatbot, in great detail; several studies have highlighted its contributions and
proposed models. Tlili et al. [5] examined this study to investigate how early adopters of ChatGPT are using
it for education. It does this by using a qualitative instrumental case study that includes three phases of
learning across many industries. Wood et al. [6] examined the performance of the ChatGPT and that of the
students for 28,085 questions from accounting assessments and textbook test banks, and the results suggest
that ChatGPT outperforms the student average for assessments that provide partial credit in 15.8% of cases.
When conducting research, ChatGPT can assist with writing or reviewing a literature review. Uludag et al.
[7] demonstrate how ChatGPT was used to construct a concise analysis of the literature to demonstrate the
development of the OpenAI ChatGPT AI application and discovered eight publications using the keyword
“ChatGPT”. However, the author investigated ChatGPT’s psychological effects. Another study shows how
ChatGPT outperformed in teaching and learning. Baidoo-Anu and Ansah [12] synthesize recent literature to
offer some potential advantages with its automatic essay grading, language translation, individualized
coaching, ChatGPT improves teaching, and learning. In order to address the subject of ChatGPT’s
capabilities, a research employed the latent Dirichlet allocation (LDA) topic modeling technique to examine
tweets regarding the platform. Taecharungroj [13] took 3 general topics including news, technology, and
reactions, and used them to analyze 233,914 English tweets. Besides, the study then revealed that ChatGPT
has the capacity to have both positive and negative effects on technology and people. Mhlanga [14] presents
the study using 55 publications from Web of Science, Google Scholar, and ResearchGate, this study
examines the moral use of ChatGPT in education and promotes more investigation into the topic. To evaluate
OpenAI in education, the responsible and ethical use of ChatGPT towards lifelong learning. As ChatGPT is
rapidly expanding, Aydın and Karaarslan [15] introduced in their paper that intends to throw light on what is
happening in the literature and gain an understanding of user expectations of ChatGPT and generative AI.
They looked at some of the most interesting LinkedIn posts as well as a selection of news items, preprint
papers, blogs, and published articles. Researchers in the medical field also use ChatGPT for educational and
assistance purposes. Khan et al. [16] describe how ChatGPT has significantly impacted medical science and
investigate how AI has a variety of beneficial roles in this modern world.
Bulletin of Electr Eng & Inf, Vol. 13, No. 4, August 2024: 2951-2958
Bulletin of Electr Eng & Inf ISSN: 2302-9285 2953
Firat [17] tries to analyze how autodidactic experiences have changed in ChatGPT, showing how
chatbots can enhance learning and increase student engagement in online courses. They also show how different
types of self-directed learning can be carried out through ChatGPT, including increased accessibility, real-time
feedback, guidance, self-assessment, and reflection. Higher education can benefit from ChatGPT as well, as
Rudolph et al. [18] discuss the project investigates how learning, teaching, and assessment procedures in higher
education can change in the future in light of AI chatbots like ChatGPT. According to Sallam [19] goal was to
investigate the usefulness of ChatGPT in health care education, research, and practice and to highlight any
potential limitations. Besides, a systematic search was done to find English-language records in
PubMed/MEDLINE and Google Scholar, where a total of 60 records were eligible for inclusion.
3. PROPOSED METHOD
According to researchers, the method that can able to access the quality of requirements and
replicate it can offer new requirements [20]. Intelligent chatbots have drawn the attention of researchers,
educators, and learners as they interact with learners and promptly solve their issues realistically. The study is
conducted using the dataset collected via an offline survey from researchers, teachers, and undergraduate
students in Bangladesh. A set of questions is set to analyze the data using machine learning techniques.
Figure 1 demonstrates the working process that we have to follow to achieve the goal starting from data
collection to result analysis.
3.5. Implementation
We used a SVM, NB, K-NN, DT, and RF models in our dataset for measuring precision (PR), recall
(RE), f-measure (F1), and support which were used to assess the multiclass machine learning models. After
training, the AI model is tested to determine how well it performs and how well it generalizes. For example,
ACC, PR, and RE are some evaluation measures that are used for evaluating how well the model works on
data that is not observed. We have split our data set 80% and 20% for training and testing respectively to
experiment using Kaggle. Kaggle is a powerful tool and resource which helps to achieve the goal using
Python. Performance is measured using a confusion matrix. This study [24] provides a proper definition of
the confusion matrix as a crucial metric for assessing the reliability of credit scoring models.
A two-way frequency table called the confusion (or misclassification) matrix has two binary
variables: actual (good or bad) and predicted (good or bad). The names of the four values-true negative (TN),
true positive (TP), false negative (FN), and false positive (FP), respectively-of the four elements-c11, c12,
c21, and c22-are not always listed in that sequence. The formula for the confusion matrix is given:
𝑇𝑃 + 𝑇𝑁
𝐴𝐶𝐶 =
𝑇𝑃+𝑇𝑁+𝐹𝑃+𝐹𝑁
Bulletin of Electr Eng & Inf, Vol. 13, No. 4, August 2024: 2951-2958
Bulletin of Electr Eng & Inf ISSN: 2302-9285 2955
𝑇𝑃
𝑃𝑅 =
𝑇𝑃 + 𝐹𝑃
𝑇𝑃
𝑅𝐸 =
𝑇𝑃+𝐹𝑁
(𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 𝑥 𝑟𝑒𝑐𝑎𝑙𝑙)
𝐹1 − 𝑠𝑐𝑜𝑟𝑒 = 2 𝑥
𝑝𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 + 𝑟𝑒𝑐𝑎𝑙𝑙
Table 3 demonstrates the optimal parameters for both of the trials. Each of the classifiers has its
value after implementing formulas.
PR and RE values of all five algorithms NB, SVM, DT, RF, and K-NN are represented by a line
graph in Figure 2. PR and RE are important for the performance assessment. In the study of five algorithms,
performance is presented by confusion matrix which represents the performance in Figure 3. Where
Figure 3(a) represents SVM, Figure 3(b) represents DTs, Figure 3(c) represents K-NN, Figure 3(d) represents
RF, and Figure 3(e) represents NB.
Figure 3. Confusion matrix of each classifier: (a) SVM, (b) DT, (c) K-NN, (d) RF, and (e) NB
appropriate operational point, or decision threshold. Figure 4 ROC curve illustrates the various spots of each
classifier. It displays the varied behavior of the research’s algorithms. On the other hand, the term “ACC”
refers to a model’s or algorithm’s ability to be true or accurate. The bar graph in Figure 5 presents a visual
comparison of the algorithms.
Figure 4. ROC curve of the algorithms Figure 5. Comparison of ACC of each classifier
Comparing the F1-score, we can see that the SVM has the highest average F1 positive class score
(0.98) and negative class score (0.98), followed by DTs positive class (0.77) and negative class (0.73), K-NN
positive class (0.86) and negative class (0.85), RF positive class (0.87) and negative class (0.86), and NB
positive class (0.89) and negative class (0.88). The study reveals that SVM is the most effective algorithm in
terms of ACC, F1-score, PR, and RE. SVM and NB have the highest average PR, followed by DTs, K-NN,
and RF. In terms of RE, SVM and NB have the highest average RE, followed by DTs K-NN, and RF. A high
RE score indicates a low false negative rate. The choice of the best algorithm depends on the specific task
and the trade-offs between different performance metrics. The results suggest that SVM and NB are the most
effective algorithms in terms of ACC, F1-score, PR, and RE.
5. CONCLUSION
A conversational chatbot for a pre-learned application with a significant amount of trained data is
called ChatGPT. It is, indeed, a world-revolutionizing invention. The incorporation of AI, as demonstrated by
ChatGPT, which is a conversational chatbot equipped with a sizable reservoir of previously learned
application data, has emerged as a disruptive force in the field of education. Our study travels across this
ground-breaking territory. The educational potential of ChatGPT is made clear by the fact that it not only acts
as a vehicle for knowledge dissemination but also actively involves users in conversation, simulating the
individualized instruction of a human tutor. In a time when online and remote learning are becoming more
and more common, this dynamic approach to learning is very helpful. We employed five machine learning
algorithms to improve teaching and learning methods in our research. Of the 2532 data we obtained, 2066
were ultimately used. The data came from a poll with 25 questions that were organized using the most recent
research articles from 2023 that are based on ChatGPT and AI. The data was collected using an online
Google form. Utilizing a variety of techniques on our dataset, including feature selection, data preprocessing
data analysis, and model application after trained data. Given that we utilized five machine learning models,
it appears that the SVM performed with the greatest ACC in our study. The SVM model’s ACC is 98%.
A comparison graph and an ROC curve were also provided for the five models. As a result of our research,
we have been able to analyze how ChatGPT functions for researchers and students. It will give ChatGPT and
the analyst and researchers a solid starting point for additional analysis. Our studies suggest future initiatives
Bulletin of Electr Eng & Inf, Vol. 13, No. 4, August 2024: 2951-2958
Bulletin of Electr Eng & Inf ISSN: 2302-9285 2957
to enhance the integration of AI-driven chatbots like ChatGPT in education by maximizing their adaptation
to different learning styles. Real-time interactions and continuous learning methods could lead to a more
dynamic teaching tool. Ethical issues and potential biases related to AI in education are also crucial for
promoting impartial and equitable learning environments.
REFERENCES
[1] X. Huang, D. Zou, G. Cheng, X. Chen, and H. Xie, “Trends, research issues and applications of artificial intelligence in language
education,” Educational Technology and Society, vol. 26, no. 1, pp. 112–131, 2023, doi: 10.30191/ETS.202301_26(1).0009.
[2] N. Gillani, R. Eynon, C. Chiabaut, and K. Finkel, “Unpacking the ‘black box’ of AI in education,” Educational Technology and
Society, vol. 26, no. 1, pp. 99–111, 2023, doi: 10.30191/ETS.202301_26(1).0008.
[3] X. Zhai, “ChatGPT and AI: the game changer for education,” AI4STEM Education Center, pp. 1–5, 2023.
[4] R. A. Abdulmunem, “Artificial intelligence in education,” Comparative Research on Diversity in Virtual Learning: Eastern vs.
Western Perspectives, pp. 241–255, 2023, doi: 10.4018/978-1-6684-3595-3.ch012.
[5] A. Tlili et al., “What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education,” Smart Learning
Environments, vol. 10, no. 1, 2023, doi: 10.1186/s40561-023-00237-x.
[6] D. A. Wood, M. P. Achhpilia, M. T. Adams, S. Aghazadeh, E. D. Almer, “The ChatGPT artificial intelligence chatbot: how well does it
answer accounting assessment questions?,” Issues in accounting education, American Accounting Association, vol. 38, pp. 1–28, 2023.
[7] K. Uludag, “The use of AI-supported chatbot in psychology,” SSRN Electronic Journal, pp. 1–11, 2023, doi:
10.2139/ssrn.4331367.
[8] ABHI GARG, “What is ChatGPT, and its possible use cases?,” 2022.
[9] S. Biswas, “ChatGPT and the future of medical writing,” Radiology, vol. 307, no. 2, Apr. 2023, doi: 10.1148/RADIOL.223312.
[10] H. H. Thorp, “ChatGPT is fun, but not an author,” Science, vol. 379, no. 6630, p. 313, 2023, doi: 10.1126/science.adg7879.
[11] M. R. King, “A conversation on artificial intelligence, chatbots, and plagiarism in higher education,” Cellular and Molecular
Bioengineering, vol. 16, no. 1, pp. 1–2, Feb. 2023, doi: 10.1007/s12195-022-00754-8.
[12] D. Baidoo-Anu and L. O. Ansah, “Education in the Era of generative artificial intelligence (AI): understanding the potential
benefits of ChatGPT in promoting teaching and learning,” Journal of Artificial Intelligence, vol. 7, no. 1, pp. 52–62, 2023, doi:
10.61969/jai.1337500.
[13] V. Taecharungroj, “What can ChatGPT do?’ analyzing early reactions to the innovative AI chatbot on twitter,” Big Data and
Cognitive Computing, vol. 7, no. 1, p. 35, Feb. 2023, doi: 10.3390/bdcc7010035.
[14] D. Mhlanga, “Open AI in education, the responsible and ethical use of ChatGPT towards lifelong learning,” SSRN Electronic
Journal, 2023, doi: 10.2139/ssrn.4354422.
[15] Ö. Aydın and E. Karaarslan, “Is ChatGPT leading generative AI? what is beyond expectations?,” SSRN Electronic Journal, 2023,
doi: 10.2139/ssrn.4341500.
[16] R. A. Khan, M. Jawaid, A. R. Khan, and M. Sajjad, “ChatGPT-reshaping medical education and clinical management,” Pakistan
Journal of Medical Sciences, vol. 39, no. 2, pp. 605–607, Feb. 2023, doi: 10.12669/pjms.39.2.7653.
[17] M. Firat, “How chat GPT can transform autodidactic experiences and open education?,” Preprint, vol. 2, no. 5, p. 255, 2023.
[18] J. Rudolph, S. Tan, and S. Tan, “ChatGPT: bullshit spewer or the end of traditional assessments in higher education?,” Journal of
Applied Learning and Teaching, vol. 6, no. 1, pp. 342–363, Jan. 2023, doi: 10.37074/jalt.2023.6.1.9.
[19] M. Sallam, “ChatGPT utility in healthcare education, research, and practice: systematic review on the promising perspectives and
valid concerns,” Healthcare (Switzerland), vol. 11, no. 6, p. 887, Mar. 2023, doi: 10.3390/healthcare11060887.
[20] E. Parra, C. Dimou, J. Llorens, V. Moreno, and A. Fraga, “A methodology for the classification of quality of requirements using
machine learning techniques,” Information and Software Technology, vol. 67, pp. 180–195, 2015, doi:
10.1016/j.infsof.2015.07.006.
[21] L. Kohnke, B. L. Moorhouse, and D. Zou, “ChatGPT for language teaching and learning,” RELC Journal, vol. 54, no. 2, pp. 537–
550, 2023, doi: 10.1177/00336882231162868.
[22] B. D. Lund and T. Wang, “Chatting about ChatGPT: how may AI and GPT impact academia and libraries?,” Library Hi Tech
News, vol. 40, no. 3, pp. 26–29, 2023, doi: 10.1108/LHTN-01-2023-0009.
[23] Y. C. Tsai, “Empowering learner-centered instruction: integrating ChatGPT Python API and Tinker learning for enhanced
creativity and problem-solving skills,” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial
Intelligence and Lecture Notes in Bioinformatics), vol. 14099 LNCS, pp. 531–541, 2023, doi: 10.1007/978-3-031-40113-8_52.
[24] G. Zeng, “On the confusion matrix in credit scoring and its analytical properties,” Communications in Statistics-Theory and
Methods, vol. 49, no. 9, pp. 2080–2093, May 2020, doi: 10.1080/03610926.2019.1568485.
[25] A. P. Bradley, “The use of the area under the ROC curve in the evaluation of machine learning algorithms,” Pattern Recognition,
vol. 30, no. 7, pp. 1145–1159, 1997, doi: 10.1016/S0031-3203(96)00142-2.
BIOGRAPHIES OF AUTHORS
A novel approach to analyzing the impact of AI, ChatGPT, and chatbot on … (Nahid Hasan)
2958 ISSN: 2302-9285
Johora Akter Polin senior lecturer for the Computer Sciences and Engineering
Department at the Daffodil International University in Bangladesh, and also works as a judge
in the Social Business Creation Competition, which is conducted by HEC Montreal, Canada.
She holds a bachelor’s degree in computer sciences and engineering. Being passionate about
research, social business, networking, and data analytics, she is especially interested in
machine learning, deep learning, human-computer interaction, and social media analysis. She
is also actively involved in the community as a public speaker and a mentor to support and
encourage young researchers. She can be contacted at email: [email protected].
Bulletin of Electr Eng & Inf, Vol. 13, No. 4, August 2024: 2951-2958