Computers and Education: Artificial Intelligence 7 (2024) 100259

Contents lists available at ScienceDirect

Computers and Education: Artificial Intelligence

journal homepage: www.sciencedirect.com/journal/computers-and-education-artificial-intelligence

Perceptions and usage of AI chatbots among students in higher education

across genders, academic levels and fields of study
Christian Stöhr * , Amy Wanyu Ou , Hans Malmström
Department of Communication and Learning in Science (CLS), Chalmers University of Technology, SE-41296, Gothenburg, Sweden

A R T I C L E I N F O A B S T R A C T

Keywords: AI chatbots have ignited discussions and controversies about their impact on teaching and learning practices in
Chatbots higher education. This study explores students’ adoption and perceptions of ChatGPT and other AI chatbots in
Artificial intelligence higher education. Based on survey data from a large sample (n = 5894) across Swedish universities, the study
ChatGPT
employs descriptive statistical methods to analyze usage, attitudes, and concerns, and inferential statistics to
Student attitudes
Higher education
identify relations between attitudes and usage and background variables (gender, academic level, and field of
study). The results reveal broad awareness and use of ChatGPT among students, but not of other AI chatbots.
More than half of the students expressed positive attitudes towards the use of chatbots in education, but almost as
many expressed concerns about future use. Statistically significant differences were found across all examined
grouping variables, particularly between genders and fields of study. Female students and students from the
humanities and medicine consistently expressed more negative attitudes and concerns about AI’s role in learning
and assessment, while males and technology and engineering students showed higher usage and optimism. These
findings not only validate the continued relevance of student backgrounds as a determinant of technology
adoption but also expose several challenges and considerations surrounding AI and chatbot usage in education.
The study supports the development of local solutions to AI in education tailored to student attributes and needs,
and it provides insights for developers, educators, and policymakers.

1. Introduction the launch of OpenAI’s ChatGPT in November 2022 (OpenAI, 2022).

This represents a paradigm shift in the domain; with over 100 million
Artificial intelligence (AI) plays an important role in higher educa­ users within the first two months, the chatbot has become the
tion and is profoundly influencing the academic and everyday lives of fastest-growing application worldwide (Dennean et al., 2023), forcing
students (Chen et al., 2020). Foremost in developed countries and China, other major tech companies to push their AI development programs.
AI applications are increasingly implemented in education with signif­ ChatGPT is based on the large language model Generative Pre-trained
icant potential to impact teaching and learning across all levels (Tahiru, Transformer (GPT), which is trained on massive collections of data in
2021). Examples include AI-based adaptive learning platforms that the form of books, articles and openly accessible webpages. In contrast
provide personalized learning experiences to students, automated to most previous chatbots, ChatGPT has not only impressed with the
assessment, or AI-powered writing tools that enhance students’ writing quality of its responses, but also showcased its unique capability to
quality by providing real-time feedback on issues with grammar, “remember” a certain number of previous interactions within the same
punctuation, and style (Holmes & Tuomi, 2022; Zawacki-Richter et al., conversation. The versatility of ChatGPT as a conversational AI extends
2019). Concomitantly, AI in education (AIED) has emerged as a vibrant well beyond specific, single-purpose applications, and its potentials to
academic field, extending the capability of AI not just to learners but also increase efficiency, accuracy and cost savings were quickly highlighted
to educators and institutions alike (Chen et al., 2020; Hwang et al., (Deng & Lin, 2022). This seemingly opens up a multitude of opportu­
2020; Nemorin et al., 2023). nities and new challenges, and the potentially disruptive impact of
Among current AI innovations, chatbots – conversational agents generative AI is subject to intensive discussion in academia and the
simulating human dialogue through natural language processing and public (Li et al., 2023; Lo, 2023).
machine learning algorithms – have gained particular traction through

* Corresponding author.
E-mail addresses: [email protected] (C. Stöhr), [email protected] (A.W. Ou), [email protected] (H. Malmström).

https://doi.org/10.1016/j.caeai.2024.100259
Received 28 August 2023; Received in revised form 22 April 2024; Accepted 23 June 2024
Available online 24 June 2024
2666-920X/© 2024 The Authors. Published by Elsevier Ltd. This is an open access article under the CC BY license (http://creativecommons.org/licenses/by/4.0/).
C. Stöhr et al. Computers and Education: Artificial Intelligence 7 (2024) 100259

1.1. Advantages of AI chatbots in education fields informed discussions and decision-making (e.g., Bates et al., 2020;
García-Peñalvo, 2023; Rudolph et al., 2023). To this end, this paper aims
In education, ChatGPT’s user-friendly and intuitive interface to examine students’ familiarity with, usage of and attitudes towards
potentially reduces barriers to its wide adoption across educational ChatGPT and other AI chatbots for different student populations based
settings and for different groups of teachers and learners (Kasneci et al., on gender, academic level and field of study. We thereby draw from a
2023), thereby overcoming many of the obstacles reported in the AIED dataset of nearly 6000 questionnaire responses across many universities
literature. ChatGPT and similar AI applications can serve as self-study and diverse academic disciplines in Sweden. A preliminary summary of
tools (Nisar & Aslam, 2023), assisting students in acquiring informa­ the survey data (i.e., general frequencies of AI chatbot usage and atti­
tion, answering questions (Chen et al., 2023), facilitating group dis­ tudes and respondent comments) has been reported in Malmström et al.
cussions, and resolving problems instantaneously (Rahman & (2023). In this study, we focus on descriptive, inferential, and correla­
Watanobe, 2023), thereby enriching students’ learning experiences, tional statistical analyses to provide a nuanced and comprehensive un­
offering personalized support, and potentially boosting academic per­ derstanding of different groups of university students’ AI usage and
formance (Kasneci et al., 2023). ChatGPT has also been effectively attitudes. Specifically, the study is guided by the following four research
deployed in the design of educational materials and creative assess­ questions.
ments, providing new avenues for content creation and curriculum
development (Cotton, Cotton, & Shipway, 2023; Dijkstra et al., 2022). • What is the overall prevalence and pattern of AI chatbot usage among
students in higher education? (RQ1)
1.2. Limitations of AI chatbots in education fields • What are the general attitudes towards AI chatbots among students in
higher education? (RQ2)
Nonetheless, as the advancement and influence of ChatGPT • How do gender, academic level, and field of study influence the usage and
permeate teaching and learning, concerns are being voiced by stake­ attitudes toward AI chatbots in an educational context? (RQ3)
holders and scholars. Critical issues in higher education involve assess­ • What is the relationship between students’ attitudes towards AI chatbots
ment, examination, and academic integrity (Cotton et al., 2023; Eke, and student’s reported use of ChatGPT in their learning process? (RQ4)
2023; Yeadon & Halliday, 2023). Advanced generic AI tools such as
ChatGPT pose a significant challenge as they are able to closely mimic RQ 1 and RQ2 highlight the present state and impact of AI chatbots
students’ work and are therefore difficult to distinguish from the stu­ in higher education, focusing on AI chatbots as student-facing tools.
dents’ own contributions. This raises concerns about untraceable Prior research has identified generally positive attitudes of the public
plagiarism and cheating, prompting a reevaluation of many established towards the use of AI in education (Latham & Goltz, 2019). Given the
assessment methods (Farazouli et al., 2023). Critics have also pointed recency of the breakthrough of generative AI, little empirical research is
out ChatGPT’s technical limitations, as it is known to sometimes create available from within the higher education sector, though there are a
incorrect information and hallucinate (Weise & Metz, 2023), leading to few recent reports to build on. A recent student survey from a Belgian
reliability concerns. Bogost (2022, p. 1) notes that “ChatGPT and the university found that, while a large majority of students had used some
technologies that underlie it are less about persuasive writing and more forms of AI tools for coursework, only 13% of the students had used
about superb bullshitting.” Large language models like ChatGPT heavily ChatGPT (Lobet et al., 2023). A U.S. survey (Welding, 2023) among
depend on training data sourced from the internet, resulting in outputs college students revealed that 43% of the respondents had experience
that can reflect existing biases present in the data. This raises concerns with ChatGPT or similar applications and about one third (32%) indi­
about the potential reinforcement of societal biases, imbalances, or cated that they used or planned to use AI tools for assignment comple­
prejudices that are prevalent in online content (Deng & Lin, 2022). tion. About half (47%) of the American students were concerned about
Consequently, a lack of understanding of AI capabilities and limitations the impact of AI on their education and 60% reported that their in­
amongst end users might lead to misuse and over-reliance on AI tech­ structors or schools had not (yet) specified how AI tools could be used
nologies by teachers and learners (Kasneci et al., 2023). As AI becomes ethically or responsibly. Nevertheless, further monitoring of students’
more deeply integrated into educational systems, there are also signifi­ use and attitudes toward artificial intelligence in educational settings is
cant concerns surrounding copyright issues, data privacy, and security essential to enhance informed decision-making by stakeholders in
(ibid.). On the other hand, the power of ChatGPT and other AI tools is higher education.
likely to provide an advantage to some users over non-users, creating an Technology adoption and usage have been a long-standing theme
imbalance in the educational landscape (Cotton et al., 2023). This can within theoretical and empirical information systems research. Several
further reinforce inequalities, as students with access to and experience theories and models have been widely employed due to their extensive
with sophisticated AI tools can outperform those without them (Adi­ predictive and explanatory power, among them Davis’ (1989) Tech­
guzel et al., 2023). Lastly, the sustainability of these technologies and nology Acceptance Model (TAM), the Theory of Planned Behavior
their energy consumption cannot be ignored, since the substantial power (Ajzen, 1991), and Venkatesh et al.’s (2003) Unified Theory of Accep­
needs of AI systems can contribute to a significant environmental foot­ tance and Use of Technology (UTAUT) as a consolidated framework of
print (Kasneci et al., 2023). These considerations indicate that the several earlier models and theories including the two others. These
emergence of ChatGPT – in addition to the euphoria about its potential – theories link perceptions and beliefs with a user’s intent and actual use
has also resulted in a more critical discourse about AI and its impact on of technology moderated by individual and contextual factors (e.g., Liu
education and society; it is likely that AI deployment in education will & Ma, 2023). Loosely building on this theoretical tradition, RQ3 in­
affect diverse groups in various educational contexts differently, raising vestigates the link between demographics, attitudes, and AI chatbot
questions of power, disadvantage and marginalization (Selwyn, 2022). usage. By examining the influence of variables such as gender, academic
level, and field of study on AI chatbot usage and attitudes, we aim to
2. Aim of the study uncover existing disparities or biases that may need to be addressed by
educators and educational institutions. This is vital for ensuring that all
Comprehensive and systematic empirical research to support or students, regardless of their demographics or academic backgrounds,
reject claims made about the benefits and challenges of AI in education, can benefit from AI chatbots in education. Additionally, it offers a
and in particular large language models is, however, scarce, and crucial background for understanding the social dynamics that might be
research on AI chatbots in education is still very much in a state of at play in the adoption and acceptance of AI chatbots in a learning
evolution (Hwang & Chang, 2021; Rudolph et al., 2023). Data from environment. Gender has been shown to be a moderating variable in AI
stakeholder perspectives are sorely needed to provide a basis for more adoption (e.g., Nouraldeen, 2022). However, gender effects tend to be

2
C. Stöhr et al. Computers and Education: Artificial Intelligence 7 (2024) 100259

stronger for older people as gender stereotypes are less likely to be Questback during the spring of 2023. The survey was split into two main
prominent among younger generations (Morris et al., 2005). Further, sections addressing chatbot usage and attitudes towards AI in education.
technology adoption has been linked to performance expectancy for
males and ease of use for females (Venkatesh & Morris, 2000). Both 3.2. Instrument and data collection
effects are relevant for the context of this study as students tend to be
younger and ChatGPT stands out with its ease of use. Thus, while gender The survey questions can be found in Appendix 1. The first section of
effects might be present among students, we expect the differences to be the survey aimed to gauge the students’ familiarity with and usage of ten
low.1 The relationship between academic level and technology adoption AI chatbots that supposedly are used for educational purposes, including
in general, or AI in particular, has received very little scholarly atten­ but not limited to ChatGPT.2 For each chatbot, respondents were asked:
tion. While Abu-Shanab (2011) found that the educational level did play ‘Rate your familiarity and frequency of use with a selection of AI chatbots,’
a moderating role, Sandu and Gide (2019) could not identify a rela­ followed by a four-item ordinal scale as answer categories: Unfamiliar;
tionship between gender, age, or the level of education of the students Familiar but never use it; Familiar but rarely use it; and Familiar and regu­
and the adoption of chatbot technology in higher education. Similarly, larly use it.
research on the academic discipline of students as a moderating factor is The second section focused on student attitudes towards AI in edu­
scarce (Chiu et al., 2023), though existing evidence suggests that there cation. This covered general attitudes towards chatbots in education,
are greater barriers to technology adoption within the fields of arts and perceived effects of chatbot use on learning and academic performance,
humanities compared to technological fields (e.g., Mercader & Gairín, ethical concerns, and issues related to institutional guidelines on chatbot
2020). usage. The latter section used a response format comprising ten agree-
RQ4 delves into the interplay between perception and practice, a disagree statements, along with a “don’t know/prefer not to say” op­
crucial aspect for the effective integration of AI chatbots in education. tion. Background information pertaining to gender, field of study, and
Investigating the relationship between students’ attitudes towards AI academic level was also collected from respondents (see Table 1).
chatbots and their actual usage of these tools in their learning process Although we also asked about university affiliation, this aspect was not
helps us understand the behavioral dynamics behind the adoption of AI analyzed in this study. A link to the survey was disseminated via mul­
in education. As an analysis for all AI chatbots would exceed the scope tiple channels, encompassing networks with various Swedish univer­
and length constraints of this article, we focus on ChatGPT as the sities and a promotional campaign on social media platforms, such as
currently most popular AI chatbot. In line with UTAUT, positive atti­ LinkedIn and Facebook. The survey was open from April 8, 2023, to May
tudes regarding chatbots are expected to be substantially correlated with 5, 2023, securing a sample of 5894 students; this convenience sample
chatbot usage (Alzahrani, 2023); conversely, concerns about perceived broadly mirrored the national distribution of students in terms of
risks and ethical issues that reflect a lack of trust in current AIED are gender, academic level, and discipline.
anticipated to diminish the adoption of chatbots (e.g., Qin et al., 2020).
In answering these four questions, this paper will contribute to a
3.3. Data analysis
more nuanced understanding of the role of AI chatbots in higher edu­
cation, in the context of the potentially disruptive impact of ChatGPT.
The analysis of the collected data encompassed both descriptive and
We expect our findings to guide developers, educators, and policy­
inferential statistical methods using the Statistical Package for the Social
makers in exploring the potential of AI while remaining cognizant of
Sciences (SPSS) software, version 28.0.1.1 (14) developed by IBM.
students’ perspectives and needs.
Descriptive statistics were employed to address RQ1 and RQ2 by sum­
marizing the distribution of responses about chatbot usage and attitudes
3. Methodology

Table 1
3.1. Research approach
Descriptive statistics of the demographic characteristics of participants.

The methodology employed for this study follows a quantitative Characteristics Frequency Proportion
(%)
research paradigm, suitable for examining usage, attitudes, and corre­
lations across a large sample of participants (Creswell, 2014). Our Gender
research design centered on survey research. We applied single-item Male 2711 46.0
Female 2871 48.7
measures to keep the survey doable within less than 5 minutes. While Non-binary 86 1.5
this method introduces a limitation, as reliability measures cannot be n.d. 226 3.8
applied, it reduces respondent burden, survey fatigue and attrition. Academic Level
Additionally, past research supports the predictive validity of First cycle (undergraduate) 3950 67.0
Second cycle (graduate/masters) 1714 29.1
single-item measures for constructs that are concrete and easily and
Third cycle (postgraduate/PhD) 142 2.4
uniformly imagined (Bergkvist & Rossiter, 2007; Rossiter, 2002) as we n.d. 88 1.5
argue is the case in this study. Field of Study
The survey items, partly inspired by the groundwork laid by Weld­ Technology (including Engineering) 1933 32.8
ing’s (2023) study on AI usage in American colleges, were developed Social Sciences (including Law, Business and 1532 26.0
Pedagogy)
and piloted among a small sample of the target population with the Humanities (including Theology and Art) 998 17.0
objective to make sure the questions were understandable and to verify Medicine and health care 539 9.2
that the time it took to complete the survey. Subsequently, some minor Natural science 494 8.4
refinements were made to a few items based on the pilot feedback. The Other/n.d. 398 6.8
finalized survey was then launched through the online survey platform

1
It is important to recognize that gender is not a binary concept and many
2
individuals do not strictly identify as male or female. Unfortunately, our data on In the survey we list Bing AI and CoPilot as separate chatbots. As of
non-binary individuals was insufficient to be included in the main analysis, but November 2023 Microsoft rebranded Bing AI as CoPilot. However, at the time
the mean values for both chatbot usage and attitudes were consistently in- of the survey, those were still separate chatbots that we asked the respondents
between those of the male and female populations. about.

3
C. Stöhr et al. Computers and Education: Artificial Intelligence 7 (2024) 100259

towards chatbots. fewer (17.9%) believed that AI chatbots generated better results than
To answer RQ3, we applied chi-square tests to test for differences they could produce on their own.
between the sub-groups, complemented by Cramer’s V as non- Concerning ethical aspects and academic integrity, a majority of
parametric effect size measure and a post hoc evaluation of corrected students (61.9%) expressed the view that using chatbots to complete
standardized residuals. Those tests were suitable as they correspond to assignments and exams amounts to cheating. However, 58% disagreed
the categorial and ordinal nature of our variables without the need to with the statement that using chatbots goes against the purpose of ed­
meet the assumption of normality. Non-binary students were not ucation, and 60.3% disagreed with the prohibition of chatbots in
included in the analysis of gender effects due to the low response educational settings. Lastly, only 19.1% of the students reported that
frequency. their teachers or universities have rules or guidelines on the responsible
To address RQ4, we adopted the Kruskal-Wallis Test as a non- use of AI chatbots.
parametric equivalent of the one-way analysis of variance (ANOVA)
without the need to meet assumptions of normality and homogeneity of 4.2. Inferential analysis
variance. Post-hoc analyses were conducted using Mood’s Median Test
with a significance level of 0.05. and an additional component to Tables 4 and 5 present differences in chatbot usage and attitudes by
identify homogeneous subsets of our data. gender. The chi-square tests indicate statistically significant gender-
Finally, we conducted a correlational analysis to assess the rela­ based differences in familiarity and usage across all chatbots with
tionship between the usage of ChatGPT as the currently most popular AI weak to medium effects (between 0.2 and 0.38). Post-hoc analysis of the
chatbot in this analysis and students’ attitudes towards AI in education. corrected standardized residuals shows significant deviations from the
Again, we employed Spearman’s rho correlation as the best fit for the expected distribution in all categories with males consistently express­
ordinal nature of our data. ing high familiarity and usage than females. For ChatGPT in particular,
Overall, we chose a conservative approach as a rigorous statistical it is interesting to note that females were more likely than expected to be
foundation for our analysis. In choosing to analyze our data as ordinal, familiar with the chatbot but never actually use it (r = 13.7).
we minimize the risk of Type I errors (falsely rejecting the null hy­ Concerning attitudes towards chatbots (see Table 5), chi-square tests
pothesis) by prioritizing the inherent order of the response options, also reveal statistically significant gender differences across all attitude
while not imposing assumptions of continuity and interval equality of statements. The gender effects are generally weak (between 0.1 and 0.3)
the response options. However, this methodological choice comes with and manifest in different directions. Female respondents were ostensibly
the trade-off of potentially increased Type II errors, where we might fail more concerned about the impact of AI on education (r = 5.6), consid­
to detect actual differences or relationships due to the tests’ more ered the use of chatbots as potentially contrary to the purpose of edu­
stringent criteria for significance. Despite this, we believe that this cation (r = 9.3), and viewed the use of chatbots in assignments and
approach strengthens the validity of our findings. By adhering closely to exams as cheating (r = 6.1) that should be prohibited (r = 9.6). Males, on
the ordinal nature of our data, we reduce the interpretative leaps and the other hand, had an overall more positive attitude towards chatbots
assumptions required by parametric tests, such as ANOVA, which would (r = 14.8) and perceived them to a greater extent as tools that can
treat the data as quasi-continuous. At the same time, this ensures that improve their learning (r = 13.3) and grades (r = 12.1).
any observed differences or relationships are not artifacts of the Furthermore, we examined differences in chatbot familiarity, usage
analytical method but are indicative of the phenomena under study. By and attitudes by academic level (Tables 6 and 7). While the chi-square
opting for a conservative statistical approach, we aim to enhance the tests indicate statistically significant differences in the familiarity and
credibility and reliability of our findings, acknowledging the limitations usage of chatbots as well as most attitude statements across academic
of our data’s scale of measurement. levels, these effect sizes are generally very weak (<0.1), as indicated by
Cramer’s V.
4. Results The corrected standard residuals show that first-cycle students had a
greater likelihood of being unfamiliar with chatbots (r between 3.4 and
4.1. Descriptive analysis 7.4), particularly with Bing AI and Bard AI. They also reported lower
rare (r between − 2.5 and − 3.7) and regular chatbot usage (r between
The descriptive analysis of students’ usage of different AI chatbots, as − 2.2 and − 3.8, non-significant for Bard AI). Second-year students
illustrated in Table 2, reveals a varied usage pattern. The most generally showed a reversed trend with fewer students than expected
frequently used chatbot is ChatGPT, with 35.4% of students being being unfamiliar with chatbots across all five chatbots (r between − 3.6
familiar and regularly using it. It is the only chatbot with a Median and − 6.7). A higher proportion of second-year students used ChatGPT (r
higher than unfamiliar. By contrast, other AI chatbots such as YouChat, = 4.5) and CoPilot (r = 2.0) on regular base. Notably, third-cycle stu­
ChatSonic, DialoGPT, Socratic, and Jasper Chat were rarely or never dents, while showing no significant differences for ChatGPT and most
used by the majority of participants (in many cases, over 90% of the other categories, had significant positive residuals for the regular usage
respondents claimed to be unfamiliar with these other chatbots).3 of all other chatbots (r between 3.3 and 5.1) indicating a higher reliance
Table 3 provides a descriptive summary of students’ attitudes toward on AI technologies beyond the popular ChatGPT among this group.
the use of AI chatbots in education. More than half of the students With regard to attitudes towards chatbots (see Table 7), first-cycle
(55.9%) had a positive attitude towards the use of AI chatbots in edu­ students generally were more negative compared to other students.
cation. However, almost as many (54.2%) expressed concern about the Not only did we find less agreement towards an overall positive attitude
future impact of AI chatbots on students’ learning, with the result that (r = − 3.9), but also regarding the efficacy of chatbots in improving their
the Median is Agree for both statements. learning effectiveness (r = − 3.9), language ability (r = − 5.9), and study
Regarding the effect of AI chatbots on learning and performance, grades (r = − 4.6). Further, these students had stronger reservations
47.7% of respondents agreed that the chatbots they used made them about the role of chatbots in education. This is reflected in the ten­
more effective learners, whereas only 17.3% confirmed the positive ef­ dencies towards seeing the use of chatbots for completing assignments as
fect of chatbots in improving their study grades. Meanwhile, only 26.8% cheating (r = 3.4) that should be prohibited (r = 3.7) and that goes
thought that chatbots improved their general language abilities. Even against the purpose of education (r = 4.4).
In contrast, second-cycle students displayed overall more positive
attitudes towards chatbots (r = 3.9). For example, they were more likely
3
For that reason, YouChat, ChatSonic, DialoGPT, Socratic, and Jasper Chat to agree that chatbots enhance their effectiveness as learners (r = 4.0),
were excluded from the inferential analysis. improve their study grades (r = 4.3) as well as their language ability (r

4
C. Stöhr et al. Computers and Education: Artificial Intelligence 7 (2024) 100259

Table 2
Descriptive analysis of students’ usage of different chatbots.
Chatbots Familiar & regularly use it (4) Familiar but rarely use it (3) Familiar but never use it (2) Unfamiliar N Median
(1)

ChatGPT 2085 (35.4%) 1623 (27.6%) 1877 (31.0%) 297 (5.0%) 5882 3
Bing AI 136 (2.3%) 378 (6.5%) 1840 (31.6%) 3472 (59.6%) 5826 1
CoPilot 133 (2.3%) 172 (3.0%) 833 (14.3%) 4672 (80.4%) 5810 1
OpenAI playground 113 (1.9%) 316 (5.4%) 1274 (21.9%) 4108 (70.7%) 5811 1
Bard AI 17 (0.3%) 78 (1.3%) 1248 (21.4%) 4476 (76.9%) 5819 1
YouChat 13 (0.2%) 28 (0.5%) 464 (8.0%) 5306 (91.3%) 5811 1
ChatSonic 12 (0.2%) 52 (0.9%) 581 (10.0%) 5165 (88.9%) 5810 1
DialoGPT 11 (0.2%) 25 (0.4%) 333 (5.7%) 5432 (93.6%) 5801 1
Socratic 10 (0.2%) 43 (0.7%) 439 (7.6%) 5310 (91.5%) 5802 1
Jasper Chat 9 (0.2%) 41 (0.7%) 452 (7.8%) 5302 (91.4%) 5804 1

their language ability to a higher extent (r = 4.6), did not show any
Table 3 significant residuals at all.
Descriptive Analysis of Students’ Attitudes towards AI chatbots in Education.
Table 8 shows the results of a Kruskal-Wallis test examining the ef­
Statements Agree Don’t Disagree N Median fect of field of study on university students’ chatbot usage and attitudes.
(3) know/ (1)
The test value column shows that there were statistically significant
Prefer not
to say (2) differences between at least two groups of students with different fields
of study for all items with p < 0.001. Effect sizes are small to medium
General attitudes towards chatbots in education
The use of chatbots is 2272 2393 1206 5871 2
and tend to be particularly strong for the items related to usage of AI
common among my (38.7%) (40.8%) (20.5%) chatbots. The differences between the fields can be seen in the subset
fellow students. columns. Each subset column shows those study fields that build a ho­
Overall, I have a 3284 745 1844 5873 3 mogenous subgroup, meaning there were no significant differences be­
positive attitude (55.9%) (12.7%) (31.4%)
tween the responses from students from fields listed in the subgroup.
towards the use of
chatbots in Subsets and groups are ordered from low to high, meaning that the
education. group on the left has the lowest usage or agreement and the highest can
I am concerned about 3186 722 1969 5877 3 be found on the right.
how AI-chatbots (54.2%) (12.3%) (33.5%) For example, for ChatGPT, medicine and healthcare students were
will impact
students’ learning
least familiar with it, followed by students in the humanities. The dif­
in the future. ferences between the students in these fields were not statistically sig­
Effects of chatbots on learning and performance nificant, so they build subgroup 1 with the lowest familiarity of
The chatbots I use 2791 1552 1512 5855 2 ChatGPT. In the same way, students from the natural sciences and social
make me more (47.7%) (26.5%) (25.8%)
sciences showed significantly higher usage of ChatGPT than the former
effective as a
learner. two, building subgroup 2. Students of technology and engineering
The chatbots I use 1570 1711 2575 5856 2 expressed significantly higher familiarity with ChatGPT than all other
improve my general (26.8%) (29.2%) (44.0%) groups and are therefore alone in subgroup 3. Students from a certain
language ability. field of study can also belong to several subgroups (as for example for
Chatbots generate 1049 1856 2952 5857 1
Bard AI). Here, students from the natural sciences showed higher usage
better results than I (17.9%) (31.7%) (50.4%)
can produce on my than students from medicine, humanities and social sciences, but the
own. difference was only statistically significant when compared to the lowest
The chatbots I use 1013 2477 2365 5855 2 group, medicine students. Thus, students of humanities and social sci­
improve my study (17.3%) (42.3%) (40.4%)
ences are both in subgroup 1 (with medicine and healthcare students)
grades.
Ethical aspects and academic integrity and 2 (with natural sciences students). Looking at all the items in the
Using chatbots goes 1638 833 3406 5877 1 survey, some consistent patterns emerge. Students in technology and
against the purpose (27.9%) (14.2%) (58.0) engineering clearly stood out from all other groups. They used chatbots
of education. to a statistically significantly higher degree and had the least concerns
Using chatbots to 3633 813 1424 5870 3
about the ethical aspects of AI usage in education. Notably, for most
complete (61.9%) (13.9%) (24.3%)
assignments and items, the difference is statistically significant in relation to all other
exams is cheating. academic fields. Conversely, students from the arts and humanities as
Using chatbots should 1377 951 3539 5867 1 well as medicine and health care showed the opposite pattern; students
be prohibited in (23.5%) (16.2%) (60.3%)
from those fields were significantly less familiar with ChatGPT. They
educational
settings. also expressed more concern, an overall less positive attitude towards
My teacher(s) or 1120 3229 1519 5868 2 the use of chatbots in education, and more students from these groups
university has rules (19.1%) (55.0%) (25.9%) believed that the use of chatbots goes against the purpose of education.
or guidelines on the They were also more supportive of a prohibition of chatbots in
responsible use of
education.
chatbots.

= 4.5). They also expressed more disagreement with ethical concerns 4.3. Correlational analysis
such as perceiving the use of chatbots as going against the purpose of
education (r = 5.9), as cheating (r = 2.6), or that chatbots should be Table 9 shows the Spearman’s rho correlation coefficients between
prohibited (r = 6.0). the usage of ChatGPT and attitudes towards AI in education and chatbot
Third-cycle students, besides perceiving chatbots as contributing to usage. The results show a statistically significant relation for all attitude
questions except one. The use of ChatGPT is strongly positively

5
C. Stöhr et al. Computers and Education: Artificial Intelligence 7 (2024) 100259

Table 4
Chi-square and corrected standardized residuals for Chatbot Usage by Gender.
Chatbot familiarity and χ2 p-value Cramer’s N Corrected Standard Residuals r*
usage V

df = 3 Unfamiliar Familiar but never Familiar but rarely Familiar and regularly
use it use it use it

ChatGPT 420.079 <0.001 0.275 5574 Male − 11.2 − 13.7 2.3 16.2
Female 11.2 13.7 − 2.3 − 16.2
Bing AI 811.955 <0.001 0.383 5523 Male − 28.0 20.0 13.1 7.8
Female 28.0 − 20.0 − 13.1 − 7.8
CoPilot 393.351 <0.001 0.267 5510 Male − 19.3 13.5 8.7 9.7
Female 19.3 − 13.5 − 8.7 − 9.7
OpenAI playground 228.143 <0.001 0.204 5509 Male − 14.5 9.6 9.2 3.8
Female 14.5 − 9.6 − 9.2 − 3.8
Bard AI 696.552 <0.001 0.355 5518 Male − 26.4 25.5 4.5 2.1
Female 26.4 − 25.5 − 4.5 − 2.1

*all differences are statistically significant (p < 0.005, corrected standardized residuals ± 1.96).

Table 5
Differences in chatbot attitudes by gender.
Attitudes towards chatbots χ2 p-value Cramer’s V N Corrected Standard Residuals r*

df = 2 Disagree Don’t Agree

know

The use of chatbots is common among my fellow students. 88.556 <0.001 0.126 5564 Male − 3.2 − 6.7 9.4
Female 3.2 6.7 − 9.4
Overall, I have a positive attitude towards the use of chatbots in education. 220.849 <0.001 0.199 5567 Male − 10.3 − 7.8 14.8
Female 10.3 7.8 − 14.8
I am concerned about how AI-chatbots will impact students’ learning in the future. 68.817 <0.001 0.111 5571 Male 8.2 − 3.4 − 5.6
Female − 8.2 3.4 5.6
The chatbots I use make me more effective as a learner. 176.018 <0.001 0.178 5550 Male − 7.5 − 7.6 13.3
Female 7.5 7.6 − 13.3
The chatbots I use improve my general language ability. 62.072 <0.001 0.105 5552 Male − 5.4 7.4
Female 5.4 − 7.4
Chatbots generate better results than I can produce on my own. 86.708 0.003 0.125 5552 Male − 6.4 8.5
Female 6.4 − 8.5
The chatbots I use improve my study grades. 146.995 <0.001 0.163 5552 Male − 5.1 − 4.2 12.1
Female 5.1 4.2 − 12.1
Using chatbots goes against the purpose of education. 171.801 <0.001 0.176 5570 Male 13.1 − 6.7 − 9.3
Female − 13.1 6.7 9.3
Using chatbots to complete assignments and exams is cheating. 68.334 <0.001 0.111 5564 Male 8.3 − 1.7 − 6.1
Female − 8.3 1.7 6.1
E3. Using chatbots should be prohibited in educational settings. 174.644 <0.001 0.177 5561 Male 13.2 − 6.6 − 9.6
Female − 13.2 6.6 9.6
E4. My teacher(s) or university has rules or guidelines on the responsible use of 10.663 0.005 0.044 5563 Male − 3.2 2.1
chatbots. Female 3.2 − 2.1

*only statistically significant differences are listed (p < 0.005, corrected standardized residuals ± 1.96).

associated with students’ belief that the use of chatbots is common education, following the introduction of potentially disruptive large
among fellow students (rho = 0.406), their positive attitude towards language models, particularly ChatGPT, on a wide scale. Through this
chatbots in education (rho = 0.581), and the belief that chatbots make exploration, a complex landscape has been unveiled, wherein multiple
them more effective learners (rho = 0.644). A weak positive correlation factors interact to shape students’ perceptions and behaviors. Notably,
was found between the usage of ChatGPT and the belief that chatbots the analysis has provided a more nuanced understanding of the roles
improve general language ability (rho = 0.197) and study grades (rho = that gender, discipline, and academic level play in this new learning
0.189). Medium and strong inverse correlations were found between the context.
usage of ChatGPT and concern about AI-chatbots’ impact on students’ First, we were interested in the general usage frequency and famil­
learning in the future (rho = − 0.304), between the views that using iarity with different chatbots among students. Here, the hype around
chatbots contradicts the purpose of education (rho = − 0.557), consti­ ChatGPT is reflected in our data since 95% of the respondents were
tutes cheating (rho = − 0.327), and that chatbots should be prohibited in familiar with ChatGPT and more than one third (35.4%) claimed to use
educational settings (rho = − 0.564). Finally, a weak negative correla­ it regularly. This was confirmed by a similar proportion of students
tion was found between ChatGPT usage and their awareness of rules or stating that the use of chatbots is common among their fellow students.
guidelines on the responsible use of chatbots (rho = − 0.049). No sta­ With regard to the true representativeness of the total student popula­
tistically significant correlation was found between ChatGPT usage and tion, these numbers need to be interpreted with caution, as students
the belief that chatbots generate better results than the respondents already familiar with chatbots might be somewhat more likely to
could produce on their own. participate in the survey compared to unfamiliar ones. Still, our results
indicate an increase in familiarity and usage compared to earlier reports
5. Discussion (Lobet et al., 2023; Vogels, 2023; Welding, 2023) that would suffer from
a similar bias, thus supporting the claim that ChatGPT, despite its nov­
This study attempted to empirically investigate university students’ elty and limitations, has quickly attained more widespread recognition
AI chatbot usage and attitudes towards artificial intelligence in and use among students in higher education. Interestingly though, this

6
C. Stöhr et al. Computers and Education: Artificial Intelligence 7 (2024) 100259

Table 6
Differences in chatbot usage by academic level.
Chatbot familiarity and χ2 p-value Cra mer’s N Corrected Standard Residuals r*
usage V

df = 6 Unfamiliar Familiar but never Familiar but rarely Familiar and regularly
use it use it use it

ChatGPT 60.152 <0.001 0.072 5798 First cycle 3.4 5.9 − 2.8 − 4.8
Second − 3.8 − 5.9 3.1 4.5
cycle
Third cycle
Bing AI 73.068 <0.001 0.080 5744 First cycle 7.4 − 5.7 − 2.5 − 2.4
Second − 6.7 5.8
cycle
Third cycle − 2.7 2.7 3.8
CoPilot 43.213 <0.001 0.061 5728 First cycle 4.6 − 2.2 − 3.7 − 3.0
Second − 3.6 2.7 2.0
cycle
Third cycle − 3.4 3.0 3.3
OpenAI playground 41.288 <0.001 0.060 5729 First cycle 5.2 − 3.0 − 3.6 − 2.2
Second − 5.4 3.6 3.3
cycle
Third cycle 5.1
Bard AI 84.369 <0.001 0.086 5737 First cycle 7.4 − 6.8 − 2.7
Second − 6.4 6.3
cycle
Third cycle − 3.4 3.8 4.1

*only statistically significant differences are listed (p < 0.005, corrected standardized residuals ± 1.96).

popularity appeared to be restricted to ChatGPT in particular and did Regarding the relationship between the familiarity and usage of AI
not expand to other chatbots at the time of the survey. Some chatbots chatbots and student attitudes, our findings provide empirical support
from larger companies, such as Bing AI and Bard, were also familiar to a for the predicted relationship between students’ attitudes towards AI
substantial proportion of university students in Sweden, but most had chatbots and their level of familiarity and usage. We found a strong
never used any of the other applications. While this pattern does not positive correlation between ChatGPT familiarity and usage and positive
apply to AI applications in general (see e.g., Malmström et al., 2023) and attitudes towards chatbots as well as perceived benefits from their use.
might change in the future, it underscores ChatGPT’s prominent role as Conversely, experience with ChatGPT was strongly negatively corre­
the key driving force behind the ongoing popularization of generative AI lated with concerns about the impact of AI on future learning and ethical
in education. It also stresses the urgent need for educators and educa­ concerns surrounding chatbot usage in education. These findings are
tional institutions to adapt education to this new situation and find ways consistent with the broader predictions of UTAUT (Venkatesh et al.,
to address the potentials and challenges connected to this technology. At 2003) and other empirical research (e.g., Alzahrani, 2023; Qin et al.,
the time of the survey, more than four out of five students were not 2020). They underscore the importance of the relationship between
aware of any rules or guidelines from teachers or their universities in exposure and hands-on experience and beliefs about technological in­
this regard. novations in educational contexts.
Examining students’ attitudes towards chatbots more specifically, we To what extent do chatbot usage and attitudes differ for different
saw that while over half of the respondents expressed an overall positive groups of students? Our results found statistically significant and
attitude towards the use of chatbots in education, almost as many consistent differences in the responses for all three examined grouping
expressed concerns about their impact in the future. This disparity of variables (gender, academic level and field of study). Regarding gender,
optimism and concern highlights the complex relationship that students we found particularly strong differences in familiarity with and usage of
have with this emerging technology. With regard to the effect of chat­ chatbots. Female students were overall more negative and concerned
bots on learning and performance, almost half of the students indicated about the impact of AI on learning and assessment. Thus, as predicted by
that chatbots make them more effective as learners, pointing at the UTAUT (Venkatesh et al., 2003) and confirmed by recent AIED empirical
potential but also the ease with which ChatGPT seems to be utilized as research (Nouraldeen, 2022), our results indicate that gender is still a
self-study tool, facilitator, or assistant for learning (Chen et al., 2023; relevant factor for AI adoption for the current student generation and in
Nisar & Aslam, 2023; Rahman & Watanobe, 2023). the context of ChatGPT; this may be indicative of underlying societal
Nevertheless, fewer than one in five of the students felt that chatbots factors or personal experiences that necessitate further investigation.
produced better results or improved their grades. This could indicate Moreover, these results prompt an essential reflection on the design and
that most students use chatbots as a complement in the learning process implementation of educational technology. If AIED tools like ChatGPT
rather than to complete assignments and exams, though the high num­ are to be effectively utilized and integrated across diverse student
ber of students choosing “don’t know/prefer not to say” requires caution populations, the identified differences must be acknowledged and
when interpreting these results. However, these findings receive some addressed. Gender-sensitive approaches, tailored interventions, and in­
confirmation when questions about academic integrity are considered: clusive design principles may be required to ensure that AI-powered
more than sixty percent believed that the use of chatbots during exam­ educational solutions cater to the unique needs and preferences of
ination is cheating, Nonetheless, a majority of students were against the various student demographics.
prohibition of AI in education settings and neither thought that chatbots The effect of academic level, even though existent, was very weak
go against the purpose of education. Thus, it appears that many students (the effect sizes were consistently small). Consistent with Abu-Shanab
were aware of the potential of chatbots to support the actual learning (2011), the post-hoc analysis of the residuals showed that
process, and more insight is needed on precisely how students use advanced-level students exhibited slightly greater familiarity and usage.
chatbots in practice. Qualitative research could potentially address this Second-cycle students were more likely to be users of ChatGPT, while
aspect. third-cycle students showed a higher tendency to regularly use other AI

7
C. Stöhr et al. Computers and Education: Artificial Intelligence 7 (2024) 100259

Table 7
Differences in chatbot attitudes by academic level.
Attitudes towards chatbots χ2 p-value Cramer’s V N Corrected Standard Residuals r*

df = 4 Dis- Don’t Agree

agree know

The use of chatbots is common among my fellow students. 23.386 <0.001 0.045 5788 First cycle 3.2 − 3.4
Second − 3.5 4.1
cycle
Third cycle
Overall, I have a positive attitude towards the use of chatbots in education. 15.881 0.003 0.037 5791 First cycle 2.8 − 3.9
Second − 3.0 3.9
cycle
Third cycle
I am concerned about how AI-chatbots will impact students’ learning in the 7.120 0.130 n.s. 5795 First cycle
future. Second
cycle
Third cycle
The chatbots I use make me more effective as a learner. 18.662 <0.001 0.040 5772 First cycle 3.5 − 3.9
Second − 3.6 4.0
cycle
Third cycle
The chatbots I use improve my general language ability. 48.496 <0.001 0.065 5774 First cycle 2.2 3.3 − 5.9
Second − 3.0 4.5
cycle
Third cycle − 3.1 4.6
Chatbots generate better results than I can produce on my own. 22.444 <0.001 0.044 5775 First cycle 4.4 − 3.1
Second − 4.1 2.8
cycle
Third cycle
The chatbots I use improve my study grades. 26.534 <0.001 0.048 5773 First cycle 2.1 − 4.6
Second − 2.0 4.3
cycle
Third cycle − 2.5
Using chatbots goes against the purpose of education. 38.992 <0.001 0.058 5795 First cycle − 6.2 3.1 4.4
Second 5.9 − 2.9 − 4.2
cycle
Third cycle
Using chatbots to complete assignments and exams is cheating. 12.051 0.017 0.032 5788 First cycle − 2.8 3.4
Second 2.6 − 3.1
cycle
Third cycle
Using chatbots should be prohibited in educational settings. 39.510 <0.001 0.058 5785 First cycle − 6.1 3.9 3.7
Second 6.0 − 3.5 − 3.9
cycle
Third cycle
My teacher(s) or university has rules or guidelines on the responsible use of 5.597 0.231 n.s. 5787 First cycle
chatbots. Second − 2.1
cycle
Third cycle

*only statistically significant differences are listed (p < 0.005, corrected standardized residuals ± 1.96).

chatbots. These effects could be linked to more specialized AI utilization students generally have positive attitudes towards AI in medicine
opportunities in advanced, research-based projects, or generally higher (Santomartino & Yi, 2022). Potentially, our students’ skepticism is
self-regulated learning skills among graduate and PhD students. Further, connected to chatbots in particular or reflects more on the health care
we found more favorable attitudes and more limited concerns about sector as a whole than medicine alone. However, this should be
using chatbots in education for second-year students, along with the considered in the sometimes euphoric discussion about the potential of
opposite tendency for first-year students. Interestingly, third-cycle stu­ ChatGPT in medical education (e.g., Lee, 2023). The consistency and
dents did not show the same attitude patterns as second-year students, decent effect sizes of our results also raise the question of the importance
indicating a potential non-linear relation between academic level and of academic disciplines and differences in disciplinary traditions and
attitudes towards chatbots. However, given the paucity of empirical practices as explanatory factors in AIED in general. Our findings moti­
studies on this question, more research is required to draw definitive vate further examination of this perspective, which is not well theorized
conclusions. and so far largely lacks empirical verification (see Orji, 2010 for an
As for disciplinary differences, our results indicate clear and exception).
consistent differences, with students from engineering apparently using Generally, our findings also have pedagogical implications for
chatbots to a much higher degree and expressing stronger optimism teachers and academic institutions. Students’ AI literacy development
towards AI. Conversely, students from the arts and humanities and from enabling them to critically evaluate, communicate with, and use AI
medicine and health care used chatbots less and were more skeptical. technologies (Long & Magerko, 2020) are likely to be significantly
While the results for humanities and arts confirm prior research (Mer­ shaped by the ways in which teachers insert and discuss AI technologies
cader & Gairín, 2020), the results for students from the medicine and in their teaching practices. Our results suggest that AI-related policies
healthcare sectors are surprising, given the fact that it is one of the fields and guidelines in that regard should not be one-size-fits-all; rather,
most actively discussing AI adoption (e.g., Zhang et al., 2023). Our support efforts need adaptation to the student characteristics and
findings also contradict the results of prior research reporting that teaching methods in specific disciplinary contexts. Thus, solutions to

8
C. Stöhr et al. Computers and Education: Artificial Intelligence 7 (2024) 100259

Table 8
Differences in chatbot usage and attitudes by field of study (kruskal-wallis test with homogeneous subgroups (p < 0.05), grouping from low to high usage and disagree
to agree).
Kruskal-Wallis H (df = Effect size Subset 1 Subset 2 Subset 3 Subset 4 N
4) ε2
Chatbot familiarity and usage
ChatGPT 589.489* 0,107 MH NS T 5489
Bing AI 454.283* 0,083 M SH N T 5440
CoPilot 511.305* 0,094 MSH N T 5424
OpenAI playground 154.880* 0,028 MHSN T 5426
Bard AI 360.236* 0,066 MHS HSN T 5435
Attitudes towards chatbots
The use of chatbots is common among my fellow students. 434.643* 0,079 HM NS T 5479
Overall, I have a positive attitude towards the use of chatbots in education. 306.434* 0,055 H M SN T 5481
I am concerned about how AI-chatbots will impact students’ learning in the 94.910* 0,017 T NS MH 5487
future.
The chatbots I use make me more effective as a learner. 389.312* 0,071 H M NS T 5466
The chatbots I use improve my general language ability. 52.291* 0,009 HS SMN T 5466
Chatbots generate better results than I can produce on my own. 13.887* 0,002 HSNM NMT 5468
The chatbots I use improve my study grades. 101.288* 0,018 HMSN T 5466
Using chatbots goes against the purpose of education. 280.126* 0,050 T NS HM 5486
Using chatbots to complete assignments and exams is cheating. 91.378* 0,016 T SN NMH 5480
Using chatbots should be prohibited in educational settings. 330.042* 0,060 T NS HM 5478
My teacher(s) or university has rules or guidelines on the responsible use of 21.084* 0,003 TSM HN 5478
chatbots.

* … p < 0.001, M … Medicine and healthcare, H … Humanities and art, N … Natural sciences, S … Social sciences, T … Technology and Engineering (homogeneous
subgroups are based on asymptotic significances with α = 0.05).

handling AI in education should be developed and implemented different student populations will require locally adapted solutions.
“locally” to address the specific needs of the local student population. Given the apparent lack of guidelines and rules, many teachers and
For example, the widespread enthusiasm and experience with AI chat­ decision-makers would appear to be unprepared to this end.
bots among technology and engineering students can provide an op­ This study also highlights the importance of addressing students’
portunity to integrate AI chatbot tasks more deeply into curricula, while concerns about the potential impact of AI on their future learning.
at the same time requiring particular efforts to create awareness of the Ongoing lively discussions about the potential and dangers of AI in ed­
technology’s limitations and ethical concerns. In other fields, different ucation need to be complemented with empirical studies of the kind
instruction strategies might be required to address a lack of comfort presented here. In addition, qualitative research is needed to better
and/or proficiency with AI chatbots, for example through feedback understand how students use AI tools in practice. Ultimately, the find­
mechanisms, allowing students to share their concerns, experiences, and ings from this study contribute to the growing body of literature on the
suggestions about AI chatbot usage. While there will always be variation role of AI in education and provide a valuable resource for developers,
between individual students that needs to be considered, our results can educators and policymakers as they navigate the emerging landscape of
help teachers adapt their handling of AI tools to their particular teaching AI chatbots within the educational sector. It is also important to
contexts. acknowledge that the data underpinning our study was collected in May
of 2023, marking a specific snapshot in the rapidly evolving landscape of
6. Conclusions and limitations AI. As such, the reported AI chatbot usage and attitudes are likely to be
subject to change as new applications emerge and awareness grows.
Almost 6000 university students in Sweden answered survey ques­ Thus, the temporal context serves both as a limitation and a springboard
tions about their usage and attitudes towards ChatGPT and other AI for future research highlighting the need for continuous investigation to
chatbots. The insights gleaned from this research underscore the update and adjust the observed findings. Certain further limitations of
importance of understanding student perceptions and experiences of AI the present study must be acknowledged. First, the sample used in this
chatbots in educational settings. The study has revealed the widespread study was not random. The respondents self-selected to participate, and
usage of ChatGPT among university students. Given that we are likely the topic of the survey might make students who already had some
only seeing the beginning of large language model applications, we degree of exposure to chatbots in their academic settings more likely
agree with other educators and AIED researchers to conclude that the choose to engage. This potential selection bias could somewhat over­
use of ChatGPT and other chatbots in education among students is state the familiarity and usage of AI chatbots. Despite these limitations,
already mainstream and likely to stay (Hajkowicz et al., 2023). While the large sample size used in this study, encompassing thousands of
concerns about academic integrity and cheating are valid and justified, responses, enhances the statistical power of the analysis and allows for
many students acknowledge the usefulness of AI for their actual the detection of even small effect sizes. Furthermore, the broad mix of
learning, and our efforts should be directed towards supporting these respondents from different academic levels and genders provides a
developments. Students still need substantive training and learning, and heterogeneous sample that has offered a rich view into the range of
ChatGPT should be treated as a tool rather than a replacement (Berda­ student experiences and attitudes towards AI chatbots. This was enabled
nier & Alley, 2023), but both students and teachers need new compe­ through the use of single-response items instead of larger attitudinal
tencies in integrating AI chatbots in the learning process. constructs, which may raise concerns about the reliability of our
Case-based inspiration and examples of how AI chatbots can pro­ approach. Thus, while we feel that the use of single items is justified in
ductively support learning are currently published on a daily basis (e.g., this particular study to provide a particular, readily interpretable
Santos, 2023). Nevertheless, some students benefit more from these snapshot of certain attitudes and behaviors, we encourage future
developments than others. Our results show multifaceted and sometimes research to build on this work by employing established or newly
conflicting views on the role of AI in education, and these views are developed multidimensional scales for a more comprehensive under­
influenced by gender and academic discipline. Addressing the needs of standing of the factors that drive students’ perceptions and interactions

9
C. Stöhr et al. Computers and Education: Artificial Intelligence 7 (2024) 100259

Table 9 Statements on ethics and open data

Correlation between usage of ChatGPT and attitudes towards AI in education
and chatbot usage. The data can be obtained by sending request e-mails to the corre­
Spearman’s p-value 95% 95% N sponding author. All procedures in the study were conducted in accor­
rho lower upper dance with applicable laws and institutional guidelines for research
bound bound ethics ensuring strict adherence to ethical principles throughout this
The use of chatbots is 0.406 <0.001 0.384 0.428 5869 study. No personally identifiable or sensitive data were collected,
common among my ensuring total participant anonymity. Detailed information about the
fellow students.
study was provided to participants before beginning the survey, reas­
Overall, I have a 0.581 0.000 0.563 0.598 5871
positive attitude suring them of their right to voluntary participation and withdrawal. All
towards the use of participants provided their informed consent for participation when
chatbots in submitting the survey.
education.
I am concerned about − 0.304 <0.001 − 0.328 − 0.280 5875
how AI-chatbots CRediT authorship contribution statement
will impact
students’ learning Christian Stöhr: Writing – review & editing, Writing – original draft,
in the future.
Project administration, Methodology, Investigation, Funding acquisi­
The chatbots I use 0.644 0.000 0.628 0.659 5853
make me more
tion, Formal analysis, Data curation, Conceptualization. Amy Wanyu
effective as a Ou: Writing – review & editing, Writing – original draft, Methodology,
learner. Investigation, Data curation, Conceptualization. Hans Malmström:
The chatbots I use 0.197 <0.001 0.172 0.223 5855 Writing – review & editing, Writing – original draft, Project adminis­
improve my general
tration, Methodology, Investigation, Data curation, Conceptualization.
language ability.
Chatbots generate − 0.009 0.492 − 0.035 0.017 5856
better results than I Declaration of generative AI and AI-assisted technologies in the
can produce on my writing process
own.
The chatbots I use 0.189 <0.001 0.164 0.215 5854
improve my study Statement: During the preparation of this work, the authors used
grades. ChatGPT in order to improve readability and language in certain places
Using chatbots goes − 0.557 0.000 − 0.575 − 0.539 5875 of the text. After using this tool/service, the authors reviewed and edited
against the purpose
the content as needed and take full responsibility for the content of the
of education.
Using chatbots to − 0.327 <0.001 − 0.350 − 0.303 5868 publication.
complete
assignments and
exams is cheating. Declaration of competing interest
Using chatbots should − 0.564 0.000 − 0.582 − 0.546 5865
be prohibited in The authors declare that they have no known competing financial
educational
settings.
interests or personal relationships that could have appeared to influence
My teacher(s) or − 0.049 <0.001 − 0.075 − 0.022 5867 the work reported in this paper.
university has rules
or guidelines on the
Acknowledgements
responsible use of
chatbots.
The authors of this report express their sincere gratitude for the
assistance and support provided by the following individuals: Jenny
with AI chatbots in educational settings. Our study serves as a step­ Palm (Chalmers University of Technology); Jörg Pareigis (Karlstad
pingstone, indicating that these larger attitudinal patterns exist and University); Malin Kjellberg and Andreas Eriksson (Chalmers University
warrant deeper exploration through more robust measurement tools and of Technology); all people involved in the promotion and distribution of
the investigation of explanatory factors such as teaching practices, the survey at the different universities and the anonymous reviewers for
educator attitudes, and the overall educational context and their influ­ their constructive feedback. This research did not receive any specific
ence on technology adoption and educational outcomes. Future research grant from funding agencies in the public, commercial, or not-for-profit
might also consider the use of other data collection methods such as sectors.
direct observation, logs of actual chatbot use, or focus group discussions
to reduce bias due to social desirability or inaccurate recall in self- Appendix A. Supplementary data
reported data that is common in studies of this kind. Related to this,
the cross-sectional design of the study does not allow for the exploration Supplementary data to this article can be found online at https://doi.
of causal relationships or changes over time. Future research could org/10.1016/j.caeai.2024.100259.
benefit from a longitudinal design, which would provide insights into
the evolution of students’ perceptions and usage of AI chatbots over
References
time. Despite these limitations, this study has provided a robust starting
point for further exploration into this rapidly evolving field of study. The Abu-Shanab, E. A. (2011). Education level as a technology adoption moderator. 2011 3rd
results provide meaningful cues for stakeholders involved in integrating International Conference on Computer Research and Development, 1, 324–328. https://
and regulating AI chatbots in higher education, underscoring the crucial doi.org/10.1109/ICCRD.2011.5764029. IEEE.
Adiguzel, T., Kaya, M. H., & Cansu, F. K. (2023). Revolutionizing education with AI:
role of student perspectives in the discourse about educational Exploring the transformative potential of ChatGPT. Contemporary Educational
technologies. Technology, 15(3), ep429. https://doi.org/10.30935/cedtech/13152
Ajzen, I. (1991). The theory of planned behavior. Organizational Behavior and Human
Decision Processes, 50(2), 179–211. https://doi.org/10.1016/0749-5978(91)90020-T
Alzahrani, L. (2023). Analyzing students’ attitudes and behavior toward artificial
intelligence technologies in higher education. International Journal of Recent

10
C. Stöhr et al. Computers and Education: Artificial Intelligence 7 (2024) 100259

Technology and Engineering, 11(6), 65–73. https://doi.org/10.35940/ijrte. Language Learning and Teaching, 0(0), 1–14. https://doi.org/10.1080/
F7475.0311623 17501229.2023.2240316
Bates, T., Cobo, C., Mariño, O., & Wheeler, S. (2020). Can artificial intelligence transform Lo, C. K. (2023). What is the impact of ChatGPT on education? A rapid review of the
higher education? International Journal of Educational Technology in Higher Education, literature. Education Sciences, 13(4), 410. https://doi.org/10.3390/educsci13040410
17(1), 1–12. https://doi.org/10.1186/s41239-020-00218-x Lobet, M., Honet, A., & Wathelet, V. (2023). Use of ChatGPT by students: Artificial
Berdanier, C. G. P., & Alley, M. (2023). We still need to teach engineers to write in the era intelligence is all the rage!. https://newsroom.unamur.be/en/news/chatgpt-by-s
of ChatGPT. Journal of Engineering Education, 2023(112), 583–586. https://doi.org/ tudents-ai-all-rage.
10.1002/jee.20541 Long, D., & Magerko, B. (2020). What is AI literacy? Competencies and design
Bergkvist, L., & Rossiter, J. R. (2007). The predictive validity of multiple-item versus considerations. Proceedings of the 2020 CHI Conference on Human Factors in Computing
single-item measures of the same constructs. Journal of Marketing Research, 44(2), Systems, 1–16. https://doi.org/10.1145/3313831.3376727
175–184. https://doi.org/10.1509/jmkr.44.2.175 Mercader, C., & Gairín, J. (2020). University teachers’ perception of barriers to the use of
Bogost, I. (2022). ChatGPT is Dumber than you think. The atlantic. https://www.theatlantic digital technologies: The importance of the academic discipline. International Journal
.com/technology/archive/2022/12/chatgpt-openai-artificial-intelligence-writing of Educational Technology in Higher Education, 17(1), 4. https://doi.org/10.1186/
-ethics/672386/. s41239-020-0182-x
Chen, L., Chen, P., & Lin, Z. (2020). Artificial intelligence in education: A review. IEEE Morris, M. G., Venkatesh, V., & Ackerman, P. L. (2005). Gender and age differences in
Access, 8, 75264–75278. https://doi.org/10.1109/ACCESS.2020.2988510 employee decisions about new technology: An extension to the theory of planned
Chen, Y., Jensen, S., Albert, L. J., Gupta, S., & Lee, T. (2023). Artificial intelligence (AI) behavior. IEEE Transactions on Engineering Management, 52(1), 69–84. https://doi.
student assistants in the classroom: Designing chatbots to support student success. org/10.1109/TEM.2004.839967.
Information Systems Frontiers, 25(1), 161–182. https://doi.org/10.1007/s10796-022- Malmström, H., Stöhr, C., & Ou, A.W. (2023). Chatbots and other AI for learning: A survey
10291-4 of use and views among university students in Sweden. (Chalmers Studies in
Chiu, T. K. F., Xia, Q., Zhou, X., Chai, C. S., & Cheng, M. (2023). Systematic literature Communication and Learning in Higher Education 2023:1) https://doi.or
review on opportunities, challenges, and future research recommendations of g/10.17196/cls.csclhe/2023/01.
artificial intelligence in education. Computers & Education: Artificial Intelligence, 4, Nemorin, S., Vlachidis, A., Ayerakwa, H. M., & Andriotis, P. (2023). AI hyped? A horizon
100–118. https://doi.org/10.1016/j.caeai.2022.100118 scan of discourse on artificial intelligence in education (AIED) and development.
Cotton, D. R. E., Cotton, P. A., & Shipway, J. R. (2023). Chatting and cheating: Ensuring Learning, Media and Technology, 48(1), 38–51. https://doi.org/10.1080/
academic integrity in the era of ChatGPT. Innovations in Education & Teaching 17439884.2022.2095568
International, 1–12. https://doi.org/10.1080/14703297.2023.2190148 Nisar, S., & Aslam, M. S. (2023). Is ChatGPT a good tool for T&CM students in studying
Creswell, J. W. (2014). Research design: Qualitative, quantitative, and mixed methods pharmacology? SSRN Scholarly Paper (4324310). https://doi.org/10.2139/
approaches. Sage Publications. ssrn.4324310
Davis, F. D. (1989). Perceived usefulness, perceived ease of use, and user acceptance of Nouraldeen, R. M. (2022). The impact of technology readiness and use perceptions on
information technology. MIS Quarterly, 13(3), 319–340. https://doi.org/10.2307/ students’ adoption of artificial intelligence: The moderating role of gender.
249008 Development and Learning in Organizations: An International Journal, 37(3), 7–10.
Deng, J., & Lin, Y. (2022). The benefits and challenges of ChatGPT: An overview. https://doi.org/10.1108/DLO-07-2022-0133
Frontiers in Computing and Intelligent Systems, 2(2), 81–83. https://doi.org/10.54097/ OpenAI. (2022). Introducing ChatGPT. Retrieved July 11, 2023, from https://openai.co
fcis.v2i2.4465 m/blog/chatgpt.
Dennean, K., Gantori, S., Limas, D. K., Pu, A., & Gilligan, R. (2023). Let’s chat about Orji, R. O. (2010). Effect of academic discipline on technology acceptance. In 2010
ChatGPT. UBS https://secure.ubs.com/public/api/v2/investment-content/documen international conference on education and management technology (pp. 617–621). IEEE.
ts/XILxY9V9P5RazGpDA1Cr_Q?apikey=Y8VdAx8vhk1P9YXDlEOo2Eoco1fqKwD https://doi.org/10.1109/ICEMT.2010.5657581.
k&Accept-Language=de-CH. Qin, F., Li, K., & Yan, J. (2020). Understanding user trust in artificial intelligence-based
Dijkstra, R., Genc, Z., Kayal, S., & Kamps, J. (2022). Reading comprehension quiz educational systems: Evidence from China. British Journal of Educational Technology,
generation using generative pre-trained transformers. In S. Sosnovsky, 51(5), 1693–1710. https://doi.org/10.1111/bjet.12994
P. Brusilovsky, & A. Lan (Eds.), Proceedings of the fourth international workshop on Rahman, M. M., & Watanobe, Y. (2023). ChatGPT for education and research:
intelligent textbooks 2022: Co-located with 23d international conference on artificial Opportunities, threats, and strategies. Applied Sciences, 13(9), 5783. https://doi.org/
intelligence in education (AIED 2022) (pp. 4–17). http://ceur-ws.org/Vol-3192/it 10.3390/app13095783
b22_p1_full5439.pdf. Rossiter, J. R. (2002). The C-OAR-SE procedure for scale development in marketing.
Eke, D. O. (2023). ChatGPT and the rise of generative AI: Threat to academic integrity? International Journal of Research in Marketing, 19(4), 305–335. https://doi.org/
Journal of Responsible Technology, 13, Article 100060. https://doi.org/10.1016/j. 10.1016/S0167-8116(02)00097-6
jrt.2023.100060 Rudolph, J., Tan, S., & Tan, S. (2023). ChatGPT: Bullshit spewer or the end of traditional
Farazouli, A., Cerratto-Pargman, T., Bolander-Laksov, K., & McGrath, C. (2023). Hello assessments in higher education? Journal of Applied Learning and Teaching, 6(1),
GPT! Goodbye home examination? An exploratory study of AI chatbots impact on 342–363. https://doi.org/10.37074/jalt.2023.6.1.9
university teachers’ assessment practices. Assessment & Evaluation in Higher Sandu, N., & Gide, E. (2019). Adoption of AI-chatbots to enhance student learning
Education, 13, 1–13. https://doi.org/10.1080/02602938.2023.2241676, 2023. experience in higher education in India. In 18th international conference on
García-Peñalvo, F. J. (2023). The perception of artificial intelligence in educational information technology based higher education and training (ITHET) (pp. 1–5). https://
contexts after the launch of ChatGPT: Disruption or panic? Education in the Knowledge doi.org/10.1109/ITHET46829.2019.8937382
Society (EKS), 24, Article e31279. https://doi.org/10.14201/eks.31279 Santomartino, S. M., & Yi, P. H. (2022). Systematic review of radiologist and medical
Hajkowicz, S., Sanderson, C., Karimi, S., Bratanova, A., & Naughtin, C. (2023). Artificial student attitudes on the role and impact of AI in radiology. Academic Radiology, 29
intelligence adoption in the physical sciences, natural sciences, life sciences, social (11), 1748–1756. https://doi.org/10.1016/j.acra.2021.12.032
sciences and the arts and humanities: A bibliometric analysis of research Santos, R. P. dos (2023). Enhancing chemistry learning with ChatGPT and bing chat as
publications from 1960-2021. Technology in Society, 74, Article 102260. https://doi. agents to think with: A comparative case study. arXiv:2305.11890. https://doi.org/
org/10.1016/j.techsoc.2023.102260 10.48550/arXiv.2305.11890
Holmes, W., & Tuomi, I. (2022). State of the art and practice in AI in education. European Selwyn, N. (2022). The future of AI and education: Some cautionary notes. European
Journal of Education, 57(4), 542–570. https://doi.org/10.1111/ejed.12533 Journal of Education, 57(4), 620–631. https://doi.org/10.1111/ejed.12532
Hwang, G.-J., & Chang, C.-Y. (2021). A review of opportunities and challenges of Tahiru, F. (2021). AI in education: A systematic literature review. Journal of Cases on
chatbots in education. Interactive Learning Environments, 1–14. https://doi.org/ Information Technology, 23(1), 1–20. https://doi.org/10.4018/JCIT.2021010101
10.1080/10494820.2021.1952615 Venkatesh, V., & Morris, M. G. (2000). Why don’t men ever stop to ask for directions?
Hwang, G.-J., Xie, H., Wah, B. W., & Gašević, D. (2020). Vision, challenges, roles and Gender, social influence, and their role in technology acceptance and usage
research issues of artificial intelligence in education. Computers & Education: Artificial behavior. MIS Quarterly, 24(1), 115–139. https://doi.org/10.2307/3250981
Intelligence, 1, Article 100001. https://doi.org/10.1016/j.caeai.2020.100001 Venkatesh, V., Morris, M. G., Davis, G. B., & Davis, F. D. (2003). User acceptance of
Kasneci, E., Sessler, K., Küchemann, S., Bannert, M., Dementieva, D., Fischer, F., information technology: Toward a unified view. MIS Quarterly, 27(3), 425–478.
Gasser, U., Groh, G., Günnemann, S., Hüllermeier, E., Krusche, S., Kutyniok, G., https://doi.org/10.2307/30036540
Michaeli, T., Nerdel, C., Pfeffer, J., Poquet, O., Sailer, M., Schmidt, A., Seidel, T., … Weise, K., & Metz, C. (2023). When A.I. Chatbots hallucinate. The New York Times https
Kasneci, G. (2023). ChatGPT for good? On opportunities and challenges of large ://www.nytimes.com/2023/05/01/business/ai-chatbots-hallucination.html.
language models for education. Learning and Individual Differences, 103, Article Welding, L. (2023). Half of college students say using AI on schoolwork is cheating or
102274. https://doi.org/10.1016/j.lindif.2023.102274 plagiarism. BestColleges. https://www.bestcolleges.com/research/college-students
Latham, A., & Goltz, S. (2019). A survey of the general public’s views on the ethics of -ai-tools-survey/.
using AI in education. In S. Isotani, E. Millán, A. Ogan, P. Hastings, B. McLaren, & Yeadon, W., & Halliday, D. P. (2023). Exploring durham university physics exams with
R. Luckin (Eds.), Lecture notes in computer science: 11625. Artificial intelligence in Large Language models. arXiv preprint arXiv:2306.15609 https://doi.org/10.4855
education. AIED 2019. Cham: Springer. https://doi.org/10.1007/978-3-030-23204- 0/arXiv.2306.15609.
7_17. Zawacki-Richter, O., Marín, V. I., Bond, M., & Gouverneur, F. (2019). Systematic review
Lee, H. (2023). The rise of ChatGPT: Exploring its potential in medical education. of research on artificial intelligence applications in higher education – where are the
Anatomical Sciences Education, 0(0), 1–6. https://doi.org/10.1002/ase.2270
Li, L., Ma, Z., Fan, L., Lee, S., Yu, H., & Hemphill, L. (2023). ChatGPT in education: A
discourse analysis of worries and concerns on social media. arXiv:2305.02201.
https://doi.org/10.48550/arXiv.2305.02201
Liu, G., & Ma, C. (2023). Measuring EFL learners’ use of ChatGPT in informal digital
learning of English based on the technology acceptance model. Innovation in

11
C. Stöhr et al. Computers and Education: Artificial Intelligence 7 (2024) 100259

educators? International Journal of Educational Technology in Higher Education, 16(1), Vogels, E.A. (2023). A majority of Americans have heard of ChatGPT, but few have tried
1–27. https://doi.org/10.1186/s41239-019-0171-0 it themselves. Pew Research Center. https://pewrsr.ch/3ICYoIx.
Zhang, W., Cai, M., Lee, H. J., Evans, R., Zhu, C., & Ming, C. (2023). AI in Medical
Education: Global situation, effects and challenges. Education and Information
Technologies. https://doi.org/10.1007/s10639-023-12009-8

12