Technovation 133 (2024) 103002

Contents lists available at ScienceDirect

Technovation
journal homepage: www.elsevier.com/locate/technovation

Future applications of generative large language models: A data-driven case

study on ChatGPT
Filippo Chiarello a, d, *, Vito Giordano a, d, Irene Spada b, d, Simone Barandoni c, d,
Gualtiero Fantoni b, d
a
Department of Energy, Systems, Territory and Construction Engineering, University of Pisa, Italy
b
Department of Civil and Industrial Engineering, University of Pisa, Italy
c
Department of Computer Science, University of Pisa, Italy
d
Business Engineering for Data Science (B4DS) Research Lab, University of Pisa, Italy

A R T I C L E I N F O A B S T R A C T

Handling Editor: Dr Stelvia Matos This study delves into the evolving role of generative Large Language Models (LLMs). We develop a data-driven
approach to collect and analyse tasks that users are asking to generative LLMs. Thanks to the focus on tasks this
Keywords: paper contributes to give a quantitative and granular understanding of the potential influence of LLMs in
Generative artificial intelligence different business areas. Utilizing a dataset comprising over 3.8 million tweets, we identify and cluster 31,747
Generative large language models
unique tasks, with a specific case study on ChatGPT. To reach this goal, the proposed method combines two
ChatGPT
Natural Language Processing (NLP) Techniques, Named Entity Recognition (NER) and BERTopic. The combi­
Social media analysis
Technology adoption nation makes it possible to collect granular tasks of LLMs (NER) and clusters them in business areas (BERTopic).
Emerging technologies Our findings reveal a wide spectrum of applications, from programming assistance to creative content genera­
tion, highlighting LLM’s versatility. The analysis highlighted six emerging areas of application for ChatGPT:
human resources, programming, social media, office automation, search engines, education. The study also
examines the implications of these findings for innovation management, proposing a research agenda to explore
the intersection of the identified areas, with four stages of the innovation process: idea generation, screening/
idea selection, development, and diffusion/sales/marketing.

1. Introduction engendering an unprecedented impact in terms of user numbers, esti­

mated to be over 100 million in January 2023 (Huang et al., 2023). The
In the rapidly evolving landscape of artificial intelligence (AI), rise of generative LLMs has been led by ChatGPT, the openAI conver­
generative large language models (LLMs) have emerged as a pivotal sational system built on the generative pre-trained transformer (GPT)
innovation. Language models are probabilistic models that enable the model launched in late 2022. Other relevant generative LLMs are
processing of natural language through algorithms, and they are the core Microsoft’s Bing, which focuses on interacting with the web to provide
of the natural language processing (NLP) techniques. The term ‘large’ answers based on present information, and Google’s PaLM, known for its
refers to the huge number of parameters needed to train these models, improved multilingual, reasoning, and coding capabilities. Meta’s
whereas the term ‘generative’ indicates a subset of LLMs specifically LLaMA is another contender in the space, with an approach to
designed to generate text. Generative LLMs are at the forefront of multi-modal learning that integrates both text and image data.
numerous applications, ranging from summarisation to translation, Despite already reaching millions of users, generative LLMs are still
typically delivered through dialogue-like interactions with the user in their infancy, aligning with what innovation literature defines as
(Zhao et al., 2023). emerging technologies. These are characterized by radical novelty, rapid
The dynamic of the dialogue eases the interaction of users on the growth, coherence, prominent impact, and a degree of uncertainty and
system (Dang et al., 2022). According to the Technology Acceptance ambiguity (Rotolo et al., 2015). Moreover, these new generative models
Model, perceived ease of use and usefulness are the factors that influence have found applications across a wide spectrum of fields, making them
user acceptance of technology (Davis, 1989). Indeed, these systems are general-purpose technologies (Bresnahan and Trajtenberg, 1995).

* Corresponding author. Department of Energy, Systems, Construction and Territory (DESTEC), Largo Lucio Lazzarino, 56122, Pisa, PI, Italy.
E-mail address: [email protected] (F. Chiarello).

https://doi.org/10.1016/j.technovation.2024.103002
Received 27 February 2023; Received in revised form 7 February 2024; Accepted 26 March 2024
Available online 29 March 2024
0166-4972/© 2024 The Author(s). Published by Elsevier Ltd. This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-
nc-nd/4.0/).
F. Chiarello et al. Technovation 133 (2024) 103002

Earlier NLP models were often highly task-specific, whereas newer NLP adopted in literature using various methods for analysing emerging
technologies exhibit remarkable versatility and can seamlessly handle a technologies. Some methods are based on bibliometric data, such as
variety of tasks (Bouschery et al., 2023). citation analysis (Small et al., 2014), whereas others are based on textual
The success of these models is not solely determined by the quality of data (Porter et al., 2019; Ranaei et al., 2020), such as patents, scientific
their output and their general-purpose nature (refer to Section 2 for paper, and social media. The analysis of textual data with NLP out­
more details), but primarily by the way users can interact with them. performs bibliometric analysis (Arts et al., 2021), and Twitter has been
Fig. 1 elucidates this interaction. A user starts by defining a task. The user confirmed to be a reliable source to explore trends in product develop­
then interacts with the model by describing a task (e.g. writing a research ment (Ozcan et al., 2021). The choice of Twitter was also informed by
paper, generating novel ideas for a market advertising, or debugging a our aim to concentrate on the tasks users assign to ChatGPT. Many re­
Python code) via a prompt (i.e., a request made to the generative sys­ searchers have utilized Twitter to capture this user feedback, even to
tem), which is then sent to the model. The model analyses the prompt analyse user opinions on emerging technologies (Kohl et al., 2018; Li
and responds with a textual output. Since the user expects a certain result, et al., 2019, 2022). Finally, we selected ChatGPT considering its wide­
the result is compared with the actual output. The user is then able to spread adoption.
iterate a new prompt to refine the output, thus engaging in a form of We collected 3,821,843 tweets talking about ChatGPT published
conversation with the system until the output does not match the ex­ between November 2022 and May 2023. Using named entity recogni­
pected result. tion (NER) techniques, a branch of NLP, we filtered the tweets
At present, generative LLMs have characteristics that limit the pos­ describing tasks assigned to ChatGPT and isolated the textual descrip­
sibility to study their behaviour and how users are interacting with tion of these tasks. Next, we clustered the tasks using a topic modelling
them. First, the systems are black boxes in the sense that their huge size technique based on bidirectional encoder representations from trans­
(i.e., 175 billion parameters for ChatGPT) limits their explainability, a formers (BERT), namely, BERTopic (Grootendorst, 2022), to identify
problem largely studied in the field of explainable artificial intelligence groups of similar tasks. We thus obtained a map of the uses of ChatGPT
(XAI) (Barredo Arrieta et al., 2020). Second, the prompts and responses and discussed the emerging areas of its application. With this map, we
of the models are proprietary to the owner companies. Some websites identified the business areas where ChatGPT is having an impact in
that are collecting this information are emerging, but the information terms of users’ requests, as well as the opportunities for research and
stored is still limited and unable to give a full understanding of LLM innovation in the realm of generative LLMs.
behaviour. Our findings show that NLP can help having a clear view of the dy­
LLMs are poised to change people’s relationship with technology and namics that stands at the intersection of an emerging technology and
with AI, a phenomenon that is challenging to study. Considering the users. NLP has proven to be a proper method to identify relevant part of
growing importance of the topic of AI for the innovation management texts from large social media databases (thanks to NER) and to synthe­
community (Mariani et al., 2023), innovation researchers are expected sise this information to give a map of LLMs tasks (thanks to topic
to better elucidate the evolution of generative LLMs. Given the chal­ modelling). Empirically, the proposed map will help orient towards
lenging characteristics of the technology, we identified a research gap LLMs applications in business. Our findings uncover a broad spectrum of
that lies in the point where the user interacts with the systems: the tasks ChatGPT applications, ranging from programming assistance to creative
(i.e. what generative LLMs are asked to do). This problem has been content generation, underscoring the versatility and potential of LLMs in
identified for AI in general, but no clear research path has been identi­ revolutionizing business processes and services. We identified six
fied to solve it (Cho et al., 2023). emerging areas of significant application: human resources, program­
We aimed to address this gap by analysing descriptions of task that ming, social media, office automation, search engines, and education.
users are asking generative LLMs. Specifically, we sought to answer the The integration of the results with the phases of innovation, will help
following research question: Which are the tasks that users are asking researchers focus on promising areas that are having an impact on users
generative LLMs? Answering such a question has implications both for practice with LLMs. In fact, we synthesized the research opportunities in
theory and practice. A clear map of LLM tasks, can enable researchers to a preliminary research agenda and highlighted future research chal­
study how users are interacting with LLMs while this technology is lenges that innovation management scholars will need to face to answer
emerging. A quantitative view in this direction is essential to comple­ the proposed questions.
ment the scientific discourse on the topic, that given the impossibility to Finally, our study also contributes to open research on the topic, to
have access to first hand data, risks to focus only on opinion papers and incentivise quantitative research, by opening the collected dataset and
qualitative research. Also, companies can explore the map to understand analysis (https://github.com/NutrientAxis/tasks_asked_to_ChatGPT).
how LLMs can be used in their context, and to spot priorities of users that
are early adopters of the technology. Given the paste at which LLMs has 2. Technological advancement of large language models
pervaded the market, companies are missing maps to explore this un­
known territory. From the theoretical side, developing the correct LLMs represent a class of NLP technologies that leverage extensive
metrics on the collection of tasks can enable to very if and how much textual data for both understanding and generating natural language. To
emerging and general-purpose technologies as LLMs fit to consolidated elucidate how these models operate and their key features, we con­
innovation theories. ducted a review of the major technological advancements in NLP from
We conducted a case study based on textual analysis of Twitter data its inception.
(renamed X from July 2023 but called Twitter in the rest of the paper for One of the first and simplest algorithms applied to extract informa­
the sake of simplicity) on ChatGPT. Textual analysis of Twitter has been tion from texts is the bag-of-words (BoW). It operates by representing a

Fig. 1. A model for the interaction between a user and a Generative LLM. Verbs on the arrows.

2
F. Chiarello et al. Technovation 133 (2024) 103002

document as a collection (‘bag’) of its constituent words, with no even faster. Finally, the versatility and broad applicability of LLMs have
consideration for the order or structure of the words in the document powered their rapid adoption.
(Hofmann et al., 2019). To address this limitation, researchers have used The last aspect is also linked to another characteristic of emerging
two-dimensionality reduction techniques, namely, latent semantic technology: prominent impact. In fact, these new generative models
analysis (LSA) and singular value decomposition (SVD). Another have found applications across a wide spectrum of fields, such as sci­
commonly used dimensionality reduction technique is latent Dirichlet entific writing (Else, 2023; Stokel-Walker, 2023), chemistry (Hocky and
allocation (LDA). LDA is employed to extract topics from a collection of White, 2022), journalism (Pavlik, 2023), psychology (Lee et al., 2022),
documents and then classify them based on the distribution of these human resources management (Budhwar et al., 2023), healthcare (Sal­
topics (Hannigan et al., 2022). Recently, NLP research has shifted its lam, 2023), and education (Graham, 2022; Stokel-Walker, 2022). Other
focus towards word-embedding algorithms, which are trained on large reports have used these models in experiments, such as programming
amounts of textual data to capture the semantic meaning of words. code, writing songs, and writing cover letters (Chatterjee and Dethlefs,
These algorithms can be categorised as LLMs, representing a significant 2023).
leap in NLP technology (Ameye et al., 2023; Just, 2024; Song et al., While its potential is vast, there are also significant challenges and
2023). Word-embedding techniques leverage artificial neural networks uncertainty regarding its future development, governance, and broader
to represent words as dense semantic vectors of real numbers (i.e. vec­ implications. Issues related to bias, fairness, accountability, and the
tors that embed the meaning of words) (Mikolov et al., 2013). Tradi­ potential for misuse (Abadie et al., 2024; Ali et al., 2024) necessitate
tional word embeddings provide a single vector for a given word, careful consideration and ongoing research. This uncertainty nature is
regardless of context and possible polysemy. In the past few years, a typical of emerging technologies.
novel class of LLM algorithms has been created, transformers (or When considering the four distinct phases of the innovation process
contextual word embeddings), of which BERT (Devlin et al., 2018) is one identified by Salerno et al. (2015), namely, idea generation, idea se­
of the first and more popular architectures. Transformers address pre­ lection, development, and diffusion, generative LLMs have been
vious system limitations by considering the textual context of words or employed for diverse purposes. In the idea generation phase, Piller et al.
phrases. (2023) employed generative LLMs to enhance the creative process.
One of the first generative LLMs was GPT. Introduced by OpenAI in Moving to the idea selection phase, Hong et al. (2022) introduced sys­
2018, GPT employs the power of transformer architecture, such as tems for screening ideas to facilitate the early stages of technological
BERT, to generate natural language. Over the years, the LLM field has development, leveraging patent analysis and Word2Vec. In the context
witnessed significant advancements, leading to the emergence of a new of the development phase, Deloose et al. (2023) created systems that
category of GPT technologies, notably OpenAI’s GPT-3 (Ouyang et al., classify failures within industrial contexts. As for the diffusion phase,
2022). GPT-3 stands out as a large, task-agnostic model capable of Arts et al. (2021) discovered that NLP technologies outperform other
performing a wide range of NLP tasks based on user prompts presented approaches in assessing the impact of new technologies on the market.
in natural language (see Section 1 and Fig. 1 for more details). Models In recent years, an increasing number of literature reviews have
like GPT-3 are trained on extensive document corpora, enabling them to explored the adoption of NLP and LLMs in innovation. Antons et al.
learn the distribution of words in text and generate new text sequences (2020) conducted a comprehensive review of the state-of-the-art NLP
(Reynolds and McDonell, 2021). In November 2022, OpenAI launched a systems employed in innovation research. Siddharth et al. (2022) delved
conversational AI, called ChatGPT (Van Dis et al., 2023), built on GPT-3 into the literature to examine NLP applications in design, shedding light
technology. Following the success of GPT-3 and ChatGPT, several other on their role in enhancing the creative and design processes. Just (2024)
companies have ventured into the development of generative LLMs. reported deeper insights into how NLP approaches enrich intermedia­
Notably, Google introduced PaLM2, and Meta unveiled LLaMA. In tion in the early stages of innovation search. These reviews collectively
March 2023, OpenAI released GPT-4, a powerful language model underscore the growing significance of NLP and LLMs in innovation
trained on a substantial number of documents (Sanderson, 2023). research and their capacity to revolutionise each of the innovation
OpenAI is actively working on launching another model, GPT-5, which phases.
is expected to be trained on an even larger and more extensive dataset.
Generative LLMs are in their infancy, aligning perfectly with the 3. Methodology
definition of emerging technologies in innovation literature. These are
characterized by radical novelty, relatively rapid growth, coherence, Fig. 2 presents the methodological steps for analysing the tasks
prominent impact, and a degree of uncertainty and ambiguity (Rotolo performed by ChatGPT. First, we collected tweets related to ChatGPT;
et al., 2015). The radical novelty of LLMs is evident in the model’s ability second, we used NER techniques to identify the tasks that users assigned
to understand and generate human-like text, pushing the boundaries of to ChatGPT; third, we normalised the identified tasks; and finally, we
natural language processing and artificial intelligence. This is not a mere clustered the tasks using BERTopic.
incremental improvement over existing technologies but a significant
leap forward, introducing capabilities previously thought to be exclusive 3.1. Twitter data collection
to human intelligence (Makridakis et al., 2023).
For what concern rapid growth, Generative LLMs reached millions of We collected 3,821,843 tweets using the Twitter API. Twitter queries
users in just a few months (Burtsev et al., 2023), a pace of adoption that used for data extraction included the following keywords: ‘chat-gpt’,
is both remarkable and indicative of their transformative potential. The ‘chat gpt’, ‘chatgpt’, ‘#chatgpt’, and ‘#gpt3’. Other generative AI sys­
rapid growth of Generative LLMs can be attributed to several factors. tems, such as AgentGPT and Google Bard, exist, and new ones may
Firstly, the technology itself is highly scalable. Once developed, emerge in the future. Nevertheless, given that OpenAI, with its product
deploying these models to serve additional users often requires rela­ ChatGPT, is the first mover in this field, we hypothesised that replicating
tively marginal increases in resources compared to the initial investment the success and popularity of ChatGPT could be challenging. For
in training and infrastructure. This scalability allows LLMs to be made instance, in May 2023, the tweets about AgentGPT (searched as
accessible to a broad audience swiftly. Moreover, the growth of LLMs is #agentgpt) and Google Bard (#googlebard) were only 2399 and 11,688,
pushed by the network effects in the technology ecosystem. As more respectively. For this reason, we assumed that our search strategy would
users engage with LLMs, more data becomes available, which, in turn, be robust. The data collection addressed tweets published from
can be used to train and refine the models, making them more accurate November 3, 2022, when OpenAI’s ChatGPT was launched, to May 16,
and effective. This self-reinforcing cycle contributes to the rapid 2023. The dataset we collected from Twitter spanned a period of six
improvement and adoption of LLMs, making their growth trajectory months. The dataset included only English tweets.

3
F. Chiarello et al. Technovation 133 (2024) 103002

Fig. 2. Flow diagram of the adopted methodology.

Twitter is recognised as a reliable tool for studying emerging tech­

Table 1
nologies (Kohl et al., 2018; Li et al., 2019, 2022). The temporal
Rules used for building the NER systems able to map the tasks.
perspective can vary widely, ranging from recent monthly analyses to
decades-long timeframes (Lacasa et al., 2003). Compared with these Technology Connection Action Verb Object
Element
studies, our research analysed 3,821,843 tweets, much larger than that
in other studies that adopted similar approaches: 3646 tweets in Li et al. gpt, (gpt), #gpt, to, explains, can, Optional Direct Object
(#gpt), gpt3, could, about, for, Adverb + Verb (excluding: me,
(2019), 29,272 tweets in Li et al. (2022), and 1,963,905 tweets in Kohl
(gpt3), #gpt3, may, might, + Optional you, it, him, her,
et al. (2018). This comparison illustrates that despite the relatively short (#gpt3),gpt-3, knows how, Word (to, us, them, this,
temporal window (about six months) of our study, our research aligns (gpt-3), gpt 3, (gpt knows how to, explains, can, that, those,
with other works that utilize Twitter to analyse emerging technologies. 3), #gpt-3, (#gpt- how to, can be could, about, these)
3), chatgpt, used to, can be for, may, might)
(chatgpt), used for, can be + Optional
3.2. Task identification with named entity recognition #chatgpt, used at, can be Article +
(#chatgpt), used in, can be Optional
chatgpt3, able to, can be Possessive
We analysed the Twitter data collected in Section 3.1 using NER (chatgpt3), chat able for, can be Adjective +
techniques to identify the tasks assigned to ChatGPT. We used NER, an gpt, (chat gpt), able at, can be Optional
NLP technique to sort specific entities in the text into specific categories. chat-gpt, (chat- able in, can be Adjective +
gpt), #chat-gpt, capable in, can be Optional
For a comprehensive understanding of the different NER techniques,
(#chat-gpt), chat- capable for, can Compound
refer to the work of Puccetti et al. (2023), section 3.2. gpt3, (chat-gpt3), be capable at, can
To the best of our knowledge, the literature on NER systems for #chat-gpt3, be capable of, can
extracting tasks performed by technology is limited. The NER methods (#chat-gpt3), chat help in, can help
aiming at categorising technical entities focus on extracting technolog­ gpt3, (chat gpt3), for, can help at,
#chatgpt3, can support in,
ical concepts, such as technologies (Giordano et al., 2021), functions (#chatgpt3), can support for,
(Yoon et al., 2013), chemical compounds (Ohms, 2021), and biomedical chatgpt-3, can support at,
entities (Huang et al., 2020). None of these works attempts to recognise (chatgpt-3), ai, how to help in,
the entity ‘task’ in the textual data. Only the work of Choi et al. (2021) artificial how to help for,
intelligence how to help at,
used rule-based NER to identify actions performed by humans in pro­
how to support in,
cedures. Nevertheless, they tailored their system to recognise tasks how to support
exclusively within a specific textual domain (procedures), within a for, how to
particular context (the nuclear field), and for a distinct case study. This support at, is
specific design limits the applicability of Choi et al.’s (2021) system to capable of, is
capable in, is
our Twitter dataset. Indeed, tweets have a conversational tone different capable to, is used
from the formal text structure of procedures. For these reasons, we to, is used for, is
developed a new rule-based NER instead of using state-of-the-art sys­ used in
tems, given that existing systems may not have been specifically trained
to identify the specific entities (‘task’) that are the focus of our research.
expressions, which can lead to incorrect task identification. Moreover,
We manually analysed 1000 tweets and then identified a recurring
for the clustering of similar tasks in the next step, we filtered the noisy
pattern for expressing a task in natural language:
ones with a manual revision of the clusters, reaching a theoretical pre­
[ChatGPT TECHNOLOGY] + [CONNECTION ELEMENT] + [ACTION cision of 100% of the overall process (i.e. only actual tasks performed by
VERB] + [OBJECT] ChatGPT were manually included in the analysis). Finally, we applied
the rule-based NER to the tweets.1 We extracted 526,861 tasks from 496,
This pattern indicates that users tend to express tasks for ChatGPT by 099 tweets, and we collected 232,449 unique tasks.
writing the name of the technology, followed by a variable sequence of The NER method has proven to be effective in achieving the desired
words (the ‘connection element’), an action verb (expressed through a outcomes. However, some limitations should be highlighted. Firstly, if
verb and other optional words, e.g. adverb, articles, and adjectives), and there is a shift in the type of entity being extracted (for instance, from
a direct object. Table 1 lists all the identified terms and sequences. We tasks to technology), it will be necessary to modify the extraction rules
created the rule-based approach by creating all possible combinations of accordingly. For guidance on extracting technology entities, one can
these elements. refer to Puccetti et al. (2023); for user entities, to Bonaccorsi et al.
We measured the precision and recall performance of the rule-based (2021); and for user needs, to Spada et al. (2023). Secondly, NER sys­
NER system (Tsai et al., 2006) using a sample of 1000 tweets that was tems that utilize machine learning and LLMs for text representation have
manually checked as a golden set. We specified whether a tweet dis­ been shown to outperform rule-based extraction methods across various
cussed a task performed by ChatGPT. The system had a precision of 0.59 NER tasks, as highlighted by Puccetti et al. (2023). However, adopting
and recall of 0.44. The lower recall performance did not negatively this approach requires the creation of a gold standard dataset, and the
affect our research, which aimed to elucidate the utilisation of LMMs, costs associated with this process are deemed to exceed the potential
not to create the most effective NER system for task identification. In benefits in terms of increased precision and recall for the system needed
other words, the low recall performance is not a problem given that the
macroscopic phenomena can still be observed. Regarding precision, the
pre-processing phase, described in Section 3.3, was designed to address
the challenges posed by informal language, misspellings, and colloquial 1
https://spacy.io, accessed 2 June 2023 6 February 2023.

4
F. Chiarello et al. Technovation 133 (2024) 103002

in the present work, as previously discussed. require specifying the classes for grouping tasks a priori, information
that was not available before our study. Lastly, the semantic map
3.3. Data pre-processing pipeline employed to present the results relies on dimensionality reduction via
the Uniform Manifold Approximation and Projection (UMAP) tech­
We then processed the list of 232,449 tasks with a data pre- nique, as described by McInnes et al. (2018). This choice, too, is based on
processing pipeline for removing noise and normalising its text. The empirical judgment. It’s worth noting that other visualization tech­
process consisted of the following steps: niques might reveal additional nuances of the data, offering further in­
sights into our findings.
1. Lemmatisation;
2. Removal of adverbs as terms that do not provide any meaningful 4. Results and discussion
information about the task;
3. Identification of the groups of tasks in our set that were synonyms By applying the methodology outlined in Section 3, we identified
(terms written with a different spelling but having the same mean­ 262,192 tweets that described a task (6.86% of the starting database of
ing) using semantic similarity methods. These algorithms mapped tweets) and a total of 31,747 unique tasks. These tasks were then clus­
each task to a vector in a 1024-dimensional semantic space using the tered using the BERTopic algorithm into 389 topics; a topic consisted of
BERT model stsb-distilbert-base2 (Devlin et al., 2018) and calculated a group of semantically similar tasks (e.g. Writing a poem, Writing a haiku,
cosine similarity between each task pair. Tasks were grouped Generating a poem, Writing a sonnet, and Creating a poem). The label of the
together if their cosine similarity was greater than 0.92; topic was assigned based on the task with the greatest number of tweets.
4. Automatic removal of tasks occurring less than six times; Table 2 shows the top 10 topics, in terms of the number of tweets citing a
5. Manual removal of entities that could not accurately be classified as task in the topic, along with the total number of tweets, number of
‘tasks performed by ChatGPT’ but were nonetheless extracted by our different tasks, and top five tasks in terms of the number of tweets. In
algorithm. The entities list was manually screened by three of the this section, we explore each line of the table, establishing connections
authors. with theories of innovation. To ensure brevity, we discuss some lines
collectively. The methodology for grouping tasks for discussion was
Subsequently, the list of tasks consisted of 31,747 unique tasks determined by the authors, who evaluated the task descriptions
identified in 262,192 tweets, representing 6.86% of the starting data­
base of tweets. The pre-processing steps used in the proposed method­
ology are largely used in text-mining techniques and have been proven Table 2
Top 10 topics of tasks mentioned in tweets.
effective in the context of technology and innovation (Antons et al.,
2020). # Topic Number of Number of Task (number of tweets citing
tweets tasks the task)

3.4. BERTopic 1 Write code 7549 578 write code (2,823), generate
code (889), explain code (118),
produce code (114), write good
We clustered the 31,747 tasks using the BERTopic algorithm code (94)
(Grootendorst, 2022). BERTopic is a clustering technique leveraging 2 Write 5452 258 write article (1,413), write tool
word embeddings and has excellent effectiveness for the topic modelling article (738), write something (692),
of tweets (Egger and Yu, 2022). The method has also been used in the write a text (408), write part
(159)
context of technology mapping and exploration (Jeon et al., 2023).
3 Write a 4569 149 write a poem (3,356), create
Details on the mathematical model and usages of the approach have poem poem (140), generate poem
been reported by Grootendorst (2022). (135), make poem (134), write a
In our study, the algorithm first generated document embeddings positive poem (102)
using the transformer all-MiniLM-L6-v2, which could map each task into 4 Answer 4244 260 answer questions (1,979), raise
questions question (157), responded to
a 384-dimensional dense vector space. Second, it reduced their dimen­ questions (139), answer some
sionality and clustered them based on semantic similarity (calculated search queries (130), answer
through the cosine similarity between vectors). Finally, it provided a follow-up question (119)
topic representation. In this way, we obtained a set of topics, with each 5 Write story 3647 220 write story (1,528), write a short
story (357), tell a story (155),
topic referring to a group of semantically similar tasks. We then analysed
generate story (133), write a
the distribution of these topics on a semantic map and the number of novel (127)
tweets associated with each topic for measuring popularity and 6 Give 3537 225 gives answers (1,272), give great
relevance. answers answers (594), generate answer
BerTopic proven to give the desired output. Anyway, the decision to (314), know answer (123), gives
better answers (121)
utilize the transformer model, all-MiniLM-L6-v2, and its corresponding
7 Write a 3510 174 write a tweet (1,638), create a
384-dimensional dense vector space is based on empirical evidence. As tweet tweet (579), write a Twitter
of now, there are no studies that definitively prove this approach to be thread (148), make a tweet (92),
the optimal solution for the task at hand. Furthermore, like the con­ reply to tweets (90)
8 Write an 3100 101 write an essay (2,270), wrote
straints encountered with rule-based NER, the performance of the
essay essays (114), writes smart essays
grouping task could be enhanced by having a golden set of similar tasks (112), generate essay (81),
for reference and switching from clustering to classification. However, generate essays (59)
the creation of such a labelled dataset falls outside the scope of this 9 Replace 2820 325 replace developer (177), replace
paper. We also believe that the potential improvement in performance developer human jobs (172), replace
programmers (168), replace
would not justify the costs associated with manually annotating the
writer (143), replace lawyer
data. Furthermore, adopting classification instead of clustering would (109)
10 Write an 2374 193 write an email (1,001), generate
email emails (232), draft an email (96),
2 write work emails (90), write
https://huggingface.co/sentence-transformers/stsb-distilbert-base,
blogs newsletter (57)
accessed 2 June 2023 6 February 2023.

5
F. Chiarello et al. Technovation 133 (2024) 103002

reviewing the whole set of tasks available in the dataset and considered critical role in the development and application of new technologies
the characteristics of the document cited in the task, such as its length, (Von Hippel, 1986). As more users engage with ChatGPT for a variety of
the technical nature of its content, and its specific field. tasks, the better would network effects come into play. The value of the
Lines 1 and 3 contain the topics Write code and Write a poem, AI system increases with the number of users, as it learns from diverse
respectively, that properly represent the versatility of ChatGPT and its input and improves its performance across tasks (Abrahamson and
span of uses. This variety of tasks demonstrated its widespread accep­ Rosenkopf, 1997).
tance across different domains. This aligns with Diffusion of Innovations Finally, lines 9 and 10 give topics in the realm of office automation.
(Rogers, 2004), indicating that ChatGPT is traversing the first stages of The topic Replace developer highlights a common perception among
the innovation adoption lifecycle. Write code shows that in the Infor­ users that AI could potentially substitute humans in performing tasks in
mation Technology and Software Development sectors, AI-driven code various fields (see topic 1). Indeed, AI can automate certain tasks.
generation and optimisation is significantly streamlining development However, AI is not intended to replace humans but rather augment their
processes, reducing the time and resources required for debugging and capabilities. For example, the ability to generate, draft, and even
optimising code. In parallel, the impact it will have on personalise emails can greatly enhance office productivity, allowing
creativity-intensive document production, such as poems, poses big workers to dedicate more time to tasks that require human creativity
challenges for creative tasks. and strategic thinking. The idea that AI can replace certain tasks but not
Line 2 shows the topic Write article, a generic one that contains ab­ entire jobs is reminiscent of the job-task approach (Autor, 2013). This
stract tasks (e.g. Write something). Thus, many users are describing tasks theory suggests that automation is more likely to replace specific tasks
with a high level of abstraction and are playing with ChatGPT to explore within a job rather than the entire job itself, leading to a change in jobs
its performance. They are having their first interaction with generative and the skills required.
LLMs through ChatGPT. ChatGPT’s purpose is to open up the power of Fig. 3 shows the results of the topic modelling represented as a
generative AI—already well-known by researchers (Radford et al., scatter plot, offering a bird’s-eye view of the collected topics of tasks and
2018)—to a wider public. The high level of abstraction points to the their similarities. In the figure, all the 31,747 extracted tasks are located
initial exploratory stage of technology adoption, reflecting the perceived in a bi-dimensional space and grouped in topics represented by colours.
ease of use and curiosity-driven engagement, which are core aspects of We performed dimensionality reduction through the Uniform Manifold
the Technology Acceptance Model (Davis, 1989). Approximation and Projection technique (McInnes et al., 2018). In
The next topics on lines 4 and 6 are focused on one of the tasks that Fig. 3, each task is represented by a dot, and the topics of tasks also
ChatGPT has proven to be good at (i.e. Answer questions and Give An­ present a label, indicating the main task that represents the group. The
swers), so much so that it poses a threat to search engines, such as distances between the tasks (and between the topics, which are clusters
Google. These tasks are so important that ChatGPT is perceived by users of tasks) in the scatter plot are calculated through their semantic simi­
as a potential game changer in the search engine market, which has been larity. If two topics are semantically similar, they are represented close
a monopoly. Owing to a large investment in OpenAI, Microsoft is to each other in the graph. The grey dots represent the outliers topic,
seeking to gain market share in this context, threatening the leading which are 8156 tasks for which BERTopic could not find a suitable
position of Google. Although ChatGPT is not a search engine, the cluster.
conversational interaction (i.e. without the need to write a query but The map in Fig. 3 offers data-driven views of the different business
asking the engine questions) it uses shows a potentially new class of areas of development of tools such as ChatGPT, highlighting six
products, indicating potential for disruptive innovation. By providing emerging areas of application: related to tasks able to improve pro­
conversational interaction, ChatGPT may redefine user expectations and ductivity (automation for repetitive office tasks), tasks belonging to
behaviours in information retrieval, challenging established players in human resources (write a resume), programming (writing and explain­
the market (Bower and Christensen, 1995). ing code), social media (from tweets to blogs to jokes), education (pass
Lines 5 and 8 give the results for the tasks Write story and Write essay, exams, write essays, generate academic paper), and search engines
respectively. These tasks pertain to two different types of content (the (answering questions, replacing Google search).
first related to narrative writing, the second to technical writing) that
are longer than the ones described before. This shows the ability of 5. Research agenda
ChatGPT to cope with longer texts. ChatGPT (and all LLMs in general)
has some limitations in terms of the length of the output texts (Zhao Building on the empirical evidence of Section 4, we formulated a
et al., 2021), which may be surmounted in the future. Notably, the po­ series of research questions to guide further studies on the potential
tential improvements in handling longer text outputs suggest incre­ impacts of generative LLMs on innovation management. To transition
mental innovations in the capabilities of transformer-based systems like from our empirical findings to a structured research agenda, we per­
ChatGPT. Also, its ability to impact diverse fields, such as software formed an in-depth analysis of each topic (the top 10 shown in Table 2,
development, creative writing, and search engines, hints at radical in­ and the 379 in others). We adopted a dual-dimensional framework,
novations that could reshape entire industries (Dewar and Dutton, intertwining the six emerging areas of application shown in section 4
1986). For example, in the Publishing and Education sectors, AI’s ability with the four main phases of the innovation process as defined by
to generate longer, complex texts can transform content creation. For Salerno et al. (2015). This approach allowed us to map a matrix that
publishers, the technology introduces questions of authenticity and captures the intersection of generative LLMs’ applications with the
copyright in AI-authored works. In education, it presents opportunities stages of innovation, from ideation to market penetration. Table 3
for personalised learning materials but also challenges in assessing synthesises the proposed research questions. Each emerging area is
students’ original work amidst AI-generated content. linked to the empirical evidence by showing the two most related topics
Line 7 gives the topic Write a tweet, which reflects the impact that the (Table 2).
tool could have on social media. The fact that ChatGPT can generate Scholars and practitioners have already engaged in discussions sur­
social media posts can destroy the content industry, by making content rounding some of the points in the identified agenda, as shown by
creators into prompt creators. Impacts, which also emerge from other Mariani et al. (2023). Nonetheless, we expected the proposed research
tasks we analysed, can be negative in the production of disinformation agenda to contribute to the literature. First, previously identified chal­
pieces or generally low-quality content. This diversity of tasks—from lenges were unrelated to generative LLMs but were discussed for AI in
creating social media content to writing stories and essays—shows how general. This specific type of technology was our focus, given its impact
users are adapting ChatGPT for their unique needs and aligns with the in the technology development field and on society in general. Second,
concept of user-driven innovation, where the users themselves play a the challenges that we identified in relation to generative LLMs emerged

6
F. Chiarello et al. Technovation 133 (2024) 103002

Fig. 3. Clusters of ChatGPT tasks mentioned by users on Twitter.

from a quantitative study. This will give the opportunity to the research the time needed to bring new software products to market. In turn,
community to monitor these challenges and compare them with others barriers to entry for software development become lower.
derived from different research approaches (e.g. qualitative ones). Social Media: Researchers working on the use of generative systems
for innovating the social media content production process should bal­
ance the potential of automated content generation with ethical con­
5.1. Business-area challenges
cerns, such as the spread of disinformation, filter bubbles, and
manipulation of public opinion. When studying novel business models,
Human Resources: HR is a knowledge-intensive area. Thus, it will be
researchers should ensure that the new approaches respect user privacy,
challenging to guide and monitor the impact that generative LLMs will
adhere to data protection regulations, and avoid intrusive data
have on the production of HR-related documents (e.g. job postings, job
collection.
profile descriptions, and curricula). Given also that this kind of docu­
Office Automation: The ability of generative AI to support specific
ment is private, researchers must monitor and help companies in
office tasks, especially knowledge-intensive and domain-specific ones,
adapting to the ever-evolving HR practices and regulations of AI-
will depend on how much companies can train generative LLMs on
generated solutions in HR. The dynamic nature of HR practices, legal
proprietary data. The lack of domain-specific data can limit the accuracy
regulations, and ethical standards can be expected to increase in the next
and effectiveness of these technologies in performing certain tasks,
decade, requiring continuous updates and fine-tuning of both HR prac­
which can hinder its adoption and impact. Scholars will need to inves­
tices and AI models.
tigate approaches for acquiring and sharing domain-specific data,
Programming: Elucidating the impact of generative LLMs on software
including data-sharing agreements and incentives for companies to
innovation will required multidisciplinary research that integrates the
share their data.
domains of management, innovation, and computer science. The inte­
Search Engines: Generative AI is progressively integrated with search
gration of generative LLMs could impact the industry of integrated
engines, which will pose an important challenge for innovation re­
development environments and other software development tools that
searchers who are monitoring the rapid evolution of this market. In this
could revolutionise the way developers work. AI-generated code sug­
business area more than the others, researchers will need to use various
gestions, real-time debugging, and code optimisation could become
methods ranging from qualitative strategies (e.g. case studies and in-
standard features in these tools, improving productivity and reducing

7
F. Chiarello et al. Technovation 133 (2024) 103002

Table 3
Opportunities for future research on generative LLMs in Innovation.
Idea generation Screening, Idea selection Development Diffusion, Market, Sales

Human How can generative LLMs aid in How can generative LLMs streamline Will generative LLMs create new To what extent will the diffusion of
Resources generating new HR policies and new HR processes for efficiency? product opportunities in the context generative LLMs impact human
(Topics 9, 10) talent acquisition strategies? Will generative LLMs help the HR of HR (e.g. CV analysis, candidate skills (support vs. substitution)?
How much will Generative LLMs function in the screening and selection selection software)? How will upskilling and reskilling
impact the production of HR-related of candidates? How can generative LLMs help be managed?
documents (e.g. resumes, job create personalised employee
postings, job descriptions)? development plans?
Programming To what extent can generative LLMs How can generative LLMs aid in the What type of new programming How will the software development
(Topics 1, 9) be used for generating software selection of programming languages, languages (high-abstraction market respond to the automation
debugging ideas in brainstorming? development frameworks, and languages) will arise with the use of of many programming tasks in
Will generative LLMs be able to libraries? generative LLMs? terms of business modelling?
come out with novel software ideas? How can generative LLMs support the Will generative LLMs speed up the How can generative LLMs
evaluation of software development software prototyping and contribute to improving products as
projects? development phases? Integrated Development
Environments?
Social Media How much will generative LLMs How much will generative LLMs How will generative LLMs change How will generative LLMs impact
(Topics 5, 7) change the way in which social cooperate with people in the existing social media and enable the online advertising market?
media content is generated? assessment/selection of social media new social media to emerge? How can generative LLMs impact
How can generative LLMs content? How can generative LLMs support the role of content creators and
contribute to new strategies for How can generative LLMs aid in the the development of advanced influencers?
combating disinformation? selection of the most effective social content moderation algorithms?
media channels?
Office How much of the creative potential To what extent will the performance of Will the use of generative LLMs in How will generative LLMs impact
Automation of generative LLMs will be disclosed idea selection and assessment be everyday office automation tools the market of office automation?
(Topics 2, 10) at an office automation level? implemented in the office automation speed up product development? How much will this technology be
How can generative LLMs facilitate products based on generative LLMs? How can generative LLMs help integrated in existing office
the identification of new How can generative LLMs assist in improve communication and automation products?
opportunities for AI in office selecting the most suitable office collaboration in an automated
automation? automation tools? office environment?
Search Engines How will the combination of Will generative LLMs enhance Will AI-driven conversational Will the search engine market
(Topics 4, 6) generative LLMs and standard web- standard search systems, making it search interfaces substitute existing switch from a monopoly to an
search change the idea generation possible to move from information ones? oligopoly?
process? search to information assessment? Will generative LLMs make it How can generative LLMs influence
Will generative LLMs inspire the How much can generative LLMs assist possible to develop more the evolution of online advertising
creation of specialised search in prioritising the most relevant search personalised search engines? in the search engine market?
engines or AI assistants? results for users?
Education Will generative LLMs support the How will generative LLMs support Will generative LLMs develop Will generative LLMs open new
(Topics 8, 6) ideation of new educational educators in the selection of personalised educational content? markets in the education field?
programs? educational materials? How can generative LLMs aid in How can generative LLMs affect
Will generative LLMs radically How can generative LLMs help in redesigning assessments to counter access to education and the digital
change the way in which creativity evaluating the effectiveness of AI-driven cheating and content divide?
is taught? educational activities? generation?

depth interviews) to empirical experimental designs (e.g. surveys, text 5.2. Innovation-phase challenges
mining, and data analytics). Qualitative approaches could allow re­
searchers to explore the involvement of generative LLMs in the inno­ Idea generation: Innovation management researchers will need to
vation process, uncovering contextual factors and human—AI dynamics learn to integrate AI-generated ideas into the ideation process effec­
that may be difficult to capture through quantitative approaches. In tively. This includes tackling challenges around managing the volume of
contrast, quantitative studies could offer a broader perspective on the ideas, ensuring their relevance, and integrating them into existing
impact of generative AI, enabling generalisation across various contexts innovation processes (Verganti et al., 2020). To do so, they must address
and industries while overcoming biases from expert judgments often two main challenges. First, researchers must ensure that LLMs-generated
found in qualitative approaches. ideas are free from cultural, social, and political biases—this requires a
Education: Innovation management researchers will face the chal­ comprehensive analysis of training data and continuous model fine
lenge of helping the educational sector adapt to generative AI, to miti­ tuning. Second, to generate innovative ideas in various business areas,
gate the barriers to the adoption of these methods (Kaplan and Haenlein, researchers must train and adapt generative LLMs to comprehend
2016). Research should elucidate the potential for digital divide issues industry-specific language, concepts, and requirements, which may
and develop strategies to ensure that all learners can benefit from involve specialised training data and domain expertise. Increasing the
AI-driven educational innovations. The digital divide poses challenges ability in simulating the human creativity of machines can help re­
to adoption and use of generative LLMs, particularly in developing searchers understand creativity phenomena. LLMs must be made
countries with limited access to technology, infrastructure, and re­ explainable, and their decision, comprehensible (Guidotti et al., 2019).
sources. Insufficient internet infrastructure, varying digital literacy However, as stated in the Introduction, access to generative LLMs’ en­
levels, language barriers, and socioeconomic disparities are factors that gine is limited to their website and API, owing to their black-box nature,
can hinder widespread adoption. Investigating these challenges and which also hinders explainability. The billions of parameters of LLMs
factors could be valuable to bridge the digital divide and promote global require significant computational power, posing challenges to scalabil­
digital inclusion, thereby ensuring that the benefits of generative LLMs ity and efficiency. There are currently no reliable tools to audit deep
are accessible and equitable. learning systems.
Screening/Idea selection: Determining the quality and relevance of
LLMs-generated ideas in various business areas can be challenging.
Generative AI systems are more useful to exploring the solution space

8
F. Chiarello et al. Technovation 133 (2024) 103002

than to selecting the best ideas (Bouschery et al., 2023). To understand policymakers can benefit from our research agenda, which offer pre­
how generative AI can be used also in the screening/idea selection, re­ liminary insights into tasks users request from generative AI. Under­
searchers must develop methods to quantify and evaluate the quality of standing these tasks is vital for anticipating potential applications and
ideas, considering factors such as feasibility, impact, and novelty. In this shaping technology adoption. With this knowledge, policymakers can
context, decision-making frameworks may emerge, which can help make informed decisions and design regulations supporting responsible
practitioners incorporate insights from AI. These, in turn, might require AI development and deployment, thereby ensuring an alignment be­
new methods of idea valuation and risk assessment tailored to tween AI-driven innovation and societal goals. The research agenda can
AI-generated ideas. These frameworks will face the challenge of deter­ also bridge the gap between academia and practice, generating action­
mining to balance human judgment and AI in idea selection. This in­ able insights and evidence-based recommendations for real-world
volves understanding when to rely on AI and when human intervention application.
is necessary. Our results should be read considering the limitations of our
Development: Researchers need to ensure that LLMs-generated solu­ approach. Our method relied heavily on Twitter, which may have
tions can be seamlessly integrated in existing systems and workflows. To introduced potential biases and limited the representativeness of the
foster product development, companies will need to share good prac­ sample. Twitter users may not fully capture the diversity of opinions of
tices in collaboration with AI. Here, scholars and practitioners may meet the broader population. Furthermore, the prominence of certain topics
the challenge of working with limited open databases of prompts. For or trends on Twitter may not accurately reflect their importance in the
example, access to ChatGPT’s proprietary prompts and responses is wider context of the research question, since they may be influenced by
restricted by OpenAI’s terms of service. Although some websites are trends and hype. To address this limitation in future work, researchers
collecting this information, the data stored remain limited and unlikely could consider incorporating data from other sources (e.g. scientific
to be made open to researchers. This limits research on generative LLMs papers or alternative sources to Twitter, such as Reddit) to provide a
and impedes the convergence on best practices for product development more comprehensive view.
using generative AI. Lastly, as AI contributes more to product develop­ Nonetheless, our study did not aim to establish an empirical setting
ment, questions about IP rights emerge. Researchers will face the chal­ suitable for addressing the dynamic landscape of generative models.
lenge of navigating IP issues related to AI-generated designs or solutions, Rather, our goal was to present a snapshot of the current situation,
potentially necessitating new approaches to IP. thereby assisting researchers in conducting quantitative and rigorous
Diffusion/Market/Sales: Innovation management researchers will studies on an emerging technology that is currently discussed in quali­
need to develop new methods for evaluating the market potential of AI- tative terms. In particular, as newer models of generative LLMs emerge,
generated innovations, which may require new market analysis tech­ new tasks may also be introduced. Although we collected 31,747 tasks,
niques and the theorisation of new business models. Generative LLMs we do not assume that we have saturated the realm of possibilities for
can create platforms around which developers can create a shared value generative systems. Future studies may demonstrate the expanse of such
for customers (Trabucchi and Buganza, 2020). Researchers will also face a realm in the context of innovation management as well as quantify the
the challenge of managing the adoption and diffusion of AI innovations, delta of the tasks performed by generative LLMs.
including understanding and overcoming barriers to adoption, devel­
oping strategies for effective market diffusion, and managing bundles of Funding
different technologies to solve real-world industrial problems (Cho
et al., 2023). Finally, researchers interested in studying the diffusion of This work was partly funded by the ENCORE Project (ENriching
generative AI technologies may focus on different characteristics of Circular use of OeR for Education) - European Union Erasmus+ Part­
companies adopting these solutions, such as firm size. As reported by nerships for Innovation: Alliances (ERASMUS-EDU-2021-PI-ALL-INNO)
Mariani et al. (2023), the literature on innovation has highlighted the (Project Number: 101055893) and PNRR - M4C2 - Investimento 1.3,
importance of firm size in AI adoption, as it affects a company’s re­ Partenariato Esteso PE00000013 – “FAIR - Future Artificial Intelligence
sources and abilities to pursue innovative activities, as has been Research” - Spoke 1 “Human-centered AI,” funded by the European
emphasised by Chen and Hambrick (1995). Commission under the NextGeneration EU programme.

6. Conclusion and limitations CRediT authorship contribution statement

This study has explored the evolving landscape of generative Large Filippo Chiarello: Writing – review & editing, Writing – original
Language Models (LLMs), with a particular focus on ChatGPT, through draft, Visualization, Validation, Supervision, Software, Resources,
the analysis of over 3.8 million tweets. Our data-driven approach, Project administration, Methodology, Investigation, Funding acquisi­
leveraging Named Entity Recognition (NER) and BERTopic algorithms, tion, Formal analysis, Data curation, Conceptualization. Vito Giordano:
has provided a granular understanding of the tasks assigned to genera­ Writing – review & editing, Writing – original draft, Software, Meth­
tive LLMs, highlighting their versatility across various business areas. odology, Investigation, Formal analysis, Data curation, Conceptualiza­
Our findings indicate that ChatGPT, among other generative LLMs, is tion. Irene Spada: Writing – original draft, Methodology, Formal
being utilized in a wide spectrum of applications, ranging from pro­ analysis, Data curation, Conceptualization. Simone Barandoni: Writing
gramming assistance to creative content generation. The identified – original draft, Visualization, Software, Methodology. Gualtiero Fan­
emerging areas of application—human resources, programming, social toni: Writing – original draft, Validation, Supervision, Funding acqui­
media, office automation, search engines, and education—underscore sition, Conceptualization.
the potential of LLMs to revolutionise business processes and services.
Furthermore, this study contributes to the field of innovation manage­ Data availability
ment by proposing a research agenda focused on exploring the inter­
section of generative LLMs with the innovation process. Our proposed We have shared the link to data and method in the paper, on an
agenda outlines key research questions that span across the phases of anonymised Github page.
idea generation, screening and selection, development, and diffusion
and marketing.
References
Our results can assist managers in understanding how to implement
generative LMMs within their own innovation processes, taking into Abadie, A., Chowdhury, S., Mangla, S.K., 2024. A shared journey: experiential
account the unique characteristics of each business area. Moreover, perspective and empirical evidence of virtual social robot ChatGPT’s priori

9
F. Chiarello et al. Technovation 133 (2024) 103002

acceptance. Technol. Forecast. Soc. Change 201, 123202. https://doi.org/10.1016/j. Hannigan, T.R., Briggs, A.R., Valadao, R., Seidel, M.L., Jennings, P.D., 2022. A new tool
techfore.2023.123202. for policymakers: mapping cultural possibilities in an emerging AI entrepreneurial
Abrahamson, E., Rosenkopf, L., 1997. Social network effects on the extent of innovation ecosystem. Res. Pol. 51, 104315 https://doi.org/10.1016/j.respol.2021.104315.
diffusion: a computer simulation. Organ. Sci. 8, 289–309. Hocky, G.M., White, A.D., 2022. Natural language processing models that automate
Ali, O., Murray, P.A., Momin, M., Dwivedi, Y.K., Malik, T., 2024. The effects of artificial programming will transform chemistry research and teaching. Dig. Dis. 1, 79–83.
intelligence applications in educational settings: challenges and strategies. Technol. https://doi.org/10.1039/D1DD00009H.
Forecast. Soc. Change 199, 123076. https://doi.org/10.1016/j. Hofmann, P., Keller, R., Urbach, N., 2019. Inter-technology relationship networks:
techfore.2023.123076. arranging technologies through text mining. Technol. Forecast. Soc. Change 143,
Ameye, N., Bughin, J., van Zeebroeck, N., 2023. How uncertainty shapes herding in the 202–213. https://doi.org/10.1016/j.techfore.2019.02.009.
corporate use of artificial intelligence technology. Technovation 127, 102846. Hong, S., Kim, J., Woo, H.G., Kim, Y.C., Lee, C., 2022. Screening ideas in the early stages
https://doi.org/10.1016/j.technovation.2023.102846. of technology development: a word2vec and convolutional neural network
Antons, D., Grünwald, E., Cichy, P., Salge, T.O., 2020. The application of text mining approach. Technovation 112, 102407. https://doi.org/10.1016/j.
methods in innovation research: current state, evolution patterns, and development technovation.2021.102407.
priorities. R D Manag. 50, 329–351. https://doi.org/10.1111/radm.12408. Huang, F., Kwak, H., An, J., 2023. Is ChatGPT better than human annotators? Potential
Arts, S., Hou, J., Gomez, J.C., 2021. Natural language processing to identify the creation and limitations of ChatGPT in explaining implicit hate speech. arXiv Preprint ArXiv:
and impact of new technologies in patent text: code, data, and new measures. Res. 2302.07736. https://doi.org/10.48550/arXiv.2302.07736.
Pol. 50, 104144 https://doi.org/10.1016/j.respol.2020.104144. Huang, M.S., Lai, P.T., Lin, P.Y., You, Y.T., Tsai, R.T.H., Hsu, W.L., 2020. Biomedical
Autor, D.H., 2013. The ‘task approach’ to labor markets: an overview. J. Labour Mark. named entity recognition and linking datasets: survey and our recent development.
Res. 46, 185–199. Briefings Bioinf. 21, 2219–2238. https://doi.org/10.1093/bib/bbaa054.
Barredo Arrieta, A.B., Díaz-Rodríguez, N., Del Ser, J., Bennetot, A., Tabik, S., Jeon, E., Yoon, N., Sohn, S.Y., 2023. Exploring new digital therapeutics technologies for
Barbado, A., Garcia, S., Gil-Lopez, S., Molina, D., Benjamins, R., Chatila, R., psychiatric disorders using BERTopic and PatentSBERTa. Technol. Forecast. Soc.
Herrera, F., 2020. Explainable Artificial Intelligence (XAI): concepts, taxonomies, Change 186, 122130. https://doi.org/10.1016/j.techfore.2022.122130.
opportunities and challenges toward responsible AI. Inf. Fusion 58, 82–115. https:// Just, J., 2024. Natural language processing for innovation search – reviewing an
doi.org/10.1016/j.inffus.2019.12.012. emerging nonhuman innovation intermediary. Technovation 129, 102883. https://
Bonaccorsi, A., Chiarello, F., Fantoni, G., 2021. Impact for whom? Mapping the users of doi.org/10.1016/j.technovation.2023.102883.
public research with lexicon-based text mining. Scientometrics 126, 1745–1774. Kaplan, A.M., Haenlein, M., 2016. Higher education and the digital revolution: about
https://doi.org/10.1007/s11192-020-03803-z. MOOCs, SPOCs, social media, and the Cookie Monster. Bus. Horiz. 59, 441–450.
Bouschery, S.G., Blazevic, V., Piller, F.T., 2023. Augmenting human innovation teams https://doi.org/10.1016/j.bushor.2016.03.008.
with artificial intelligence: exploring transformer-based language models. J. Prod. Kohl, C., Knigge, M., Baader, G., Böhm, M., Krcmar, H., 2018. Anticipating acceptance of
Innovat. Manag. 40, 139–153. https://doi.org/10.1111/jpim.12656. emerging technologies using twitter: the case of self-driving cars. J. Bus. Econ. 88,
Bower, J.L., Christensen, C.M., 1995. Disruptive Technologies: Catching the Wave. 617–642. https://doi.org/10.1007/s11573-018-0897-5.
Bresnahan, T.F., Trajtenberg, M., 1995. General purpose technologies ‘Engines of Lacasa, I.D., Grupp, H., Schmoch, U., 2003. Tracing technological change over long
growth’. J. Econ. 65, 83–108. https://doi.org/10.1016/0304-4076(94)01598-T. periods in Germany in chemicals using patent statistics. Scientometrics 57, 175–195.
Budhwar, P., Chowdhury, S., Wood, G., Aguinis, H., Bamber, G.J., Beltran, J.R., https://doi.org/10.1023/A:1024133517484.
Boselie, P., Lee Cooke, F., Decker, S., DeNisi, A., Dey, P.K., Guest, D., Knoblich, A.J., Lee, P., Fyffe, S., Son, M., Jia, Z., Yao, Z., 2022. A paradigm shift from ‘human writing’ to
Malik, A., Paauwe, J., Papagiannidis, S., Patel, C., Pereira, V., Ren, S., Rogelberg, S., ‘machine generation’ in personality test development: an application of state-of-the-
Saunders, M.N.K., Tung, R.L., Varma, A., 2023. Human resource management in the art natural language processing. J. Bus. Psychol. 1–28 https://doi.org/10.1007/
age of generative artificial intelligence: perspectives and research directions on s10869-022-09864-6.
ChatGPT. Human Res. Mgmt Journal. 33, 606–659. https://doi.org/10.1111/1748- Li, X., Wen, Y., Jiang, J., Daim, T., Huang, L., 2022. Identifying potential breakthrough
8583.12524. research: a machine learning method using scientific papers and Twitter data.
Burtsev, M., Reeves, M., Job, A., 2023. The working limitations of Large Language Technol. Forecast. Soc. Change 184, 122042. https://doi.org/10.1016/j.
Models. MIT Sloan Manag. Rev. 65 (1), 1–5. techfore.2022.122042.
Chatterjee, J., Dethlefs, N., 2023. This new conversational AI model can be your friend, Li, X., Xie, Q., Jiang, J., Zhou, Y., Huang, L., 2019. Identifying and monitoring the
philosopher, and guide and even your worst enemy. Patterns (N Y) 4, 100676. development trends of emerging technologies using patent analysis and Twitter data
https://doi.org/10.1016/j.patter.2022.100676. mining: the case of perovskite solar cell technology. Technol. Forecast. Soc. Change
Chen, M.J., Hambrick, D.C., 1995. Speed, stealth, and selective attack: how small firms 146, 687–705. https://doi.org/10.1016/j.techfore.2018.06.004.
differ from large firms in competitive behavior. Acad. Manag. J. 38, 453–482. Mariani, M.M., Machado, I., Magrelli, V., Dwivedi, Y.K., 2023. Artificial intelligence in
https://doi.org/10.2307/256688. innovation research: a systematic review, conceptual framework, and future
Cho, J., Destefano, T., Kim, H., Kim, I., Paik, J.H., 2023. What’s driving the diffusion of research directions. Technovation 122, 102623. https://doi.org/10.1016/j.
next-generation digital technologies? Technovation 119, 102477. https://doi.org/ technovation.2022.102623.
10.1016/j.technovation.2022.102477. Makridakis, S., Petropoulos, F., Kang, Y., 2023. Large language models: their success and
Choi, Y., Nguyen, M.D., Kerr Jr, T.N., 2021. Syntactic and semantic information impact. Forecasting 5 (3), 536–549. https://doi.org/10.3390/forecast5030030.
extraction from NPP procedures utilizing natural language processing integrated McInnes, L., Healy, J., Melville, J., 2018. Umap: Uniform manifold approximation and
with rules. Nucl. Eng. Technol. 53, 866–878. https://doi.org/10.1016/j. projection for dimension reduction. arXiv Preprint ArXiv:1802.03426. https://doi.
net.2020.08.010. org/10.48550/arXiv.1802.03426.
Dang, H., Mecke, L., Lehmann, F., Goller, S., Buschek, D., 2022. How to prompt? Mikolov, T., Chen, K., Corrado, G., Dean, J., 2013. Efficient estimation of word
Opportunities and challenges of zero-and few-shot learning for human-AI interaction representations in vector space. arXiv Preprint ArXiv:1301.3781. https://doi.org/
in creative applications of generative models. arXiv Preprint ArXiv:2209.01390. 10.48550/arXiv.1301.3781.
https://doi.org/10.48550/arXiv.2209.01390. Ohms, J., 2021. Current methodologies for chemical compound searching in patents: a
Davis, F.D., 1989. Perceived usefulness, perceived ease of use, and user acceptance of case study. World Patent Inf. 66, 102055 https://doi.org/10.1016/j.
information technology. MIS Q. 13, 319–340. https://doi.org/10.2307/249008. wpi.2021.102055.
Deloose, A., Gysels, G., De Baets, B., Verwaeren, J., 2023. Combining natural language Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C.L., Mishkin, P., Lowe, R., 2022.
processing and multidimensional classifiers to predict and correct CMMS metadata. Training Language models to follow instructions with human feedback. arXiv
Comput. Ind. 145, 103830 https://doi.org/10.1016/j.compind.2022.103830. Preprint ArXiv:2203.02155. https://doi.org/10.48550/arXiv.2203.02155.
Devlin, J., Chang, M.W., Lee, K., Toutanova, K., 2018. Bert: pretraining of deep Ozcan, S., Suloglu, M., Sakar, C.O., Chatufale, S., 2021. Social media mining for ideation:
bidirectional transformers for language understanding. arXiv Preprint ArXiv: identification of sustainable solutions and opinions. Technovation 107, 102322.
1810.04805. https://doi.org/10.48550/arXiv.1810.04805. https://doi.org/10.1016/j.technovation.2021.102322.
Dewar, R.D., Dutton, J.E., 1986. The adoption of radical and incremental innovations: an Pavlik, J.V., 2023. Collaborating with ChatGPT: considering the implications of
empirical analysis. Manag. Sci. 32, 1422–1433. generative artificial intelligence for journalism and media education. Journal. Mass
Egger, R., Yu, J., 2022. A topic modeling comparison between lda, nmf, top2vec, and Commun. Educat. 78, 84–93. https://doi.org/10.1177/10776958221149577.
bertopic to demystify twitter posts. Front. Sociol 7, 886498. https://doi.org/ Piller, F.T., Bouschery, S.G., Blazevic, V., 2023. Hybrid intelligence for innovation:
10.3389/fsoc.2022.886498. augmenting NPD teams with artificial intelligence and machine learning. The PDMA
Else, H., 2023. Abstracts written by ChatGPT fool scientists. Nature 613, 423. https:// Handbook of Innovation and New Product. Development 407.
doi.org/10.1038/d41586-023-00056-7, 423. Porter, A.L., Garner, J., Carley, S.F., Newman, N.C., 2019. Emergence scoring to identify
Giordano, V., Chiarello, F., Melluso, N., Fantoni, G., Bonaccorsi, A., 2021. Text and frontier R&D topics and key players. Technol. Forecast. Soc. Change 146, 628–643.
dynamic network analysis for measuring technological convergence: a case study on https://doi.org/10.1016/j.techfore.2018.04.016.
defense patent data. IEEE Trans. Eng. Manag. 70, 1490–1503. https://doi.org/ Puccetti, G., Giordano, V., Spada, I., Chiarello, F., Fantoni, G., 2023. Technology
10.1109/TEM.2021.3078231. identification from patent texts: a novel named entity recognition method.
Graham, F., 2022. Daily briefing: will ChatGPT kill the essay assignment? Nature. Technology Identification from Patent Texts. Technol. Forecast. Soc. Change 186,
https://doi.org/10.1038/d41586-022-04437-2. 122160. https://doi.org/10.1016/j.techfore.2022.122160.
Grootendorst, M., 2022. BERTopic: neural topic modeling with a class-based TF-IDF Radford, A., Narasimhan, K., Salimans, T., Sutskever, I., 2018. Improving language
procedure. arXiv Preprint ArXiv:2203.05794. https://doi.org/10.48550/ understanding by generative pretraining. https://cdn.openai.com/research-cover
arXiv.2203.05794. s/language-unsupervised/language_understanding_paper.pdf.
Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D., 2019. Ranaei, S., Suominen, A., Porter, A., Carley, S., 2020. Evaluating technological
A survey of methods for explaining black box models. ACM Comput. Surv. 51, 1–42. emergence using text analytics: two case technologies and three approaches.
https://doi.org/10.1145/3236009. Scientometrics 122, 215–247. https://doi.org/10.1007/s11192-019-03275-w.

10
F. Chiarello et al. Technovation 133 (2024) 103002

Reynolds, L., McDonell, K., 2021. Prompt programming for large language models: Stokel-Walker, C., 2022. AI bot ChatGPT writes smart essays - should academics worry?
beyond the few-shot paradigm. In: Extended. Abstracts of the 2021 CHI Conference Nature. https://doi.org/10.1038/d41586-022-04397-7.
on Human Factors in Computing Systems, pp. 1–7. https://doi.org/10.1145/ Stokel-Walker, C., 2023. ChatGPT listed as author on research papers: many scientists
3411763.3451760. disapprove. Nature 613, 620–621. https://doi.org/10.1038/d41586-023-00107-z.
Rogers, E.M., 2004. A prospective and retrospective look at the diffusion model. Trabucchi, D., Buganza, T., 2020. Fostering digital platform innovation: from two to
J. Health Commun. 9 (S 1), 13–19. multi-sided platforms. Creativ. Innovat. Manag. 29, 345–358. https://doi.org/
Rotolo, D., Hicks, D., Martin, B.R., 2015. What is an emerging technology? Res. Pol. 44, 10.1111/caim.12320.
1827–1843. https://doi.org/10.1016/j.respol.2015.06.006. Tsai, R.T.H., Wu, S.H., Chou, W.C., Lin, Y.C., He, D., Hsiang, J., Sung, T.Y., Hsu, W.L.,
Salerno, M.S., Gomes, L.A., Da Silva, D.O., Bagno, R.B., Freitas, S.L.T.U., 2015. 2006. Various criteria in the evaluation of biomedical named entity recognition.
Innovation processes: which process for which project? Technovation 35, 59–70. BMC Bioinf. 7, 92. https://doi.org/10.1186/1471-2105-7-92.
https://doi.org/10.1016/j.technovation.2014.07.012. Van Dis, E.A., Bollen, J., Zuidema, W., van Rooij, R., Bockting, C.L., 2023. ChatGPT: five
Sallam, M., 2023. ChatGPT utility in healthcare education, research, and practice: priorities for research. Nature 614 (7947), 224–226. https://doi.org/10.1038/
systematic review on the promising perspectives and valid concerns. Healthcare 11, d41586-023-00288-7.
887. https://doi.org/10.3390/healthcare11060887. Verganti, R., Vendraminelli, L., Iansiti, M., 2020. Innovation and design in the age of
Sanderson, K., 2023. GPT-4 is here: what scientists think. Nature 615, 773. https://doi. artificial intelligence. J. Prod. Innovat. Manag. 37, 212–227. https://doi.org/
org/10.1038/d41586-023-00816-5. 10.1111/jpim.12523.
Siddharth, L., Blessing, L., Luo, J., 2022. Natural language processing in-and-for design Von Hippel, E., 1986. Lead users: a source of novel product concepts. Manag. Sci. 32,
research. Des. Sci. 8, e21 https://doi.org/10.1017/dsj.2022.16. 791–805.
Small, H., Boyack, K.W., Klavans, R., 2014. Identifying emerging topics in science and Yoon, J., Park, H., Kim, K., 2013. Identifying technological competition trends for R&D
technology. Res. Pol. 43, 1450–1467. https://doi.org/10.1016/j. planning using dynamic patent maps: SAO-based content analysis. Scientometrics
respol.2014.02.005. 94, 313–331. https://doi.org/10.1007/s11192-012-0830-6.
Song, B., Luan, C., Liang, D., 2023. Identification of emerging technology topics (ETTs) Zhao, K., Ding, H., Ye, K., Cui, X., Fu, Z., 2021. A latent variable model with hierarchical
using BERT-based model and sematic analysis: a perspective of multiple-field structure and GPT-2 for long text generation. In: Artif. Neural Netw. Mach. Learn.–
characteristics of patented inventions (MFCOPIs). Scientometrics 128, 5883–5904. ICANN. Proceedings of the Part V 30: 30th International Conference on Artificial
https://doi.org/10.1007/s11192-023-04819-x. Neural Networks, Bratislava, Slovakia, 14–17 Sep, 2021 2021. Springer International
Spada, I., Barandoni, S., Giordano, V., Chiarello, F., Fantoni, G., Martini, A., 2023. What Publishing, pp. 297–308. https://doi.org/10.1007/978-3-030-86383-8_24.
users want: a Natural Language Processing approach to discover Users’Needs from Zhao, W.X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Wen, J.R., 2023. A survey of
online reviews. Proceedings of the Design Society 3, 3879–3888. https://doi.org/ Large Language Models. arXiv Preprint ArXiv:2303.18223.
10.1017/pds.2023.389.

11