Real-World Natural Language Processing: Practical applications with deep learning

Part 1 Basics

Welcome to the beautiful and exciting world of natural language processing (NLP)! NLP is a subfield of artificial intelligence (AI) that concerns computational approaches to processing, understanding, and generating human languages. NLP is used in many technologies you interact with in your daily life—spam filtering, conversational assistants, search engines, and machine translation. This first part of the book is intended to give you a gentle introduction to the field and bring you up to speed with how to build practical NLP applications.

In chapter 1, we’ll begin by introducing the what and why of NLP—what is NLP, what is not NLP, how NLP technologies are used, and how it’s related to other fields of AI.

In chapter 2, you’ll build a complete, working NLP application—a sentiment analyzer—within an hour with the help of a powerful NLP framework, AllenNLP. You’ll also learn to use basic machine learning (ML) concepts, including word embeddings and recurrent neural networks (RNNs). Don’t worry if this sounds intimidating—we’ll introduce you to the concepts gradually and provide an intuitive explanation.

Chapter 3 provides a deep dive into the one of the most important concepts for deep learning approaches to NLP—word and sentence embeddings. The chapter demonstrates how to use and even train them using your own data.

Chapters 4 and 5 cover fundamental NLP tasks, sentence classification and sequence labeling. Though simple, these tasks have a wide range of applications, including sentiment analysis, part-of-speech tagging, and named entity recognition.

This part familiarizes you with some basic concepts of modern NLP and we’ll build useful NLP applications along the way.

1 Introduction to natural language processing

This chapter covers

What natural language processing (NLP) is, what it is not, and why it’s such an interesting, yet challenging, field

How NLP relates to other fields, including artificial intelligence (AI) and machine learning (ML)

What typical NLP applications and tasks are

How a typical NLP application is developed and structured

This is not an introductory book to machine learning or deep learning. You won’t learn how to write neural networks in mathematical terms or how to compute gradients, for example. But don’t worry, even if you don’t have any idea what they are. I’ll explain those concepts as needed, not mathematically but conceptually. In fact, this book contains no mathematical formulae—not a single one. Also, thanks to modern deep learning libraries, you don’t really need to understand the math to build practical NLP applications. If you are interested in learning the theories and the math behind machine learning and deep learning, you can find a number of great resources out there.

But you do need to be at least comfortable enough to write in Python and know its ecosystems. However, you don’t need to be an expert in software engineering topics. In fact, this book’s purpose is to introduce software engineering best practices for developing NLP applications. You also don’t need to know NLP in advance. Again, this book is designed to be a gentle introduction to the field.

You need Python version 3.6.1 or higher and AllenNLP 2.5.0 or higher to run the code examples in this book. Note that we do not support Python 2, mainly because AllenNLP (https://allennlp.org/), the deep natural language processing framework I’m going to heavily use in this book, supports only Python 3. If you haven’t done so, I strongly recommend upgrading to Python 3 and familiarizing yourself with the latest language features such as type hints and new string-formatting syntax. This will be helpful, even if you are developing non-NLP applications.

Don’t worry if you don’t have a Python development environment ready. Most of the examples in this book can be run via the Google Colab platform (https://colab.research.google.com). You need only a web browser to build and experiment with NLP models!

This book will use PyTorch (https://pytorch.org/) as its main choice of deep learning framework. This was a difficult decision for me, because several deep learning frameworks are equally great choices for building NLP applications, namely, TensorFlow, Keras, and Chainer. A few factors make PyTorch stand out among those frameworks—it’s a flexible and dynamic framework that makes it easier to prototype and debug NLP models; it’s becoming increasingly popular within the research community, so it’s easy to find open source implementations of major models; and the deep NLP framework AllenNLP mentioned earlier is built on top of PyTorch.

1.1 What is natural language processing (NLP)?

NLP is a principled approach to processing human language. Formally, it is a subfield of artificial intelligence (AI) that refers to computational approaches to process, understand, and generate human language. The reason it is part of AI is because language processing is considered a huge part of human intelligence. The use of language is arguably the most salient skill that separates humans from other animals.

1.1.1 What is NLP?

NLP includes a range of algorithms, tasks, and problems that take human-produced text as an input and produce some useful information, such as labels, semantic representations, and so on, as an output. Other tasks, such as translation, summarization, and text generation, directly produce text as output. In any case, the focus is on producing some output that is useful per se (e.g., a translation) or as input to other downstream tasks (e.g., parsing). I’ll touch upon some popular NLP applications and tasks in section 1.3.

You might wonder why NLP explicitly has natural in its name. What does it mean for a language to be natural? Are there any unnatural languages? Is English natural? Which is more natural: Spanish or French?

The word natural here is used to contrast natural languages with formal languages. In this sense, all the languages humans speak are natural. Many experts believe that language emerged naturally tens of thousands of years ago and has evolved organically ever since. Formal languages, on the other hand, are types of languages that are invented by humans and have strictly and explicitly defined syntax (i.e., what is grammatical) and semantics (i.e., what it means).

Programming languages such as C and Python are good examples of formal languages. These languages are defined in such a strict way that it is always clear what is grammatical and ungrammatical. When you run a compiler or an interpreter on the code you write in those languages, you either get a syntax error or not. The compiler won’t say something like, Hmm, this code is maybe 50% grammatical. Also, the behavior of your program is always the same if it’s run on the same code, assuming external factors such as the random seed and the system states remain constant. Your interpreter won’t show one result 50% of the time and another the other 50% of the time.

This is not the case for human languages. You can write a sentence that is maybe grammatical. For example, do you consider the phrase The person I spoke to ungrammatical? There are some grammar topics where even experts disagree with each other. This is what makes human languages interesting but challenging, and why the entire field of NLP even exists. Human languages are ambiguous, meaning that their interpretation is often not unique. Both structures (how sentences are formed) and semantics (what sentences mean) can have ambiguities in human language. As an example, let’s take a close look at the next sentence:

He saw a girl with a telescope.

When you read this sentence, who do you think has a telescope? Is it the boy, who’s using a telescope to see a girl (from somewhere far), or the girl, who has a telescope and is seen by the boy? There seem to be at least two interpretations of this sentence as shown in figure 1.1.

Figure 1.1 Two interpretations of He saw a girl with a telescope.

The reason you are confused upon reading this sentence is because you don’t know what the phrase with a telescope is about. More technically, you don’t know what this prepositional phrase (PP) modifies. This is called a PP-attachment problem and is a classic example of syntactic ambiguity. A syntactically ambiguous sentence has more than one interpretation of how the sentence is structured. You can interpret the sentence in multiple ways, depending on which structure of the sentence you believe.

Another type of ambiguity that may arise in natural language is semantic ambiguity. This is when the meaning of a word or a sentence, not its structure, is ambiguous. For example, let’s look at the following sentence:

I saw a bat.

There is no question how this sentence is structured. The subject of the sentence is I and the object is a bat, connected by the verb saw. In other words, there is no syntactical ambiguity in it. But how about its meaning? Saw has at least two meanings. One is the past tense of the verb to see. The other is to cut some object with a saw. Similarly, a bat can mean two very different things: is it a nocturnal flying mammal or a piece of wood used to hit a ball? All in all, does this sentence mean that I observed a nocturnal flying mammal or that I cut a baseball or cricket bat? Or even (cruelly) that I cut a nocturnal animal with a saw? You never know, at least from this sentence alone.

Ambiguity is what makes natural languages rich but also challenging to process. We can’t simply run a compiler or an interpreter on a piece of text and just get it. We need to face the complexities and subtleties of human languages. We need a scientific, principled approach to deal with them. That’s what NLP is all about.

Welcome to the beautiful world of natural languages.

1.1.2 What is not NLP?

Now let’s consider the following scenario and think how you’d approach this problem: you are working as a junior developer at a midsized company that has a consumer-facing product line. It’s 3 p.m. on a Friday. The rest of the team is becoming more and more restless as the weekend approaches. That’s when your boss drops by at your cubicle.

Hey, got a minute? I’ve got something interesting to show you. I just sent it to you.

Your boss just sent you an email with a huge zip file attached to it.

OK, so this is a giant TSV file. It contains all the responses to the survey questions about our product. I just got this data from the marketing team.

Obviously, the marketing team has been collecting user opinions about one of the products through a series of survey questions online.

The survey questions include standard ones like ‘How did you know about our product?’ and ‘How do you like our product?’ There is also a free-response question, where our customers can write whatever they feel about our product. The thing is, the marketing team realized there was a bug in the online system and the answers to the second question were not recorded in the database at all.

Wait, so there’s no way to tell how the customers are feeling about our product? This sounds weirdly familiar. This must be a copy-and-paste error. When you first created an online data-collection interface, you copied and pasted the backend code without modifying the ID parameters, resulting in a loss of some data fields.

So, your boss continues. I was wondering if we can recover the lost data somehow. The marketing team is a little desperate now because they need to report the results to the VP early next week.

At this point, your bad feeling has been confirmed. Unless you come up with a way to get this done as quickly as possible, your weekend plans will be ruined.

Didn’t you say you were interested in some machine learning? I think this is a perfect project for you. Anyway, it’d be great if you can give it a stab and let me know what you find. Do you think you can have some results by Monday?

Well, I’ll give it a try.

You know no is not an acceptable answer here. Satisfied with your answer, your boss smiles and walks off.

You start by skimming the TSV file. To your relief, its structure is fairly standard—it has several fields such as timestamps and submission IDs. At the end of each line is a lengthy field for the free-response question. Here they are, you think. At least you know where to look for some clues.

After a quick glance over the field, you find responses such as A very good product! and Very bad. It crashes all the time! Not too bad, you think. At least you can capture these simple cases. You start by writing the following method that captures those two cases:

def get_sentiment(text):

    "Return 1 if text is positive, -1 if negative.

      Otherwise, return 0."

    if 'good' in text:

        return 1

    elif 'bad' in text:

        return -1

    return 0

Then you run this method on the responses in the file and log the results, along with the original input. As intended, this method seems to be able to capture a dozen or so of the responses that contains good or bad.

But then you start to see something alarming, as shown next:

I can’t think of a single good reason to use this product: positive

It’s not bad.: negative

Oops, you think. Negation. Yeah, of course. But this is pretty easy to deal with. You modify the method as follows:

def get_sentiment(text):

    "Return 1 if text is positive, -1 if negative.

      Otherwise, return 0."

    sentiment = 0

    if 'good' in text:

        sentiment = 1

    elif 'bad' in text:

        sentiment = -1

    if 'not' in text or n't in text:

        sentiment *= -1

    return sentiment

You run the script again. This time, it seems to be behaving as intended, until you see an even more complicated example:

The product is not only cheap but also very good!: negative

Hmm, you think. This is probably not as straightforward as I initially thought after all. Maybe the negation has to be somewhere near good or bad for it to be effective. Wondering what steps you could take next, you scroll down to see more examples, which is when you see responses like these:

I always wanted this feature badly!: negative

It’s very badly made.: negative

You silently curse to yourself. How could a single word in a language have two completely opposite meanings? At this point, your little hope for enjoying the weekend has already disappeared. You are already wondering what excuses you use with your boss next Monday.

As a reader of this book, you’ll know better. You’ll know that NLP is not about throwing a bunch of ifs and thens at natural language text. It is a more principled approach to processing natural language. In the following chapters, you’ll learn how you should approach this problem before writing a single line of code and how to build a custom-made NLP application just for your task at hand.

1.1.3 AI, ML, DL, and NLP

Beforedelving into the details of NLP, it’d be useful to clarify how it relates to other, similar fields. Most of you have at least heard about artificial intelligence (AI) and machine learning (ML). You may also have heard of deep learning (DL), because it’s generating a lot of buzz in popular media these days. Figure 1.2 illustrates how those different fields overlap with each other.

Figure 1.2 The relationship among different fields: AI, ML, DL, and NLP

Artificial intelligence (AI) is a broad umbrella field that is concerned with achieving human-like intelligence using machines. It encompasses a wide range of subfields, including machine learning, natural language processing, computer vision, and speech recognition. The field also includes subfields such as reasoning, planning, and search, which do not fall under either machine learning or natural language processing and are not in the scope of this book.

Machine learning (ML) is usually considered a subfield of artificial intelligence that is about improving computer algorithms through experience and data. This includes learning a general function that maps inputs to outputs based on past experience (supervised learning), drawing hidden patterns and structures from data (unsupervised learning), and learning how to act in a dynamic environment based on indirect rewards (reinforcement learning). Throughout this book, we’ll make a heavy use of supervised machine learning, which is the main paradigm for training NLP models.

Deep learning (DL) is a subfield of machine learning that usually uses deep neural networks. These neural network models are called deep because they consist of a number of layers. A layer is just a fancy word for a substructure of neural networks. By having many stacked layers, deep neural networks can learn complex representations of data and can capture highly complicated relationships between the input and the output.

As the amount of available data and computational resources increases, modern NLP makes a heavier and heavier use of machine learning and deep learning. Modern NLP applications and tasks are usually built on top of machine learning pipelines and trained from data. But also notice in figure 1.2 that a part of NLP does not overlap with machine learning. Traditional methods such as counting words and measuring similarities between text are usually not considered to be machine learning techniques per se, although they can be important building blocks for ML-based models.

I’d also like to mention some other fields that are related to NLP. One such field is computational linguistics (CL). As its name suggests, computational linguistics is a subfield of linguistics that uses computational approaches to study human language. The main distinction between CL and NLP is that the former encompasses scientific approaches to study language, whereas the latter is concerned with engineering approaches for making computers perform something useful related to language. People often use those terms interchangeably, partly due to some historical reasons. For example, the most prestigious conference in the field of NLP is called ACL, which actually stands for the Association for Computational Linguistics!

Another related field is text mining. Text mining is a type of data mining targeted at textual data. Its focus is on drawing useful insights from unstructured textual data, which is a type of text data that is not formatted in a form that is easily interpretable by computers. Such data is usually collected from various sources, such as crawling the web and social media. Although its purpose is slightly different from that of NLP, these two fields are similar, and we can use the same tools and algorithms forboth.

1.1.4 Why NLP?

If you are reading this, you have at least some interest in NLP. Why is NLP exciting? Why is it worth learning more about NLP and, specifically, real-world NLP?

The first reason is that NLP is booming. Even without the recent AI and ML boom, NLP is more important than ever. We are witnessing the advent of practical NLP applications in our daily lives, such as conversational agents (think Apple Siri, Amazon Alexa, and Google Assistant) and near human-level machine translation (think Google Translate). A number of NLP applications are already an integral part of our day-to-day activities, such as spam filtering, search engines, and spelling correction, as we’ll discuss later. The number of Stanford students enrolled in NLP classes grew fivefold from 2008 to 2018 (http://realworldnlpbook.com/ch1.html#tweet1). Similarly, the number of attendees for EMNLP (Empirical Methods in Natural Language Processing), one of the top NLP conferences, doubled within just one year (http://realworldnlpbook .com/ch1.html#tweet2). Other major NLP conferences are also experiencing similar increases in participants and paper submissions (http://realworldnlpbook.com/ch1 .html#nivre17).

The second reason is that NLP is an evolving field. The field of NLP itself has a long history. The first experiment to build a machine translation system, called The Georgetown-IBM Experiment, was attempted back in 1954. For more than 30 years since this experiment, most NLP systems relied on handwritten rules. Yes, it was not much different from what you just saw in section 1.1.1. The first milestone, which came in the late 1980s, was the use of statistical methods and machine learning for NLP. Many NLP systems started leveraging statistical models trained from data. This led to some recent successes in NLP, including, most notably, IBM Watson. The second milestone was more drastic. Starting around the late 2000s, the use of so-called deep learning, that is, deep neural network models, took the field by storm. By the mid-2010s, deep neural network models became the new standard in the field.

This second milestone was so drastic and fast that it’s worth noting here. New neural network-based NLP models are not only more effective but also a lot simpler. For example, it used to take a lot of expertise and effort to replicate even a simple, baseline machine translation model. One of the most popular open source software packages for statistical machine translation, called Moses (http://www.statmt.org/moses/), is a behemoth, consisting of 100,000s of lines of code and dozens of supporting modules and tools. Experts spent hours just installing the software and making it work. On the other hand, as of 2018, anyone with some prior programming experience could run a neural machine translation system more powerful than traditional statistical models with a fraction of the code size—less than a few thousand lines of code (e.g., see TensorFlow’s neural machine translation tutorial at https://github.com/tensorflow/nmt). Also, the new neural network models are trained end-to-end, which means that those big, monolithic networks take the input and directly produce the output. Entire models are trained to match the desired output. On the other hand, traditional machine learning models consist of (at least) several submodules. These submodules are trained separately using different machine learning algorithms. In this book, I’ll mainly discuss modern neural network-based approaches to NLP but also touch upon some traditional concepts as well.

The third and final reason is that NLP is challenging. Understanding and producing language is the central problem of artificial intelligence, as we saw in the previous section. The accuracy and performance in major NLP tasks such as speech recognition and machine translation got drastically better in the past decade or so. But human-level understanding of language is far from being solved.

To verify this quickly, open up your favorite machine translation service (or simply Google Translate), and type this sentence: I saw her duck. Try to translate it to Spanish or some other language you understand. You should see words like pato, which means a duck in Spanish. But did you notice another interpretation of this sentence? See figure 1.3 for the two interpretations. The word duck here could be a verb meaning to crouch down. Try adding another sentence after this, such as She tried to avoid a flying ball. Did the machine translation change the first translation in any way? The answer is probably no. You should still see the same pato in the translation. As you can see, most (if not all) commercial machine translation systems that are available as of today do not understand the context outside of the sentence that is being translated. A lot of research effort is spent on this problem in academia, but this is still one of many problems in NLP that is considered unsolved.

Figure 1.3 Two interpretations of I saw her duck.

Compared to other AI fields such as robotics and computer vision, language has its own quirks. Unlike images, utterances and sentences have variable length. You can say a very short sentence (Hello.) or a very long one (A quick brown fox . . .). Most machine learning algorithms are not good at dealing with something of variable length, and you need to come up with ways to represent languages with something more fixed. If you look back at the history of the field, NLP is largely concerned with the problem of how to represent language mathematically. Vector space models and word embeddings (discussed in chapter 3) are some examples of this.

Another characteristic of language is that it is discrete. What this means is that things in languages are separate as concepts. For example, if you take a word rat and change its first letter to the next one, you’ll have sat. In computer memory, the difference is just a single bit. However, there is no relationship between those two words except they both end with at, and maybe a rat can sit. There is no such thing as something that is in between rat and sat. These two are totally discrete, separate concepts that happen to have similar spelling. On the other hand, if you take an image of a car and change the value of a pixel by a single bit, you still have a car that is almost identical to the one before this change. Maybe it has a slightly different color. In other words, images and sounds are continuous, meaning that you can make small modifications without greatly affecting what they are. Many mathematical toolkits, such as vectors, matrices, and functions, are good at dealing with something continuous. The history of NLP is actually a history of challenging this discreteness of language, and only recently have we begun to see some successes on this front, for example, with word embeddings.

1.2 How NLP is used

As I mentioned previously, NLP is already an integral part of our daily life. In modern life, a larger and larger portion of our daily communication is done online, and our online communication is still largely conducted in natural language text. Think of your favorite social networking services, such as Facebook and Twitter. Although you can post photos and videos, a large portion of communication is still in text. As long as you are dealing with text, there is a need for NLP. For example, how do you know if a particular post is spam? How do you know which posts are the ones you are most likely to like? How do you know which ads you are most likely to click?

Because many large internet companies need to deal with text in one way or another, chances are many of them are already using NLP. You can also confirm this from their careers page—you’ll see that they are always hiring NLP engineers and data scientists. NLP is also used to a varying extent in many other industries and products including, but not limited to, customer service, e-commerce, education, entertainment, finance, and health care, which all involve text in some ways.

Many NLP systems and services can be classified into or built by combining some major types of NLP applications and tasks. In this section, I’m going to introduce some of the most popular applications of NLP as well as common NLP tasks.

1.2.1 NLP applications

An NLP application is a software application whose main purpose is to process natural language text and draw some useful information from it. Similar to general software applications, it can be implemented in various ways, such as an offline data-processing script, an offline standalone application, a backend service, or a full-stack service with a frontend, depending on its scope and use cases. It can be built for end users to use directly, for other backend services to consume its output, or for other businesses to use as a SaaS (software as a service).

You can use many NLP applications out of the box, such as machine translation software and major SaaS products (e.g., Google Cloud API), if your requirement is generic and doesn’t require a high level of customization. You can also build your own NLP applications if you need customizations and/or you need to deal with a specific target domain. This is exactly what you’ll learn throughout this book!

Machine translation

Machine translation is probably one of the most popular and easy-to-understand NLP applications. Machine translation (MT) systems translate a given text from one language to another language. An MT system can be implemented as a full-stack service (e.g., Google Translate), as well as a pure backend service (e.g., NLP SaaS products). The language the input text is written in is called the source language, whereas the one for the output is called the target language. MT encompasses a wide range of NLP problems, including language understanding and generation, because MT systems need to understand the input and then generate the output. MT is one of the most well-studied areas in NLP, and it was one of the earliest applications of NLP as well.

One challenge in MT is the tradeoff between fluency and adequacy. Translation needs to be fluent, meaning that the output has to sound natural in the target language. Translation also needs to be adequate, meaning that the output has to reflect the meaning expressed by the input as closely as possible. These two are often in conflict, especially when the source and the target languages are not very similar (e.g., English and Mandarin Chinese). You can write a sentence that is a precise, verbatim translation of the input, but doing so often leads to something that doesn’t sound natural in the target language. On the other hand, you can make up something that sounds natural but might not reflect the precise meaning. Good human translators address this tradeoff in a creative way. It’s their job to come up with translations that are natural in the target language while reflecting the meaning of the original.

Grammatical and spelling error correction

Most major web browsers nowadays support spelling correction. Even if you forget how to spell Mississippi, you can do your best and type what you remember, and the browser highlights it with a correction. Some word-processing software applications, including recent versions of Microsoft Word, do more than just correct spelling. They point out grammatical errors such as uses of it’s instead of its. This is not an easy feat, because both words are, in a sense, correct (no mistakes in spelling), and the system needs to infer whether they are used correctly from the context. Some commercial products (most notably, Grammarly, https://www.grammarly.com/) specialize in grammatical error correction. Some products go a long way and point out incorrect usage of punctuation and even writing styles. These products are popular among both native and non-native speakers of the language.

Research into grammatical error correction has been active due to the increasing number of non-native English speakers. Traditionally, grammatical error correction systems for non-native English speakers dealt with individual types of mistakes one by one. For example, you could think of a subcomponent of the system that detects and corrects only incorrect uses of articles (a, an, the, etc.), which is very common among non-native English speakers. More recent approaches to grammatical error correction are similar to the ones for machine translation. You can think of the (potentially incorrect) input as one language and the corrected output as another. Then your job is to translate between these two languages!

Search engine

Another application of NLP that is already an integral part of our daily lives is search engines. Few people would think of search engines as an NLP application, yet NLP plays such an important role in making search engines useful that they are worth mentioning here.

Page analysis is one area where NLP is heavily used for search engines. Ever wonder why you don’t see any hot dog pages when you search for dogs? If you have any experience building your own full-text search engines using open source software such as Solr and Elasticsearch, and if you simply used a word-based index, your search result pages would be littered with hot dogs, even when you want just dogs. Major commercial search engines solve this problem by running the page content being indexed through NLP pipelines that recognize that hot dogs are not a type of dogs. But the extent and types of NLP pipelines that go into page analysis is confidential information for search engines and is difficult to know.

Query analysis is another NLP application in search engines. If you have noticed Google showing a box with pictures and bios when you search for a celebrity or a box with the latest news stories when you search for certain current events, that’s query analysis in action. Query analysis identifies the intent (what the user wants) of the query and shows relevant information accordingly. A common way to implement query analysis is to make it a classification problem, where an NLP pipeline classifies queries into classes of intent (e.g., celebrity, news, weather, videos), although again, the details of how commercial search engines run query analysis are usually highly confidential.

Finally, search engines are not only about analyzing pages and classifying queries. They have many other functionalities that make your searches easier, one of which is query correction. This comes into play when you make a spelling or a grammatical mistake when formulating the query, and Google and other major search engines show corrections with labels such as showing results for: and Did you mean. How this works is somewhat similar to grammatical error correction that I mentioned earlier, except it is optimized for the types of mistakes and queries that search engine users use.

Dialog systems

Dialog systems are machines that humans can have conversations with. The field of dialog systems has a long history. One of the earliest dialog systems, ELIZA, was developed in 1966.

But it’s only recently that dialog systems have found their ways into our daily lives. We have seen an almost exponential increase in their popularity in recent years, mainly driven by the availability of consumer-facing conversational AI products such as Amazon Alexa and Google Assistant. In fact, according to a survey in 2018, 20% of US homes already own a smart speaker. You may also remember being mind-blown watching the keynote at Google IO in 2018, where Google’s conversational AI called Google Duplex was shown making a phone call to a hair salon and a restaurant, having natural conversations with the staff at the business, and making an appointment on behalf of its user.

The two main types of dialog systems are task-oriented and chatbots. Task-oriented dialog systems are used to achieve specific goals (for example, reserving a plane ticket), obtaining some information, and, as we saw, making a reservation at a restaurant. Task-oriented dialog systems are usually built as an NLP pipeline consisting of several components,