Building and evaluating conversational agents / Joao Sedoc.

Format:

Book

Thesis/Dissertation

Author/Creator:

Sedoc, Joao, author.

Contributor:

University of Pennsylvania. Department of Computer and Information Science, degree granting institution.

Ungar, Lyle, degree supervisor.

Language:

English

Subjects (All):

Computer science.

Computer and Information Science--Penn dissertations.

Penn dissertations--Computer and Information Science.

Local Subjects:

Computer science.

Computer and Information Science--Penn dissertations.

Penn dissertations--Computer and Information Science.

Genre:

Academic theses.

Physical Description:

1 online resource (120 pages)

Contained In:

Dissertations Abstracts International 81-09B.

Place of Publication:

[Philadelphia, Pennsylvania] : University of Pennsylvania ; Ann Arbor : ProQuest Dissertations & Theses, 2019.

Language Note:

English

System Details:

Mode of access: World Wide Web.

text file

Summary:

With the advent of personal assistants such as Siri and Alexa, there has been a renewed focus on dialog systems, including open domain conversational agents which can ideally converse about any topic. Although many conversational "chit-chat'' agents are non-task driven, they often have a goal of engagement, entertainment, trust-building, or information seeking about the conversational partner. Recent end-to-end dialog generation methods use neural recurrent neural network encoder-decoder models~\\citep{vinyals2015neural}. However, these neural dialog generation (NDG) models lack the ability to create topical and stylistic coherence in conversations. Furthermore, the evaluation of these methods has proven difficult due to a lack of automatic metrics for assessing conversation agents. This dissertation attempts to tackle two primary areas of research: (1) developing novel deep learning architectures to build chatbots with topical and stylistic coherence, and (2) producing methods for the systematic evaluation and comparison of novel chatbot models. Maintaining topical coherence and style motivate our theoretical research into novel deep learning methods. Specifically, we address topical coherence with a novel hierarchical ensemble NDG model which explicitly models topical sequencing. Next, address two different aspects of stylistic coherence, first starting with maintaining branded style and then focusing on demographic matching. To this end, we create an NDG agent for both mimicking Star Trek style and Youngbot for evincing youthful style in both Twitter and chat. The second half of the thesis focuses on evaluation, presenting a theoretical framework for the systematic evaluation of open-domain conversational agents, including the usage of Item Response Theory \\citep{lord1968statistical} for efficient chatbot evaluation and evaluation set creation. We also present ChatEval, a software tool for chatbot evaluation.

Notes:

Source: Dissertations Abstracts International, Volume: 81-09, Section: B.

Advisors: Ungar, Lyle; Committee members: Dean Foster; Jean Gallier; Chris Callison-Burch; Chris Dyer.

Department: Computer and Information Science.

Ph.D. University of Pennsylvania 2019.

Local Notes:

School code: 0175

ISBN:

9781392623435

Access Restriction:

Restricted for use by site license.

This item is not available from ProQuest Dissertations & Theses.

This item must not be sold to any third party vendors.

This item must not be added to any third party search indexes.