PhD Proposal: Learning to fulfill natural language requests via communication with humans

Despite their promising advantages, deploying natural language interfaces for automated agents is notoriously challenging because it requires training these agents to understand and generate natural language. For basic applications, these agents must be able to interpret requests specified in natural language and carry them out in situated environments. Despite the progress that has been made, machine learning-based agents have been shown to struggle with fulfilling high-level, under-specified requests in novel complex environments.In practice, these agents would be surrounded by humans, an immense source of knowledge. Unfortunately, they currently possess limited capabilities of leveraging human assistance to better accomplish tasks. In contrast, when humans encounter situations that are beyond their knowledge and skill levels, they can seek help from other humans in the same environment through various communicative acts such as asking clarifying questions, seeking additional instructions, or requesting feedback. My work aims to empower machine learning-based agents with similar capabilities.I propose two novel machine learning paradigms that enable agents to effectively and efficiently leverage language-based human assistance in fulfilling natural language requests. The first paradigm considers human assistance in forms of clarifying instructions. I present an algorithm to train agents to ask nearby humans for additional instructions when they get lost, and to interpret the instructions to make progress. The second paradigm, which is work in progress, is a general interactive learning framework that allows using language-based feedback to train request-fulfilling agents. I study a specific type of feedback called language activity description, which is a language utterance of a human teacher describing an agent’s execution of a request. The use of this new communication model gives rise to a new family of interactive learning algorithms that offer complementary advantages compared to traditional algorithms like imitation learning or reinforcement learning.Examining Committee:

Chair: Dr. Hal Daumé III Dept rep: Dr. Tom Goldstein Members: Dr. Abhinav Shrivastava