This project is about semantic analysis of English. We will work with data annotated and released as part of an upcoming DiMSUM SemEval shared task. It is concerned with predicting a broad-coverage representation of lexical semantics. The representation consists of two closely connected facets: a segmentation into minimal semantic units, and a labeling of some of those units with semantic classes known as supersenses.
We will take a Learning to Search approach to solve this joint prediction problem, and implement it using the Python interface to the Vowpal Wabbit (VW) library.
You can download the datasets we'll use and scaffolding code as a tar archive (UPDATED ON 12/1!). You can install your own copy of VW, or use the shared installation on the newjunkfood machines by adding /fs/junkfood/yogarshi/Public/vowpal_wabbit/python to your PYTHONPATH.
Evaluation: Your code will be autograded for technical correctness. Please do not change the names of any provided functions or classes within the code, or you will wreak havoc on the autograder. However, the correctness of your implementation -- not the autograder's output -- will be the final judge of your score. If necessary, we will review and grade assignments individually to ensure that you receive due credit for your work.
Academic Dishonesty: We will be checking your code against other submissions in the class for logical redundancy. If you copy someone else's code and submit it with minor changes, we will know. These cheat detectors are quite hard to fool, so please don't try. We trust you all to submit your own work only; please don't let us down. If you do, we will pursue the strongest consequences available to us.
Getting Help: You are not alone! If you find yourself stuck on something, contact the course staff for help. Office hours, class time, and Piazza are there for your support; please use them. We want these projects to be rewarding and instructional, not frustrating and demoralizing. But, we don't know when or how to help unless you ask. One more piece of advice: if you don't know what a variable is, print it out.
As a warm-up, we will solve the task of tagging contiguous multiword expressions (MWEs) in a learning to search framework.
First, walk through the part-of-speech tagging example to make sure you know how to set up a sequence labeling task.
You are now ready to address the multiword expression tagging task. In the first 2 parts of the project, we will address a simplified version of the DiMSUM task, and we will only consider contiguous multiword expressions. The training and test data used will not contain any discontiguous nor nested MWEs. In this setting, we can view MWE detection as a simple tagging task, where each token in a sentence is assigned one of three tags: B stands for "BEGIN" and marks the first word of a MWE, I stands for "INSIDE" and marks MWE words after the first word, and O stands for "OUTSIDE" and marks words that are not in a MWE.
Open the mwedimsum.py file, and take a look at the code. Your task is to complete all the TODO items, so ythat you can run through several iterations of training, and make MWE predictions on the held-out dev set, as well as on the test data.
You can check whether your code is doing something reasonable by inspecting the predictions made on the subset of training examples that are held-out for development. In our reference implementation, we obtain a Hamming Loss of 869 / 7172 (percentage of incorrect tags) on the held-out dev data, after training for 5 iterations.
What you should hand in: mwedimsum.py.
In this section, we improve and augment the contiguous MWE detector so that it also predicts supersense tags. Following Schneider et al., we can perform supersense tagging and multiword expression jointly. In our sequence labeling framework, this will require modifying the tags used.
Specifically, you are asked to complete the following tasks (UPDATED on 12/3: tasks have been reordered, so that testing your implementation is easier and faster!):
What you should hand in: dimsum.py, dimsim16.test.contiguous.out, and whatwedid.txt. The first file should be a python script that takes 3 arguments:
dimsum.py dimsum16.train.contiguous dimsum16.test.contiguous dimsum16.test.contiguous.outwhere
In this optional part of the project, your task is to tackle the full DiMSUM task. Specifically, this requires extending your code to handle MWEs that are not contiguous. Take a look at the dimsum16.train file. Unlike the simplified version of the data we've been working on so far (dimsum16.train.contiguous), the actual training set contains MWE tags such as o, b, and i, which are used to indicate gaps within MWEs as well as MWEs that occur inside the gap of another MWE.
You have the opportunity to receive up to 10\% extra-credit by tackling this task.
What you should hand in:
If you solve this problem, you are almost in a position to participate in the shared task, and see how well your system does compared to other research systems! The evaluation will take place in January 2016. Talk to the instructors if you want to give this a try.
If you want to explore Learning to Search in more depth, you can start with this tutorial, and then turn to the recommended readings on slide 52.