:::::::::::::: misc.diane :::::::::::::: Time: For TOOT, which I did after trains, it took me much of the afternoon. In general, the CGU analysis was less time consuming than I expected. The trains IU analysis was long. Comments: I think we should have another homework with a CGU analysis given to us. Maybe you are more optimistic than I am, but I bet the disagreement in the CGU codings will make our IU analyses impossible to compare. I am not sure what I think about using the CGU's as the basic units for the IU analysis. At times the CGU level seemed too big. For example, I would have liked to put an IU break in my trains CGU 30. I also found it very unintuitive when there were mainly implicit groundings, as in TOOT, due to all the overlapping. I was not happy with my TOOT CGU's 14 and 20. It would have been helpful if you gave an example about coding speech recognition errors. It also would have been helpful to see an example TOOT dialogue coded in the manual. There seemed to be some omissions of overlapping speech in the TRAINS transcripts. I coded per the transcript, not the recording. :::::::::::::: misc.heeman :::::::::::::: Trains segmentation has a couple of problems. The overlapping speech was not marked, and twice came in the wrong order. u.8.1 should come before s.7.2 u.49.1 should come before s.48.2 Also, the okays were sometimes not put into their own unit, in particular s.18.1 and u.23.1. The Toot dialog has many instances in which the system misunderstood the user, but the system never really realizes this. This makes coding CGU somewhat difficult, since common ground is not reached. I took the approach that only misunderstandings that are corrected right away are coded in the same CGU. The Trains dialog took me a half an hour to do the CGU annotation, while Toot took me 15 minutes. Note that I wrote a small program that would give a default CGU segmentation, in which all tokens for a speaker's turn plus the first token from the next turn are put into a CGU. This really speeded up the process, since I just had to mark diviations from this. Of course, a graphical tool would also have helped. For annotating IU's, a tool would be really helpful. I ended up writing a simple script to take the CGU annotations in pull in the corresponding text. I then got rid of the text that was just serving as acknowledgements and next relevant contributions in order to arrive at a overview of what each CGU was about. Then I could start grouping them together into a hierarchy and describe what each group was about. The actual annotation (not including preparation) took about 45 minutes for Trains and about 30 minutes for Toot. For annotating the Trains dialog with IU's, the user doesn't establish a joint goal with the system to solve the overall goal. In fact, the system is never even told what the joint goal is. The result is that the intentional structure that the user has really isn't shared with the system. This is especially true during the parts of the dialog where the user is gathering information. The system doesn't know what purpose this is for. The discourse structure I give is from the standpoint of the user. Sometimes CGU include two separate speech acts that belong to different IU's. For instance, in Toot: the system gives the results from the database and then asks the system to modify the query. I coded this as one CGU, since it is the entire thing that the user grounds. General comments I think we will really need to examine the rules given in Section 4.4 and reconcile this with what the backward people are doing for coding understanding. I think we should also discuss the whole notion of keeping CGU's open. More examples of this in the coding scheme would be helpful as well. I can see that it is useful in MOO and in `unix talk'. But not clear how appropriate it is for spoken dialogue. We also need to talk about the implication of speech repairs. My feeling is that the CGU's work on a level above speech repairs. Hence, we should take the cleaned up speech (with repairs identified), rather than accounting for speech repairs in the CGU annotation. David and I have debated this issue before. This would simplify the annotation of CGU's. Doing the discourse structure of a toot dialogue might not be very illuminating. The discourse structure is of course going to be dictated by its finite state dialogue model, no matter what the user tries to do. :::::::::::::: misc.jeanc :::::::::::::: I've put some "left out intentionally" markers, but forgot to mark every intentional omission. I don't like the CGU analysis of questions in the manual. It seems odd to me to have two CGUs, one grounding the question and one grounding the answer if the answer is at all complex. I don't understand when we're to do the same for other back-and-forth exchanges (like negotiating about suggestions or even saying "I didn't hear you"). The manual itself is too baroque to really be a manual. It's especially a problem that IU coding can't be explained without recourse to lots of different background material --- are we to believe that no one who hasn't done lots of reading in this field would understand these distinctions? I think it would be a useful exercise to boil the instructions down to what you'd need to know to do it if you were a naive coder. I honestly think a sheet of A4 per scheme would do it and would much improve the schemes. The relationship to the rest of the literature shouldn't be definition for how to apply the scheme, but how to interpret it. With Trains, it was difficult to tell if the subject was making a discourse marker or acknowledging material. I omitted discourse markers since the manual said to. I left U.4.2 as a cancelled CGU because I think the wizard could take something away from it. U.43.1-5 may look similar, but the content is so garbled, with no prior content to hang it from, that I didn't include it in any CGU. U.43.5 should probably be split into two utternaces; it's a repair. "For an engine" sort of goes with U.43.6 except I think at this point he probably realised that one doesn't get separate boxcar and engine timings. Toot: I left out the closing because I think I heard her hang up. I was tempted to divide the system's complex retrieval responses (no trains/early/late) into multiple CGUs but I didn't because the system itself tried to ground them together (by asking if user wanted a repetition) and the user seemed to understand that. Timings: an hour each, CGU analysis; one hour TOOT IU (done first), half hour Trains IU, all done on paper. It took probably 15 minutes just to type in the CGU analysis for each file and 5 minutes the IU analysis. I thought the typing took too long. :::::::::::::: misc.jencc :::::::::::::: a. Utterance tokenization: Trains: I think U.21.2 should be split into two utterance tokens, with the break being where the repair occurred. b. Time: I didn't do everything in one sitting, so the times are rough estimates. CGU analysis: Trains: 1.5-2 hours Toot: 45 minutes IU analysis: Trains: 1 hour Toot: 30 minutes c. Log: Trains: CGU analysis: * I wasn't sure what to do with CGU #4. U.4.2 proposes something that is continued in U.13.2 (CGU #13). However, the current coding scheme prevents me from adding U.13.2 to CGU #4, even though they contribute to the same action. I'm not sure I like the rule in question here... * In CGU #8, I wasn't sure if U.9.1 should be included. I included it here because it grounds the response in S.7.1,S.7.2, and U.8.1. However, the response is later canceled. On the other hand, U.11.1 grounds the revised answer, but it is an answer to a completely unrelated question and doesn't even follow (in sequence) the utterances being grounded (therefore I'm also unsure about including it in this CGU). * CGU #9 is somewhat of a funny one. U.8.1 and S.7.2 completely overlap each other, and I can't tell whether S.7.2 is given as a response to U.8.1 or if S just realized the ambiguity in S.7.1 and decided to clarify it. I coded it according to the former interpretation, in which case U.8.1 is a confirmation question and S.7.2 is its answer. * I have the same problem with CGU #32 as with #4 where something is started but continued later (in #37). IU analysis: * I'm not sure how much look-ahead I'm supposed to do. For instance, U.2.1 doesn't really state the problem, but directly jumps in to obtain information for solving a specific problem that U has in mind. If I were to reconstruct the whole process after reading the entire dialogue (which is what I did), I know that U is trying to ship two boxcars of oranges to Dansville at that point. However, there is no way that anybody just reading the dialogue for the first time up to that point will be able to figure that out. * I'm having some trouble distinguishing between canceled CGU's and canceled IU's. The guideline I'm using here in coding is that if a proposition is acknowledged, then it is entered into the common ground whether or not it's modified later on in the dialogue. Thus CGU #9 is canceled at the IU level, but not at the CGU level. Toot: CGU analysis: * I had some problems with CGUs #10 and #11 because in turn 11 the system really tried to achieve two goals: to provide the information the user requested and to ask what the user wants to do next. Neither of these is grounded until the user takes over the turn in U.12.1. Maybe there isn't a problem with this, but it's one of the cases that's not as straightforward as the others. * I'm also unsure about places where the speech recognizer fails to understand the utterances. I coded them as canceled CGU's by themselves, since they were never grounded. IU analysis: * I had more trouble doing the IU analysis for the TOOT dialogue than for the TRAINS dialogue. I think the main problem is that the TOOT problem is "flatter" and the same process repeated three times. At first I came up with a completely flat iu list (one action after another, nothing embedded), then I changed it to group some actions under one heading and allowed more embedding. I find the plan structure more natural in the TRAINS dialogue.:::::::::::::: misc.owen :::::::::::::: General Question about CGUs --------------------------- Why are CGUs tied to content? they could be tied to communicative intentions as well. For example: S: Can I help you? H: Yes. Common ground established that S is waiting for H's request or question. After all, part of what RST (and G&S?) is all about is that the H must understand not just the content, but the intention of S, and presumably "understand" means "have in CG". Well, in fact you already make a distinction between questions and answers, and questions do not really have content separate from the answers. This is unclear to me. So I guess I am not entirely clear about the definition of a CGU. Comments on first dialog ------------------------ Tokenizations I would do differently: Trains 4.3/4.4 should be one, 20.1/20.2 should be one, 18.1 should be two, 43.5 is surely two tokens, it is neither a syntactic nor an intonational phrase! Same with 21.2. The description in the manual already seems a bit vague. What would make most sense to me is to use discourse purpose as determining criterion, which you specifically want to exclude. Comments on second dialog ------------------------- I don't think the dialog with the ATT system is really suited for debugging a coding scheme. The whole problem with that system is that common ground does not work, it is not really using natural language-style interaction at all -- while it may be interesting to use a coding scheme to figure out exactly what is going wrong, I think that may be a bit premature. (Presumably what is going on is that the poor user of the system understands and models the system's shortcomings.) For example, the interaction starting with 16.1 is just plain weird. 16.2 never enters the common ground, but one cannot really say that 16.1 enters the CG either, since it is wrong. While this is a puzzle, it is really due to the artificial agent -- normally, U would make sure that the right info entered the CG and the analysis would reflect this. But U cannot get a word in edgewise. Similarly for 26.1, which contrary to expectations has not entered CG; normally, this would be repaired as soon as detected, but because of the system, it cannot really be repaired at all. In a sense, this conversation is pathological in an uninteresting way. Odd tokenizations: 17.1, 17.2 (should be one) and related ones Missing bit of in transcription (before S.45.1). :::::::::::::: misc.traum :::::::::::::: trains: CGU analysis: approx 1.2 hours 4.2 is abandoned, but the material is re-introduced in 11.1, which is putatively a reformulation of 9.1. The actual suggestion restarts at 13.2. A few places it seems like an utterance is too long - with an embedded boundary tone. The place where this is most significant is at "the" (7th word) in U.21.2 (perhaps this is just a long pause which creates this impression, but perhaps that should be significant utterance boundary). First part is abandoned, later part is question answered. 21.2 Also, perhaps for 47.1 (no pause, but prosodically seems prominent), first part is strictly ack of CGU34, rest is just beginning of new CGU36) 43.1 - very significant from IU point of view - transition to second subplan/topic switch marker, but ignored at the grounding level - not really acknowledging anything or acknowledged in own right. It's an align move according to HCRC taxonomy. Not sure how to count this. Similarly 41.1 (especially prosody) signals that the next step will be the last in this plan segment. Also same for first part of 60.1 CGU35 - this is abandoned *after* a correction which could be seen as grounding the content of the suggestion. Instead, 48.2 is seen as a repair of the suggestion rather than acknowledgment and initiation. "and" at the beginning of a followup question can be seen as acknowledging the answer or just coherence with previous question. CGU39 this is assumed grounded, but no real evidence from 54.1 (except "next relevant contribution" which is hard to judge). Real evidence perhaps comes from 54.3 where the time limit must be seen as acceptable. Notes on IU analysis: 35 min. 1 - top level plan is never stated, but available from http://www.cs.rochester.edu/research/speech/93dialogs/problems.html#2-G -to the user at least - I'm not sure if the system knew the goal. CGU4 is problematic for stack-based IU analysis coupled with my analysis - this prematurely starts the suggest phase, but then is cancelled and the preconditions phase is restarted. though perhaps that is a theme rather than phase? Also, aborted suggestion inside 21.2. - not sure I have a good justification for grouping iu.1.1.3.5 as separate from the rest of iu.1.1.3, except it's part of the same checking sequence. iu.1.1.2 is not strictly cancelled, but becomes irrelevant with the fuller, more correct plan which also includes the engine moving and coupling in iu.1.1.4 46 is really a "commit phase" of the whole big thing. I guess really I should have one more level in between 1 and 1.1,1.2 - call it 1.a, the planning phase for the whole problem, with two subplans 1.a.1 = 1.1, and 1.a.2 =1.2, then 1.b would be the commit phase, including 46, but this seems a bit overkill. TOOT CGU about 1 hour 6.2 would be different if from system point of view - would initiate a new CGU with 7.1, whereas from user point of view it looks like a repair. undecided whether to make 10.1 initiate a new CGU with the acceptance (acknowledge with 11.1). Similar with 22.1,23.1 - if seen as an accept, then probably not, if seen as ynq, probably - borderline case, also 32.1, 34.1 CGU10 is assumed grounded by user waiting as instructed. Likewise for the missing part of 45.1, and CGU51. -some problems with toklabs file - utterances within turn over 10 on turn 11 were truncated to S.11.1 - 17.1 and 17.2 both seen as acks of 15 - first one implicit by asking followup question, second confirming the understood info (wrongly in this case). -27.1 seen as implicit acknowledge - in this case full confirmation would have been better,... before 45.1 there is another "query warning, and a snort after that. 53.8,53.9 should be merged according to intonation (actually this makes the utterance sound un-natural -speech synthesis should have been given a boundary marker here. Questions about whether to split cgu52 at 53.8 and consider first part implicitly grounded - similar question for cgu11 at 11.9 and cgu43 at 45.6 IU analysis: about 20 min -not sure how deep to go with queries and search. Also, a bit arbitrary to combine date and time - works well for 1.1.1, but not so well in 1.2.1, 1.3.1 where these could easily be two separate sub-segments - not sure whether to keep as one for consistency or separate based on exchange structure. Likewise, could easily have decided thematically to group source and dest together. -cgu52 is "too big" for this style analysis - first part up to 53.8 is part of results phase of 1.3.5, second part is another "offer new search like 19 and 31, which belongs at top level (in an adjacency part with 54). :::::::::::::: misc.venditti :::::::::::::: TIME: TRAINS ============ CGU labs -- 2:15hrs IU labs -- 1:00hrs NOTES: TRAINS ============= CGU 11 (S.10.12, S.10.3) is not acknowledged by other person TIME: TOOT ============ CGU labs -- 1:15hrs IU labs -- 0:30hrs NOTES: TOOT =========== CGU 1 (S.1.1, S.1.2, S.1.3, U.2.1) -- first two phrases are introduction and belong to highest iu.1, while second two phrases belong to iu.1.1. that is, one CGU needs to be split into two. sorry i don't have many notes. hopefully i'll have more for the next HMK2. :::::::::::::: misc.walker :::::::::::::: Comments: I did TOOT before trains. When doing the CGU coding for TOOT, it seemed unnatural to me to have to ground that the user understood the system's question, in a separate unit from grounding the answer to the question. I had to constantly watch myself that I followed the coding manual in this respect. Thus it would have been more natural to me if none of the requests-info speech acts were pulled out as separate grounding units and these CGUs would have been part of the following CGU that specifies the context that was actually grounded, e.g. "departure city is Washington D.C." For TOOT, it was also difficult to decide how to label the final confirmation utterances by the system when it asks "do you want me to find the trains... now?' The 'now' gave me an out of saying that the system was signalling that it was ready to query the web. But the system could simply have said "do you want me to find the trains ...?", in which case the question is completely redundant with what has gone before and therefore grounds no content. Its role is obviously as a confirmation of what the user wants before incurring the cost of the query. TOOT uses implicit confirmation, but if it used a no-confirmation strategy then it would have seemed natural to have said that S.51.1 would be sufficient to have grounded U.50.1. So the grounding criterion seems to be sensitive to the discourse strategies that are being used in the conversation. For TOOT, it took me about an hour to do the CGU analysis and about 20 minutes to do the IU analysis. For TRAINS, it took me about 55 minutes to do the CGU analysis and about 10 minutes to do the IU analysis. In TRAINS, I would have liked to split out the initial Okay in U.47.1:::::::::::::: misc.ward :::::::::::::: After starting the IU file for toot, I find that I wanted to go back and change some of my decisions on the corresponding CGU file. For example, I had originally coded "Do you want to continue and find a new set of trains?" (S.53.9) to be part of the CGU where the agent informs the customer of train availability. Now, I would code it as a distinct CGU. Another case where I regret not splitting up CGUs is in the trains corpus; I ad coded U.43.1-6 as part of a single CGU; now I would break it up into U.43-1-5 and U.43.6.