Generate turn type dynamics

[bvreede]

Adding utterance properties:

• Number of words in an utterance (nwords)
• Number of characters in an utterance (nchar)
• list of words in the utterance
• FTO; compound of (FTO)
  • Dyadic conversation in window; boolean -- needs to be True to return an FTO
  • FTO calculation
• Overlap properties moved to Overlap calculations #46

Closes #5
Closes #8
Closes #42

[mdingemanse]

Issue #8 is relevant here too — FTO (Floor Transfer Offset) is computed as part of the turn dynamics.

Since there's already some windowed computation done —which needs checking by the way, as that code is not our production code— the number of participants in the prior 10s window should be known, which also means we know it's dyadic or triadic.

[mdingemanse]

This is a first (untested) python port of my R 'windowed transitions' code. As it needs to run on every single row in the db (creating a 10s window and computing these summary measures for it) it's likely to be quite a drag on performance — optimization and modularization needed.

[mdingemanse]

[also just testing the review mechanics — don't mind me if I'm doing this wrong)

[sonarqubecloud]

SonarCloud Quality Gate failed.   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
37 Code Smells

0.0% Coverage
0.0% Duplication

[mdingemanse]

Here's some code that is based on trawling the diverse corpora.

First a bit I use to set utterance to NA:

    na_strings <- c("[unk_utterance]","[unk_noise]","[distortion]","[background]",
                    "[background] M","[static]","untranscribed",
                    "[noise]","[inintel]","[distorted]","tlyam kanəw")

Then some code I use to classify types of conduct other than talk (this should really not be hardcoded, ideally)

    # add talk / other conduct classification
    d <- d %>%
      mutate(nature = case_when(
        utterance == "[laugh]" ~ "laugh",
        utterance == "[breath]" ~ "breath",
        utterance %in% c("[cough]","[sneeze]","[nod]","[blow]","[sigh]",
                         "[yawn]","[sniff]","[clearsthroat]","[lipsmack]",
                         "[inhales]","[groan]") ~ utterance,
        is.na(utterance) ~ as.character(NA),
        TRUE ~ "talk"
      ))

Then some code I use to generate nchar

    # create stripped and squished version of utterance and add length measures (note that
    # "\\W+" doesn't play well with unicode, so we count spaces)
    d <- d %>%
      mutate(utterance_stripped = gsub("\\[+[^()]*\\]+?", "", trimws(utterance))) %>%
      mutate(utterance_stripped = gsub("[\\(\\)]+", "", utterance_stripped)) %>%
      mutate(utterance_stripped = ifelse(utterance_stripped == "",NA,str_squish(utterance_stripped))) %>%
      mutate(
           nwords = str_count(utterance_stripped, " ") + 1, # crudely count spaces
           nchar = str_count(utterance_stripped)            # NOTE that nchar = bad for ideographs
           ) 

[mdingemanse]

Also this reminds me that we have two more utterance-level column that are really quite useful, but that relate to the corpus (and therefore often the language) as a whole:

• rank : the rank of this utterance in the frequency distribution of utterances in this language
• n : the number of tokens of this utterance type in this language

E.g. below you can see that the n of î (third row) is 53: there are 53 utterances like this in the language. Its rank is 1: it is the most frequently attested turn format in the language.

Debatable whether this is truly an utterance feature, but we do use it for a lot of things, e.g. to pull up the most frequent utterance formats in a language, identify 'streaks' of similarly frequent utterances, etc.

[jiqicn]

Hi @bvreede, I have left mostly minor questions and remarks.

Let me know if you want to further discuss any of the comments.

[jiqicn]

A minor one: from a semantics and OOP perspective, I'm having a bit of a hard time imagining what/how a conversation can inherit from a writer. Is there any special reason that the base class is call writer?

[bvreede]

Thanks for this question, and I hope my answer makes sense, this is all quite new to me too!
Writer is not a proper parent class, but a class that is used to collect writer functionality that can be used by different objects. Is this weird architecture? Please let me know 😅 CC @carschno

[carschno]

I fully understand the conceptual doubts, as Writer really does not make sense as a parent class for a Conversation. However, the Writer here takes the role of a MixIn in Python terminology (similar to an Interface in other languages). Python does not make a technical distinction between a class serving as a parent class in the strict sense, and a class serving as a MixIn.
The distinction is typically visible by the order of inheritance, where the parent class is the first, followed by one or multiple MixIns (e.g. Conversation(AbstractConversation, Writer)). In this case, however, Conversation does not have a parent class in the common sense, but only a MixIn.

Specifically, the Writer class enables other classes to inherit and/or override common serialization methods -- for writing, in this case.

[jiqicn]

Nice explanation @bvreede and @carschno! I didn't really check the other modules/classes defined outside this PR, so mostly I missed the context there. But since you called it a MixIn, I'm expecting to either see some other similar classes provided as optional features to be inherited, or classes rather than Covnersation who will also use this particular feature (and maybe some other features). Otherwise, making Writer a MixIn may not make a lot of sense (at least to myself). But I also understand that this can be a future work.

[bvreede]

Thanks for the explanation @carschno! I didn't know about MixIns yet, but having a separate class that is used to provide some default methods here is logical to me. We indeed have another class that inherits from Writer: Corpus. There is some common functionality that we want to keep DRY as both Conversation and Corpus are objects that e.g. will be saved as json (#21) and csv (#47).

[mdingemanse]

@bvreede some belated responses to this

• you have access to that basecamp thread now
• I agree that your Scenario 1 is perfectly sensible, and that Scenarios 2 and 3 are increasingly more questionable. However, there is nothing we can do about that without making another arbitrary determination of when an utterance stops being "short". This is why I introduced prior_by (link to basecamp comment). Quoting from my Basecamp thread:

this allows us to decide whether FTO should only be done when prior is by "other" (most conservative), versus perhaps also when prior row in db is by "self during other" (also reasonable)

So my thinking at the time was that folks might want to choose between the most conversative measure versus a slightly looser one. And for ElPaCo we went with the looser one:

next we deal with self during other. The difference is hard to see but now turns whose prior is a self-during-other are also getting an FTO, timed relative to the nearest prior turn by other (instead of being treated as a self-transition)

[sonarqubecloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
1 Code Smell

93.3% Coverage
0.0% Duplication

review comments were used to provide clarifications