Thank you for this, I have been avoiding LLMs since I have no use for them, but it is good to have a look at what is going on. But I have some reservations.
"In most tasks, humans performed better than GPT-3. The two tasks that were the exception to this rule were the Short Stories Task (ShS) and the Strange Stories Task (StS)"
- but actually the graphs suggest that GPT-3 was just below Human level for Strange Stories, apparently within the 95% limits, but not actually better : though indeed well above for Short Stories.
I would add that both from a literary criticism point of view and a linguistician's* understanding, both of these present problems given the paucity of information about the actual question. The Short Stories task appears particularly confused.
Reading the story leaves me again thinking that Hemingway is a slapdash, or perhaps heavy-handed writer, quite apart from my general lack of sympathy with his discourse topics. But that is not the issue here. I was concerned that Trott and Bergen (2018) and then Jones et al (2023) may have accepted too readily the work of Dodell-Feder et al (2013) - unfortunately I do not have access to the latter's supporting information detailing the actual questions: it is referred to but I cannot see a link to it; while Trott & Bergen's piece would cost me 45€ - pah. So what follows is mostly a naïve reaction to your essay, I'm afraid.
Overall the tale is unexceptional, discussing falling out of love - which can happen in a moment, like falling in love - and how to convey this with being too hurtful. And the contradictory state of mind of Nick is what leads to the self-contradictory remark, "I've taught you everything. You know you do. What don't you know, anyway?"
But "You know everything" is indeed essentially saying that Marjorie is a smarty-pants, a know-all ("know-it-all" is not in my lexicon). The suggestion that it is said to provoke a quarrel is a second order assessment and is not necessarily the right answer - Nick is trying to avoid emotion: so provoking a quarrel, while one way of ending the relationship, is not what he has set out to do. The remark is more likely to be an involuntary expression of frustration that Marjorie is too docile rather than a deliberate tactic.
Accordingly both answers could, in the abstract, be correct: but the one you choose as correct - that he is being sarcastic to provoke a fight - seems to me to be essentially wrong, in terms of the story. Would that change one's view of what LLMs can do ?
I should note that in Jones et al (2023), the question is different from your presentation, as a free text response was accepted. There the options include provoking a fight or unhappy and nervous and ignores the consequences or he is just nasty. The middle one scores 1 - neither correct, 2 nor wrong, 0 - which I still think is wrongly characterizing the situation. And the visual representation of the figures seems to include averages across many questions, not just the ones specific to theory of mind. I remain confused.
Later you say: " It’s interesting (though perhaps not surprising) that GPT-3 was most likely to out-perform humans on tasks that required these more elaborate explanations, as opposed to tasks often viewed as more “rote”, such as multiple choice." Personally I would a priori assume humans were better at attributing motive and so forth: but my lack of experience in this field is evident from my bafflement at this remark, among others, in Jones et al, " If language exposure is sufficient for human ToM, then the statistical information learned by LLMs could account for variability in human responses" which seems to be implying a cybernetic effect of LLMs on humans, just as their question 3, "Do LLMs fully explain human behaviour..." needs a lot of unpacking.
I would add that the "wrong" response to the Strange Story question seems to me perfectly acceptable, albeit as a second-order assessment. And your reference to a violin plot leaves me wondering which shape of violin you are using - I simply do not understand this.
Given the amount I would have to learn, together with what seem to me detailed confusion, I think I am probably right to stay away from this research, and I will not even try to join in the argument whether Theory of Mind exists or for what or whom. No thanks are necessary but again, thank you for this brief glimpse.
*someone who studies linguistics should be a "linguistician", as "linguist" is already taken: indeed most linguisticians are linguists: but many of them are far too slapdash with words.
Thank you for this, I have been avoiding LLMs since I have no use for them, but it is good to have a look at what is going on. But I have some reservations.
"In most tasks, humans performed better than GPT-3. The two tasks that were the exception to this rule were the Short Stories Task (ShS) and the Strange Stories Task (StS)"
- but actually the graphs suggest that GPT-3 was just below Human level for Strange Stories, apparently within the 95% limits, but not actually better : though indeed well above for Short Stories.
I would add that both from a literary criticism point of view and a linguistician's* understanding, both of these present problems given the paucity of information about the actual question. The Short Stories task appears particularly confused.
Reading the story leaves me again thinking that Hemingway is a slapdash, or perhaps heavy-handed writer, quite apart from my general lack of sympathy with his discourse topics. But that is not the issue here. I was concerned that Trott and Bergen (2018) and then Jones et al (2023) may have accepted too readily the work of Dodell-Feder et al (2013) - unfortunately I do not have access to the latter's supporting information detailing the actual questions: it is referred to but I cannot see a link to it; while Trott & Bergen's piece would cost me 45€ - pah. So what follows is mostly a naïve reaction to your essay, I'm afraid.
Overall the tale is unexceptional, discussing falling out of love - which can happen in a moment, like falling in love - and how to convey this with being too hurtful. And the contradictory state of mind of Nick is what leads to the self-contradictory remark, "I've taught you everything. You know you do. What don't you know, anyway?"
But "You know everything" is indeed essentially saying that Marjorie is a smarty-pants, a know-all ("know-it-all" is not in my lexicon). The suggestion that it is said to provoke a quarrel is a second order assessment and is not necessarily the right answer - Nick is trying to avoid emotion: so provoking a quarrel, while one way of ending the relationship, is not what he has set out to do. The remark is more likely to be an involuntary expression of frustration that Marjorie is too docile rather than a deliberate tactic.
Accordingly both answers could, in the abstract, be correct: but the one you choose as correct - that he is being sarcastic to provoke a fight - seems to me to be essentially wrong, in terms of the story. Would that change one's view of what LLMs can do ?
I should note that in Jones et al (2023), the question is different from your presentation, as a free text response was accepted. There the options include provoking a fight or unhappy and nervous and ignores the consequences or he is just nasty. The middle one scores 1 - neither correct, 2 nor wrong, 0 - which I still think is wrongly characterizing the situation. And the visual representation of the figures seems to include averages across many questions, not just the ones specific to theory of mind. I remain confused.
Later you say: " It’s interesting (though perhaps not surprising) that GPT-3 was most likely to out-perform humans on tasks that required these more elaborate explanations, as opposed to tasks often viewed as more “rote”, such as multiple choice." Personally I would a priori assume humans were better at attributing motive and so forth: but my lack of experience in this field is evident from my bafflement at this remark, among others, in Jones et al, " If language exposure is sufficient for human ToM, then the statistical information learned by LLMs could account for variability in human responses" which seems to be implying a cybernetic effect of LLMs on humans, just as their question 3, "Do LLMs fully explain human behaviour..." needs a lot of unpacking.
I would add that the "wrong" response to the Strange Story question seems to me perfectly acceptable, albeit as a second-order assessment. And your reference to a violin plot leaves me wondering which shape of violin you are using - I simply do not understand this.
Given the amount I would have to learn, together with what seem to me detailed confusion, I think I am probably right to stay away from this research, and I will not even try to join in the argument whether Theory of Mind exists or for what or whom. No thanks are necessary but again, thank you for this brief glimpse.
*someone who studies linguistics should be a "linguistician", as "linguist" is already taken: indeed most linguisticians are linguists: but many of them are far too slapdash with words.