Acceptability Judgements via Examining the Topology of Attention Maps - Université Paris Cité Accéder directement au contenu
Communication Dans Un Congrès Année : 2022

Acceptability Judgements via Examining the Topology of Attention Maps

Résumé

The role of the attention mechanism in encoding linguistic knowledge has received special interest in NLP. However, the attention heads' ability to judge the grammatical acceptability of a sentence has been underexplored. This paper approaches the paradigm of acceptability judgments with topological data analysis (TDA), showing that the topological properties of the attention graph can be efficiently exploited for two standard practices in linguistics: binary judgments and linguistic minimal pairs. Topological features enhance the BERTbased acceptability classifier scores by up to 0.24 Matthew's correlation coefficient score on COLA in three languages (English, Italian, and Swedish). By revealing the topological discrepancy between attention graphs of minimal pairs, we achieve the human-level performance on the BLIMP benchmark, outperforming nine statistical and Transformer LM baselines. At the same time, TDA provides the foundation for analyzing the linguistic functions of attention heads and interpreting the correspondence between the graph features and grammatical phenomena. We publicly release the code and other materials used in the experiments 1. * Equal contribution. 1 github.com/danchern97/tda4la
Fichier principal
Vignette du fichier
2022.findings-emnlp.7.pdf (953.16 Ko) Télécharger le fichier
Origine : Fichiers produits par l'(les) auteur(s)
Licence : CC BY - Paternité

Dates et versions

hal-03995313 , version 1 (01-03-2023)

Identifiants

Citer

Daniil Cherniavskii, Eduard Tulchinskii, Vladislav Mikhailov, Irina Proskurina, Laida Kushnareva, et al.. Acceptability Judgements via Examining the Topology of Attention Maps. EMNLP 2022 Empirical Methods in Natural Language Processing, ACL Association for Computational Linguistics, Dec 2022, Abu-Dhabi, United Arab Emirates. pp.88-107, ⟨10.18653/v1/2022.findings-emnlp.7⟩. ⟨hal-03995313⟩
27 Consultations
16 Téléchargements

Altmetric

Partager

Gmail Facebook X LinkedIn More