SMASH at StanceEval 2024: Prompt engineering LLMs for Arabic stance detection

Youssef Al Hariri; Ibrahim Abu Farha

SMASH at StanceEval 2024: Prompt engineering LLMs for Arabic stance detection

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract / Description of output

This paper presents our submission for the Stance Detection in Arabic Language (StanceEval) 2024 shared task conducted by Team SMASH of the University of Edinburgh. We evaluated the performance of various BERT-based and large language models (LLMs). MARBERT demonstrates superior performance among the BERT-based models, achieving F1 and macro-F1 scores of 0.570 and 0.770, respectively. In contrast, Command R model outperforms all models with the highest overall F1 score of 0.661 and macro F1 score of 0.820.

Original language	English
Title of host publication	Proceedings of The Second Arabic Natural Language Processing Conference
Publisher	Association for Computational Linguistics (ACL)
Pages	800-806
Number of pages	7
ISBN (Electronic)	9798891761322
Publication status	Published - 16 Aug 2024
Event	The Second Arabic Natural Language Processing Conference - Hybrid, Bangkok, Thailand Duration: 16 Aug 2024 → 16 Aug 2024 Conference number: 2 https://arabicnlp2024.sigarab.org/

Conference

Conference	The Second Arabic Natural Language Processing Conference
Abbreviated title	ArabicNLP 2024
Country/Territory	Thailand
City	Bangkok
Period	16/08/24 → 16/08/24
Internet address	https://arabicnlp2024.sigarab.org/

Access to Document

HaririFarhaArabicNLP2024SMASHAtStanceEval2024Final published version, 229 KBLicence: Creative Commons: Attribution (CC-BY)

https://aclanthology.org/2024.arabicnlp-1.92Licence: Creative Commons: Attribution (CC-BY)

Cite this

@inproceedings{f63eef47c3d04d9b8cec90818f01cea3,

title = "SMASH at StanceEval 2024: Prompt engineering LLMs for Arabic stance detection",

abstract = "This paper presents our submission for the Stance Detection in Arabic Language (StanceEval) 2024 shared task conducted by Team SMASH of the University of Edinburgh. We evaluated the performance of various BERT-based and large language models (LLMs). MARBERT demonstrates superior performance among the BERT-based models, achieving F1 and macro-F1 scores of 0.570 and 0.770, respectively. In contrast, Command R model outperforms all models with the highest overall F1 score of 0.661 and macro F1 score of 0.820.",

author = "{Al Hariri}, Youssef and {Abu Farha}, Ibrahim",

year = "2024",

month = aug,

day = "16",

language = "English",

pages = "800--806",

booktitle = "Proceedings of The Second Arabic Natural Language Processing Conference",

publisher = "Association for Computational Linguistics (ACL)",

note = "The Second Arabic Natural Language Processing Conference, ArabicNLP 2024 ; Conference date: 16-08-2024 Through 16-08-2024",

url = "https://arabicnlp2024.sigarab.org/",

}

Al Hariri, Y & Abu Farha, I 2024, SMASH at StanceEval 2024: Prompt engineering LLMs for Arabic stance detection. in Proceedings of The Second Arabic Natural Language Processing Conference. Association for Computational Linguistics (ACL), pp. 800-806, The Second Arabic Natural Language Processing Conference, Bangkok, Thailand, 16/08/24. <https://aclanthology.org/2024.arabicnlp-1.92>

TY - GEN

T1 - SMASH at StanceEval 2024

T2 - The Second Arabic Natural Language Processing Conference

AU - Al Hariri, Youssef

AU - Abu Farha, Ibrahim

N1 - Conference code: 2

PY - 2024/8/16

Y1 - 2024/8/16

N2 - This paper presents our submission for the Stance Detection in Arabic Language (StanceEval) 2024 shared task conducted by Team SMASH of the University of Edinburgh. We evaluated the performance of various BERT-based and large language models (LLMs). MARBERT demonstrates superior performance among the BERT-based models, achieving F1 and macro-F1 scores of 0.570 and 0.770, respectively. In contrast, Command R model outperforms all models with the highest overall F1 score of 0.661 and macro F1 score of 0.820.

AB - This paper presents our submission for the Stance Detection in Arabic Language (StanceEval) 2024 shared task conducted by Team SMASH of the University of Edinburgh. We evaluated the performance of various BERT-based and large language models (LLMs). MARBERT demonstrates superior performance among the BERT-based models, achieving F1 and macro-F1 scores of 0.570 and 0.770, respectively. In contrast, Command R model outperforms all models with the highest overall F1 score of 0.661 and macro F1 score of 0.820.

M3 - Conference contribution

SP - 800

EP - 806

BT - Proceedings of The Second Arabic Natural Language Processing Conference

PB - Association for Computational Linguistics (ACL)

Y2 - 16 August 2024 through 16 August 2024

ER -

SMASH at StanceEval 2024: Prompt engineering LLMs for Arabic stance detection

Abstract / Description of output

Conference

Access to Document

Fingerprint

Cite this