The 4th Workshop on Evaluation and Comparison for NLP systems (Eval4NLP), co-located at the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics (IJCNLP-AACL 2023), invites the submission of long and short papers, with a theoretical or experimental nature, describing recent advances in system evaluation and comparison in NLP.
All deadlines are 11.59 pm UTC -12h (“Anywhere on Earth”).
This year’s version will come with a shared task on explainable evaluation of generated language (MT and summarization) with a focus on LLM prompts. Please find more information on the shared task page.
The workshop welcomes two types of submission -- long and short papers. Long papers may consist of up to 8 pages of content, plus unlimited pages of references. Short papers may consist of up to 4 pages of content, plus unlimited pages of references. Please follow the ACL ARR formatting requirements, using the official templates. Final versions of both submission types will be given one additional page of content for addressing reviewers’ comments. The accepted papers will appear in the workshop proceedings.
The review process is double-blind. Therefore, no author information should be included in the papers. Self-references that reveal the author's identity must be avoided. Papers that do not conform to these requirements will be rejected without review.
Eval4NLP features two modes of submissions. Standard submissions: We invite the submission of papers that will receive up to three double-blind reviews from the Eval4NLP committee, and a final verdict from the workshop chairs. Pre-reviewed: To a later deadline, we invite unpublished papers that have already been reviewed, either through ACL ARR, or recent AACL/EACL/ACL/EMNLP/COLING venues (these papers will not receive new reviews but will be judged together with their reviews via a meta-review; authors are invited to attach a note with comments on the reviews and describe possible revisions).
Final verdicts will be either accept, reject, or conditional accept, i.e., the paper is only accepted provided that specific (meta-)reviewer requirements have been met. Please also note the multiple submission policy.
The submisson site (OpenReview) for direct submissions is available here. Further, the submisson site (OpenReview) for pre-reviewd submissions is available here.
Authors are allowed to submit (optional) supplementary materials (e.g., appendices, software, and data) to improve the reproducibility of results and/or to provide additional information that does not fit in the paper. All of the supplementary materials must be zipped into one single file (.tgz or .zip) and submitted via OpenReview together with the paper. However, because supplementary materials are completely optional, reviewers may or may not review or even download them. So, the submitted paper should be fully self-contained.
Papers uploaded to preprint servers (e.g., ArXiv) can be submitted to the workshop. There is no deadline concerning when the papers were made publicly available. However, the version submitted to Eval4NLP must be anonymized, and we ask the authors not to update the preprints or advertise them on social media while they are under review at Eval4NLP.
This year, our workshop also welcomes submissions that have received 3 reviews from ACL Rolling Review (ARR) or another venue (e.g., IJCNLP-AACL, COLING, EMNLP, etc.). Authors of any papers that have 3 reviews ready by TBD may submit their papers and reviews to our workshop TBD (this is a different link than the direct submission link). Please note the multiple submission policy.
Eval4NLP allows authors to submit a paper that is under review in another venue (journal, conference, or workshop) or to be submitted elsewhere during the Eval4NLP review period. However, the authors need to withdraw the paper from all other venues if they get accepted and want to publish in Eval4NLP.
Note that the IJCNLP-AACL and ARR do not allow double submissions. So, papers submitted both to the main conference and IJCNLP-AACL workshops (including us) will violate the multiple submission policy of the main conference. If authors would like to submit a paper under review by IJCNLP-AACL to the Eval4NLP workshop, they need to withdraw their paper from IJCNLP-AACL and submit it to our workshop before the workshop submission deadline.
We will award three prizes to the best three paper submissions (subject to availability; more details to come soon). Both long and short submissions will be eligible for prizes.
If you want to present a paper which has been published recently elsewhere (such as other top-tier AI conferences) at our workshop, you may send the details of your paper (Paper title, authors, publication venue, abstract, and a link to download the paper) directly to eval4nlp@gmail.com. We will select a few high-quality and relevant papers to present at Eval4NLP. This allows such papers to gain more visibility from the workshop audience and increases the variety of the workshop program.
The submission deadline for this track is TBD and the notification of acceptance will be sent on TBD. The submitted papers will be judged separately from the main track (submitted via softconf). Note that the chosen papers are considered as non-archival here and will not be included in the workshop proceedings.