SciIG · Introduction Generation
EMNLP 2026 · Submission

Towards AI-Assisted Research Writing:
Benchmarking LLMs for AI/ML Introduction Generation

A large-scale benchmark for Scientific Introduction Generation (SciIG) — generating coherent, citation-grounded introductions from a paper's title, abstract, and related work.

Krishna Garg1*   Firoz Shaik2*   Sambaran Bandyopadhyay3   Cornelia Caragea2
1University of Oxford  ·  2University of Illinois Chicago  ·  3Adobe Research, India
* Equal contribution. Work performed while at University of Illinois Chicago.
Paper (PDF) arXiv Code Datasets

Links will be finalized on release · code released under MIT, datasets under CC BY 4.0.

§Abstract

As researchers increasingly adopt LLMs as writing assistants, generating high-quality research paper introductions remains both challenging and essential. We benchmark Scientific Introduction Generation (SciIG), evaluating how well LLMs produce coherent introductions from titles, abstracts, and related works in the Machine Learning domain. Curating new datasets from NAACL 2025 and ICLR 2025 papers, we assess five state-of-the-art models — open-source (DeepSeek-V3, Gemma-3-12B, LLaMA-4 Maverick, Mistral-Small-3.1) and closed-source (GPT-4o) — across lexical overlap, semantic similarity, content coverage, faithfulness, consistency, citation correctness, and narrative quality. Our framework combines automated metrics with LLM-as-a-judge evaluation. LLaMA-4 Maverick shows the strongest overall performance, particularly on lexical overlap, content coverage, citation quality and perplexity; three-shot prompting outperforms fewer-shot approaches more than 50% of the time on formula-based metrics and consistently on human-evaluation metrics. These findings offer practical insight into building research-writing assistants and set realistic expectations for LLM-assisted academic writing.

3,900Benchmark samples
5SOTA models
7Prompting strategies
30Evaluation metrics

◇Contributions

SciIG establishes a common experimental and evaluation foundation for a task that has, until now, been under-benchmarked in the LLM era.

01 · Benchmark

First large-scale study of introduction generation

A systematic benchmark for research-paper introduction generation in the LLM era, with a shared experimental and evaluation foundation.

02 · Evaluation suite

30 metrics across three families

Formula-based automatic metrics, LLM-as-a-judge metrics, and human evaluation — capturing both surface-level and higher-order qualities like faithfulness, consistency, and writing quality.

03 · Data pipeline

End-to-end dataset construction

PDF text extraction, LLM-assisted citation identification, and structured retrieval of cited titles and abstracts via the Semantic Scholar API — yielding 3,900 samples with a 94.0% citation-resolution rate.

04 · Prompting study

Few-shot & instruction-level analysis

A systematic exploration of prompting strategies, complemented by a human-evaluation study on a subset of samples, giving practical guidance for LLM-assisted scientific writing.

▤Dataset construction pipeline

Papers from NAACL'25 and ICLR'25 are retrieved and parsed (Phase I), their citations extracted and mapped to reference titles (Phase II), and each reference enriched with Semantic Scholar metadata (Phase III). Each sample pairs (Title, Abstract, Related Work) with the published introduction as target.

SciIG dataset construction pipeline: five steps across Phase I acquisition and parsing, Phase II citation processing, and Phase III metadata enrichment, producing 3,900 samples.
Figure 1 · The five-step construction pipeline. In-text “Author et al., Year” citations are mapped to reference titles by LLaMA-4 Maverick, then enriched with abstracts and author lists via the Semantic Scholar API.
800NAACL 2025 samples
3,100ICLR 2025 samples
597–771Avg. intro words
13.4–15.3Avg. citations / paper

▥Main resultsElaborate prompting

Five models on NAACL 2025 (800 samples) with the Elaborate strategy. LLaMA-4 Maverick emerges as the overall front-runner, leading on lexical overlap, content coverage, citation quality, and perplexity. Best value per row is highlighted. Lower perplexity is better.

Full per-metric definitions, formulae, and the ICLR 2025 results appear in the paper. Values shown are for the NAACL 2025 set under the Elaborate prompt.

✎Annotated sample

Generations for one NAACL 2025 paper — "Zero-Shot Keyphrase Generation: Investigating Specialized Instructions and Multi-Sample Aggregation on Large Language Models" — with the ground truth and three LLaMA-4 configurations plus GPT-4o.

▤Prompting strategies

The seven prompts used to elicit introductions. Runtime values appear as {placeholders}: {title} and {abstract} are the target paper's; {cited_nodes_pretty} is the JSON-formatted related-papers block with APA citations. Few-shot prompts prepend 1–3 held-out examples to the Elaborate prompt.

"Citation

If you find SciIG useful, please consider citing our work. (BibTeX will be finalized on publication.)

@inproceedings{garg2026sciig, title = {Towards AI-Assisted Research Writing: Benchmarking LLMs for AI/ML Introduction Generation}, author = {Garg, Krishna and Shaik, Firoz and Bandyopadhyay, Sambaran and Caragea, Cornelia}, booktitle = {Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing (EMNLP)}, year = {2026}, note = {To appear} }