DIALOG-22 RuATD: Generated Text Detection

Background

The rapid advancement of neural text generation models has created an urgent need for reliable methods to distinguish human-written text from machine-generated content. While much of the early research on AI text detection focused on English, the problem is equally pressing for other languages. Russian, with its rich morphology, flexible word order, and complex case system, presents unique challenges for both text generation and detection. Models trained to detect AI-generated English text do not transfer straightforwardly to Russian, making language-specific research essential.

The DIALOG-22 conference organized the RuATD (Russian Artificial Text Detection) shared task to address this gap. The shared task provided a standardized benchmark and evaluation framework, inviting research teams to develop systems that could reliably identify whether a given Russian text passage was written by a human or produced by a generative model. This kind of community-driven evaluation is critical for advancing the state of the art, as it allows fair comparison of diverse approaches on the same data under the same conditions.

Beyond academic interest, the practical implications are significant. Russian-language misinformation campaigns, automated content farms, and academic dishonesty all rely on increasingly sophisticated text generation. Detection tools tailored to Russian are needed by content moderators, educators, and platform operators who cannot rely solely on English-centric solutions.

Methodology

Our approach explored both transformer-based and classical machine learning pipelines. On the transformer side, we fine-tuned pretrained language models that had been trained on Russian corpora, adapting them to the binary classification task of distinguishing human from machine text. These models capture deep contextual patterns in language use -- subtle differences in coherence, lexical choice, and syntactic structure that statistical features alone may miss.

In parallel, we developed a feature engineering pipeline that extracted linguistic and statistical signals from the text. These features included measures of lexical diversity, sentence length distributions, punctuation patterns, and morphological characteristics specific to Russian. Classical classifiers such as gradient boosting were trained on these feature sets, providing interpretable baselines and complementary signals to the neural approaches.

We experimented with ensemble strategies that combined the strengths of both paradigms. The RuATD dataset included texts generated by several different models, which meant our classifiers needed to generalize across generation methods rather than overfitting to the artifacts of a single generator. We used stratified cross-validation during development to ensure robust performance estimates before submitting to the shared task leaderboard.

Key Contributions

• Russian-specific detection: Methods tailored for Russian language AI text detection, accounting for morphological complexity and flexible syntax that distinguish Russian from English-centric approaches
• Feature engineering: Linguistic and statistical features for classification, including lexical diversity metrics, morphological tag distributions, and punctuation patterns characteristic of human versus machine writing in Russian
• Model comparison: Systematic evaluation of transformer-based and classical approaches, showing how pretrained language models and handcrafted features each capture different aspects of the human-machine distinction
• Benchmark results: Competitive performance on the RuATD dataset, contributing to the community's understanding of which detection strategies are most effective for Russian text

Results

Our system achieved competitive results on the RuATD shared task leaderboard. The transformer-based models generally outperformed classical approaches in isolation, consistent with the broader trend in NLP where pretrained models excel at capturing nuanced linguistic patterns. However, the classical feature-based models provided useful complementary signals, particularly for shorter texts where contextual models had less material to work with.

The ensemble approach, combining neural predictions with feature-based classifiers, yielded the most robust performance across different text types in the dataset. We observed that certain generators were easier to detect than others -- texts produced by older or less sophisticated models exhibited more detectable artifacts, while outputs from larger, more recent models were harder to distinguish from human writing.

Error analysis revealed that the most challenging cases involved texts on factual or formulaic topics, where both humans and machines tend to produce similar patterns. Conversely, creative or opinion-rich texts showed more distinctive human signatures that our models could leverage for classification.

Discussion

This work was conducted before the widespread release of ChatGPT and the subsequent explosion of interest in AI text detection. In retrospect, the RuATD shared task was prescient in identifying the need for robust, language-specific detection methods. The approaches we developed -- combining transformer fine-tuning with interpretable feature engineering -- remain relevant strategies, though the detection problem has grown substantially harder as generation models have improved.

With the proliferation of ChatGPT and other LLMs, detecting AI-generated text has become crucial for academic integrity, content moderation, and misinformation prevention. This early work on Russian text detection established important baselines for the field. The key lesson from this research is that detection methods must be continuously updated as generators evolve, and that language-specific evaluation benchmarks like RuATD are indispensable for ensuring detection tools work beyond English.

Related Topics

Low-Resource Text Classification · LLM Security · Trojan Detection