$\mu $-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts

T Koike-Akino, J Liu, Y Wang - arXiv preprint arXiv:2505.18451, 2025

To tackle the huge computational demand of large foundation models, activation-aware compression techniques without retraining have been introduced. However, since these rely on calibration data, domain shift may arise for unknown downstream tasks …


$\texttt {Droid} $: A Resource Suite for AI-Generated Code Detection

D Orel, I Paul, I Gurevych, P Nakov - arXiv preprint arXiv:2507.10583, 2025

In this work, we compile $\textbf{$\texttt{DroidCollection}$}$, the most extensive open data suite for training and evaluating machine-generated code detectors, comprising over a million code samples, seven programming languages, outputs …


'Nobody's Framework': Article 4 CDSM and the Broken Promise of TDM in the Age of AI

Generative Artificial Intelligence (GenAI) has introduced significant legal and regulatory challenges, particularly concerning the use of copyrighted content to train models through text and data mining (TDM). Within the European Union (EU), this …


(Mis) Fitting: A Survey of Scaling Laws

M Li, S Kudugunta, L Zettlemoyer - arXiv preprint arXiv:2502.18969, 2025

Modern foundation models rely heavily on using scaling laws to guide crucial training decisions. Researchers often extrapolate the optimal architecture and hyper parameters settings from smaller training runs by describing the relationship …


13 Libyan Translators' Attitudes toward the Profession in the Era of Automation

NASA Ali, M Babchikh - … Intelligence in Translation: Possibilities, Processes and …, 2025

Artificial intelligence (AI) has affected many aspects of human life and transformed various industries, including translation. Like many professionals, some translators are concerned about the future of their business and how AI will affect it. Others …


13 Machine Learning in Phishing URL Detection: A Review of Recent Progress

A Simhadri, M Rishikesh, M Subramaniam - Power Energy and Secure Smart …, 2025

In 2023, the Anti-Phishing Working Group, a prominent cybersecurity organization, reported five million phishing attacks that affected systems globally, thereby sending a worldwide signal of the alarming increase in incidents. Phishing remains a favored …


2.5 Years in Class: A Multimodal Textbook for Vision-Language Pretraining

W Zhang, H Zhang, X Li, J Sun, Y Shen, W Lu, D Zhao… - arXiv preprint arXiv …, 2025

… These corpora, consisting of sequences of text paragraphs interspersed with images, are typically crawled from webpage and document, such as Common Crawl Pretraining on a combination of interleaved corpus and image-pair datasets enables …


2nd International Workshop on Natural Scientific Language Processing and Research Knowledge Graphs (NSLP 2025): Preface

G Rehm, S Schimmler, S Dietze, N Manola - International Workshop on Natural …, 2025

Scientific research is almost exclusively published in unstructured text formats, which are not readily machine-readable. While technological approaches can help to get this flood of scientific information and new knowledge under control, the …


3 Jeopardizing Linguistic Diversity

VI English - … Intelligence in Translation: Possibilities, Processes and …, 2025

… The Common Crawl Archive from January 2025, for instance, shows that among the top three languages used online, English makes up 43.37%, followed by Russian at 6.05% and German at 5.59%(Common Crawl nd). Nevertheless, even for …


3,000+ Trabajos

… They trained on 2 trillion tokens of English and Chinese text gotten by deduplicating the Common Crawl. [26] … Further pretrain with 500B tokens (6% DeepSeekMath Corpus, 4% AlgebraicStack, 10% arXiv, 20% GitHub code, 10 …


3. Digital Europe from below: Alternative routes to the Digital Decade

A Mager - Project Europe: The Making of European Digital …, 2025

… engines as search engines that follow a social cause, such as privacy-friendly search engines (eg Startpage and DuckDuckGo),‘green’search engines that donate parts of their revenue to rainforest projects (eg Ecosia), or decentralized, open-source …


7. The Governance of Generative AI: Three Conditions for Research and Policy1

… GPT-3.5, for example, was trained on 45 terabytes of text data, which adds up to approximately 300 billion words extracted from public sources like Wikipedia, CommonCrawl, and GitHub, but also from undisclosed other sources. Open models …


90th Minute: A First Look to Collateral Damages and Efficacy of the Italian Piracy Shield

R Sommese, A Sperotto, A Prado, J van der Ham… - 2025 21th International …, 2025

In the fight against illegal football streaming, Italy introduced Piracy Shield, a platform through which copyright holders can notify the national regulator (AGCOM), which in turn orders ISPs to block infringing resources--such as IP addresses and …


" Amazing, They All Lean Left"--Analyzing the Political Temperaments of Current LLMs

WR Neuman, C Coleman, A Dasdan, S Ali, M Shah… - arXiv preprint arXiv …, 2025

Recent studies have revealed a consistent liberal orientation in the ethical and political responses generated by most commercial large language models (LLMs), yet the underlying causes and resulting implications remain unclear. This paper …


\'Eclair--Extracting Content and Layout with Integrated Reading Order for Documents

I Karmanov, AS Deshmukh, L Voegtle, P Fischer… - arXiv preprint arXiv …, 2025

… We also create a high-quality humanannotated dataset consisting of documents sampled from the Common Crawl corpus [12]. Additionally… In this section, we present examples of predictions from ECLAIR on samples from the Common Crawl …


“Nothing Too Serious”: Corpus Resources and Methods for Data–Driven Approaches to Polarity Sensitivity

This dissertation introduces the Polar Bigrams Resource (PBR), a large-scale corpus-based dataset designed to support data-driven investigations of polarity sensitivity. To address a major challenge for bottom-up processing—the lack of overt indicators of …


“We Share an Unbreakable Bond:” Sociality and Language Ideologies in Human Relationships with Artificial Intelligence

Replika, an artificial intelligence (AI) companion, is part of a growing number of social chatbots. This paper examines the multimodal semiotic signs influencing how users perceive realness in their chatbots. I argue that what users describe as real/alive …


A 4.69 mW LLM Processor with Binary/Ternary Weights for Billion-Parameter Llama Model

S Kim, J Lee, B Kim, HJ Yoo - 2025 IEEE Hot Chips 37 Symposium (HCS), 2025

… is Reduced by 21.3%Och2Och4Och1Och3Och0Chip Photograph and Summary2) Bit means precision of weight & All model's activation is INT84) EMA is included (with DDR3 interface)1) Llama benchmark test @ 50 MHz, 0.65V3) Dataset consists of Pile …


A Bayesian Hybrid Parameter-Efficient Fine-Tuning Method for Large Language Models

Y Chai, Y Liu, Y Zhou, J Xie, DD Zeng - arXiv preprint arXiv:2508.02711, 2025

Large Language Models (LLMs) have demonstrated transformative potential in reshaping the world. As these models are pretrained on general corpora, they often require domain-specific fine-tuning to optimize performance in specialized business …


A BERT-Based Approach to Keyword Search for Digital Evidence Analysis

G Rathnayaka, S Gonawala, P Dananjana… - 2024 6th International …, 2024

Existing forensic tools, especially those relying on traditional keyword searches, often fall short in efficiency and accuracy. These tools are vulnerable to human error, bias, and investigator fatigue, which can result in the omission of crucial evidence …


A bi-level multi-modal fake generative news detection approach: from the perspective of emotional manipulation purpose

L Zhang, Y Shi, M Cui - Humanities and Social Sciences Communications, 2025

As conversational bot Large Models become a daily channel available to everyone, fake Artificial Intelligence Generated Contents (fake AIGCs) have emerged as a serious threat in cyberspace security, with severity varying significantly across …


A Bilingual Legal NER Dataset and Semantics-Aware Cross-Lingual Label Transfer Method for Low-Resource Languages

P Tulajiang, Y Sun, Y Zhang, Y Le, K Xiao, H Lin - ACM Transactions on Asian and Low …

… Our study focuses on domain adaptation and vocabulary expansion rather than solely relying on larger or newer pre-trained models, and XLM-R was trained on a significantly larger CommonCrawl corpus, making it better suited for domain …


A BRIEF SURVEY OF MODEL COMPRESSION IN LANGUAGE MODELS

… To ensure that the quantized model generalized correctly, we gave GPTQ data outside of WikiText, namely a subset of the C4 (Colossal Cleaned Common Crawl) dataset. GPTQ also separates parameters into groups of a fixed size and chooses a …


A Cartography of Open Collaboration in Open Source AI: Mapping Practices, Motivations, and Governance in 14 Open Large Language Model Projects

J Linåker, C Osborne, J Ding, B Burtenshaw - arXiv preprint arXiv:2509.25397, 2025

… (eg, CommonCrawl or The Pile). Developers frequently collaborate by building upon established open datasets like CommonCrawl and … data from “two primary sources: CommonCrawl and the Pile datasets,” noting that “CommonCrawl datasets …


a clinical narrative corpus on nut allergy: annotation schema, guidelines and use case

A González-Moreno, A Ramos-González… - Scientific Data, 2025

… In addition, (3) XLM model ROBERTA Base24 is pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages and was pre-trained on raw texts only, without any human tagging in any way with an automatic process to generate entries …


A Common Pool of Privacy Problems: Legal and Technical Lessons from a Large-Scale Web-Scraped Machine Learning Dataset

R Hong, J Hutson, W Agnew, I Huda, T Kohno… - arXiv preprint arXiv …, 2025

We investigate the contents of web-scraped data for training AI systems, at sizes where human dataset curators and compilers no longer manually annotate every sample. Building off of prior privacy concerns in machine learning models, we ask …


A Comparative Analysis of BERT, RoBERTa, and XLM-RoBERTa for Bengali SMS Multiclass Spam Detection

WA Alvi, R Talukdar, MT Hossain, MS Sayed

… For multilingual applications, Conneau et al. extended RoBERTa to create XLM-RoBERTa, trained on over two terabytes of CommonCrawl text … XLM-R is pretrained on 100 languages using the CommonCrawl corpus, making it suitable for multilingual tasks …


A Comparative Analysis of Static Word Embeddings for Hungarian

M Gedeon - arXiv preprint arXiv:2505.07809, 2025

This paper presents a comprehensive analysis of various static word embeddings for Hungarian, including traditional models such as Word2Vec, FastText, as well as static embeddings derived from BERT-based models using different extraction …


A Comparative Analysis of Transformer Models for the Prediction of Arabic Punctuation

A Aboutaib, A El Allaoui, I Zeroual - … Conference on Artificial Intelligence and Smart …, 2025

We present a comprehensive comparative analysis of different transformer models for the task of punctuation prediction in Arabic text. The models evaluated include Asafaya-BERT, XLM-RoBERTa, Google BERT Multi-lingual, AraBERT, MarBERT …


A Comparative Approach for Auditing Multilingual Phonetic Transcript Archives

F Samir, EP Ahn, S Prakash, M Soskuthy, V Shwartz… - Transactions of the …, 2025

Curating datasets that span multiple languages is challenging. To make the collection more scalable, researchers often incorporate one or more imperfect classifiers in the process, like language identification models. These models …


A Comparative Study of Korean Text Summarization Performance According

M Song - Intelligent Sustainable Systems: Selected Papers of …

In the current NLP research status, there are active studies that have attempted to improve performance through fine-tuning or scaling up by suggesting various PLMs. However, it is difficult to find research analyzing which architecture features of PLMs …


A COMPARATIVE STUDY OF MACHINE LEARNING AND DEEP LEARNING MODELS: STRENGTHS, LIMITATIONS, AND APPLICATIONS

Artificial Intelligence (AI) by enabling systems to learn from data and make intelligent decisions. This paper presents an in-depth analysis of various ML and DL models, comparing their architectures, applications, strengths, and limitations. By exploring …


A Comparative Study of Static and Contextual Embeddings for Analyzing Semantic Changes in Medieval Latin Charters

Y Liu, G Tilahun, X Gao, Q Wen, M Gervers - Proceedings of the First Workshop on …, 2025

The Norman Conquest of 1066 CE brought profound transformations to England’s administrative, societal, and linguistic practices. The DEEDS (Documents of Early England Data Set) database offers a unique opportunity to explore these changes …


A Comparative Study of Task Adaptation Techniques of Large Language Models for Identifying Sustainable Development Goals

A Cadeddu, A Chessa, V De Leo, G Fenu, E Motta… - arXiv preprint arXiv …, 2025

… GPT3, the foundation of GPT-3.5, was trained on a diverse array of datasets, notably including 60% of its foundational training data from a curated version of the Common Crawl dataset47. Other significant data sources include WebText248 …


A Comparative Study on the Development of a Thai Legal QA Framework Using Large Language Models and Mixed Legal Datasets

S Hanwiboonwat, C Thavornthaveekul, P Boonkwan… - International Conference on …, 2025

… The Multilingual E5 [31], is designed for text embedding and provides a more accurate alternative to the older paraphrase-multilingual-mpnet-base-v2 model by utilizing a diverse dataset called CCPairs, which includes community QA content …


A Comparative Survey of Large Language Models: Foundation, Instruction-Tuned, and Multimodal Variants

The rapid evolution of large language models (LLMs) has transformed natural language processing, enabling machines to perform complex language understanding, generation, and reasoning tasks with unprecedented fluency and …


A Comparative Survey on Large Language Models for Biological Data

R Mousa, A Sarabadani, T Taami, AA Bengari… - 2025

The development of large language models (LLMs) has grown exponentially since the release of ChatGPT. Large language models have gained attention for their robust performance across various tasks. The ability of LLMs to understand and …


A Comprehensive Overview and Analysis of Large Language Models: Trends and Challenges

Large Language Models (LLMs) have transformed numerous fields by offering innovative solutions that drive advancements across a wide range of applications. However, their widespread adoption presents several challenges, including …


A Comprehensive Study of LLM and Evolution, Varieties, and Their Role in Software Engineering and Cybersecurity

H Rasel, ABS Didar, AAM Dinar, FI Fahad, MAJ Khan… - 2025

… With 175 billion parameters, which is over 100 times larger than GPT-2, GPT-3 was trained on a mix of Common Crawl, Wikipedia, books, and other large web datasets. It kept the same Transformer decoder-only structure but pushed the limits …


A comprehensive survey on Arabic text augmentation: approaches, challenges, and applications

AA ElSabagh, SS Azab, HA Hefny - Neural Computing and Applications, 2025

Arabic is a linguistically complex language with a rich structure and valuable syntax that pose unique challenges for natural language processing (NLP), primarily due to the scarcity of large, reliable annotated datasets essential for training models. The …


A Comprehensive Survey on Long Context Language Modeling

J Liu, D Zhu, Z Bai, Y He, H Liao, H Que, Z Wang… - arXiv preprint arXiv …, 2025

Efficient processing of long contexts has been a persistent pursuit in Natural Language Processing. With the growing number of long documents, dialogues, and other textual data, it is important to develop Long Context Language Models (LCLMs) …


A Concise Survey on Modern Web‐Based Phishing Techniques and Advanced Mitigation Strategies

D Panneerselvam, SC Sethuraman, AJ Emerson… - Transactions on Emerging …, 2025

Phishing is a tactical technique practiced by cyber‐criminals, wherein the target systems are approached, made vulnerable, and exploited. A Phisher who does the act of phishing is always creative, calculative, and persistent. This potentially leads …


A Cyclical Loss-Based Optimization Algorithm for Pretraining LLMs on Noisy Data

HT Kesgin, MF Amasyali - Knowledge-Based Systems, 2025

Large language models (LLMs) depend on vast web-scale datasets, which frequently include noisy or low-quality samples that degrade performance and fairness—despite conventional data cleaning. This paper introduces an in-training …


A Data-Driven Exploration of Niche Web Community Behavior

The Internet has been instrumental in connecting groups of people with similar interests and viewpoints, enabling Web communities to have a voice through platforms designed for social interaction and engagement. While the Web allows us …


A Deep Dive Into Cross-Dataset Entity Matching with Large and Small Language Models

Z Zhang, P Groth, I Calixto, S Schelter - 2025

… T5 is pretrained on the C4 dataset, a cleaned version of the Common Crawl web corpus. We download the ’C4/en’ version ( 350 GiB) from HuggingFace4 and conduct a sanity check. Each data sample in this dataset includes a URL field that …


A Domain Knowledge-Guided Industrial Large Model Framework: A Case Study in Battery Health Estimation and Recycling

B Chen, H Shao, Y Qin, Y Jin, X Hu - IEEE Transactions on Industrial Informatics, 2025

Accurate prediction of battery state of health (SOH) is essential for optimizing recycling processes. However, existing deep learning models often struggle to adapt to batteries with diverse materials and operating conditions. Some studies have …


A Dual Contrastive Learning Framework for Enhanced Hate Speech Detection in Low-Resource Languages

K Chavinda, U Thayasivam - Proceedings of the First Workshop on Challenges in …, 2025

Hate speech on social media platforms is a critical issue, especially in low-resource languages such as Sinhala and Tamil, where the lack of annotated datasets and linguistic tools hampers the development of effective detection systems. This …


A Feasibility Study & Implementation

TA Bloch, I Inusa - Proceedings of 5th International Conference on Recent …

Natural Language Processing (NLP) is the emerging field research studies of the interaction between human and computing systems. With advancement of NLP techniques, machines are becoming increasingly proficient in understanding …


A Framework for Auditing Chatbots for Dialect-Based Quality-of-Service Harms

E Harvey, RF Kizilcec, A Koenecke - arXiv preprint arXiv:2506.04419, 2025

… Researchers have found that widely-used sources of training data, like the Common Crawl,2 contain hate speech and other harmful content [58], and that this content increases as datasets scale [10]. To address this, LLM developers have …


A Framework for Safe AI: Data Governance and Ecosystem Structure

WT Tsai, L Zhang - 2025 IEEE International Conference on Artificial …, 2025

… Recent complaints in the United States against Cohere’s Command family of models [11] detail extensive use of third-party corpora such as Common Crawl’s C4 without authorization—actions that, according to plaintiffs, constitute both copyright …


A framework for spatial clustering of textual objects: applications in topic clustering and text segmentation

G Guex - Cahiers du Centre de Linguistique et des Sciences du …, 2025

We present a general, classical, framework of spatial clustering which can be applied to various textual objects (eg character n-grams, words, sentences). This framework proposes to cluster objects according to users defined linguistic similarity …


A Grey-box Text Attack Framework using Explainable AI

E Chiramal, KSB Kai - arXiv preprint arXiv:2503.08226, 2025

Explainable AI is a strong strategy implemented to understand complex black-box model predictions in a human interpretable language. It provides the evidence required to execute the use of trustworthy and reliable AI systems. On the other hand …


A High Performance Computing Web Search Engine Based on Big Data and Parallel Distributed Models

… In this paper, we use Common Crawl as a dataset, which is an open-source web … In this paper, a dataset for the month of January 2023 is selected from Common Crawl, which … To reduce the size of the Common Crawl dataset from 300 TB to …


A High-Quality Dataset and Reliable Evaluation for Interleaved Image-Text Generation

Y Feng, J Sun, C Li, Z Li, J Ai, F Zhang, Y Chang… - arXiv preprint arXiv …, 2025

Recent advancements in Large Multimodal Models (LMMs) have significantly improved multimodal understanding and generation. However, these models still struggle to generate tightly interleaved image-text outputs, primarily due to the …


A Hybrid Architecture with Efficient Fine Tuning for Abstractive Patent Document Summarization

N Jayatilleke, R Weerasinghe - arXiv preprint arXiv:2503.10354, 2025

Automatic patent summarization approaches that help in the patent analysis and comprehension procedure are in high demand due to the colossal growth of innovations. The development of natural language processing (NLP), text mining …


A Hybrid CNN-BLSTM Model for Phishing Attack Detection Using Deep Learning to Strengthen Internet Security

AA Alsabri, MA Al-Hadi - Sana'a University Journal of Applied Sciences and …, 2025

… The dataset was populated from the parents of the main sources of legitimate URLs based on Common Crawl to complement PhishTank … The dataset was compiled from PhishTank, OpenPhish, and Common Crawl. After cleansing and …


A Japanese Language Model and Three New Evaluation Benchmarks for Pharmaceutical NLP

I Sukeda, T Fujii, K Buma, S Sasaki, S Ono - arXiv preprint arXiv:2505.16661, 2025

… We first sampled a subset of documents from the Common Crawl dataset (CC100). A high-performing LLM (Qwen2.5-72B) was prompted to … high-quality, pharmaceutical-related documents (totalling 1.2 billion tokens) from the deduplicated …


A large language model algorithm for green finance innovation for digital technology innovation of heavily polluting enterprises

Y Shen, W Lu - Proceedings of the 10th International Conference on …, 2025

… In the pre-training stage, selfsupervised learning is conducted through massive unannotated text data (such as Common Crawl, Wikipedia, etc.) to enable the model to master general language representation capabilities. In the fine-tuning stage, the …


A map of words: Retrieving the spatial layout of medium-scale geographical maps through distributional semantics

G Anceresi, D Gatti, T Vecchi, M Marelli, L Rinaldi - Neuropsychologia, 2025

Recent evidence has indicated that spatial representations, such as large-scale geographical maps, can be retrieved from natural language alone through cognitively plausible distributional-semantic models, which capture word meanings …


A Memory Efficient Randomized Subspace Optimization Method for Training Large Language Models

Y Chen, Y Zhang, Y Liu, K Yuan, Z Wen - arXiv preprint arXiv:2502.07222, 2025

The memory challenges associated with training Large Language Models (LLMs) have become a critical concern, particularly when using the Adam optimizer. To address this issue, numerous memory-efficient techniques have been proposed …


A Multi-Modal Large Language Model for Free-Form, Open-Ended, and Interactive Prediction of Properties and Mechanisms of Candidate Drug Molecules

Y Liang, R Zhang, Z Ma, D Singh, Y Li, M Huo, C Gao…

Accurately predicting the mechanisms and properties of candidate drug molecules is critical for advancing drug discovery. However, existing models are often limited to structured outputs, fixed task sets, and static, one-shot predictions. We present …


A Neural Network Approach to Sentiment Analysis

SK Singh, M Srivastava, N Kumar, N Singh, A Singh

… In the case of pre-trained embeddings like GloVe or Word2Vec, one would download a pre-trained embeddings file (eg, 300-dimensional vectors trained on Google News or Common Crawl) and create an embedding matrix for the …


A new Approach to Programming: AI Agents, LLMs, and an SQL Generation Case Study

The rise of Large Language Models (LLMs) and AI agents is transforming software development, introducing new paradigms in automation and human-machine collaboration. This thesis, conducted in collaboration with Poseidon, a company …


A New Pair of GloVes

R Carlson, J Bauer, CD Manning - arXiv preprint arXiv:2507.18103, 2025

This report documents, describes, and evaluates new 2024 English GloVe (Global Vectors for Word Representation) models. While the original GloVe models built in 2014 have been widely used and found useful, languages and the world continue to …


A novel approach for mitigating class imbalance in Arabic text classification

E Nabil, AE Nagib, M Hany, S Faizullah, WH Gomaa - IEEE Access, 2025

… In this study, the embedding model we employed was XLMRoBERTa-Large, a transformer-based multilingual language model built upon the RoBERTa architecture and trained on a large-scale corpus comprising 2.5 TB of filtered …


A Novel Approach to Automated Detection of AI-Generated Text

HM Abbas - Journal of Al-Qadisiyah for Computer Science and …, 2025

Detecting machine-generated text involves identifying whether text has been created by artificial intelligence models or written by humans. This task has become increasingly significant due to the potential misuse of AI-generated text for producing …


A Novel Assistant for Question-Answering from Training Video Sessions Using RAG

Q Kembellec, K Boutalbi, O Le Van - 2025 IEEE 49th Annual Computers, Software …, 2025

… developed, including the Robustly Optimized BERT Pretraining Approach (RoBERTa) [24], XLM-RoBERTa (large-sized model) [12] pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. A few models are pre-trained on …


A Novel Dual-Strategy Approach for Constructing Knowledge Graphs in the Home Appliance Fault Domain

D Zhang, J Zhang, Y Jia, M Liao - Algorithms, 2025

Knowledge graph technology holds significant importance for efficient fault diagnosis in household appliances. However, the scarcity of public fault diagnosis data and the lack of automated knowledge extraction pose major challenges to …


A Paradigm Gap in Urdu

F Adeeba, R Bhatt - arXiv preprint arXiv:2509.01084, 2025

In this paper, we document a paradigm gap in the combinatorial possibilities of verbs and aspect in Urdu: the perfective form of the -ya: kar construction (eg ro-ya: ki: cry-Pfv do.Pfv) is sharply ungrammatical in modern Urdu and Hindi, despite being …


A Peek Behind the Curtain: Using Step-Around Prompt Engineering to Identify Bias and Misinformation in GenAI Models

D Hickerson, M Perkins - arXiv preprint arXiv:2503.15205, 2025

This research examines the emerging technique of step-around prompt engineering in GenAI research, a method that deliberately bypasses AI safety measures to expose underlying biases and vulnerabilities in GenAI models. We discuss how …


A Proposal of Post-OCR Spelling Correction Using Monolingual Byte-level Language Models

SS de Araújo, BLD Bezerra, AF de Sousa Neto - … of the 2025 ACM Symposium on …, 2025

This work presents a proposal for a spelling corrector using monolingual byte-level language models (Monobyte) for the post-OCR task in texts produced by Handwritten Text Recognition (HTR) systems. We evaluate three Monobyte models …


A review of advanced prompting techniques in Large Language Models (LLMs)

Abstract This study investigates sophisticated prompting methods used to guide large language models more effectively. I analyzed techniques like zero-shot, CoT, ToT, and persona-based prompting for their ability to improve performance, accuracy …


A review of large language models and the recommendation task

J Munson, T Cuezze, S Nesar, D Zosso - Discover Artificial Intelligence, 2025

Recommender systems are now ubiquitous across the internet, from streaming services to online shopping to social media. Traditional systems operate behind the scenes, often invisible to the end user. While these systems have enjoyed prolific …


A REVIEW ON THE FUTURE OF GENERATIVE AI SYSTEMS

GK Dixit, S Kumar, H Kaur, S Choudhary, V Kumar

Generative AI is reshaping industries through its ability to create new content—from text and images to audio and code—by learning patterns from vast datasets. In this paper, we examine the origins and evolution of Generative AI, explore its …


A REVISED TECHNIQUE TO TRAIN TERM/WORD VECTOR SPACE MODELS APPLYING THE ONTOLOGY-RELATED APPROACH

OV Palagin, VY Velychko, KS Malakhov, OS Shchurov

… The most common datasets include an entire corpus of Wikipedia texts, the common crawl dataset [43], or the … [Online] Available from http://commoncrawl.org [Accessed: 03 February 2020]. … [Online] Available from http://commoncrawl.org [Accessed: 03 …


A Scalable Model for Frequency Distribution of Low Occurrence Multi-words Towards Handling Very Large Spectrum of Text Corpora Sizes

Predicting the diversity of words and multi-words (n-grams) in a text corpus and their frequency distributions is important in NLP and language modeling, and is becoming critical to enable the design of modern applications, namely Large …


A Semantic Parsing Framework for End-to-End Time Normalization

X Su, S Yu, P Howard, S Bethard - arXiv preprint arXiv:2507.06450, 2025

Time normalization is the task of converting natural language temporal expressions into machine-readable representations. It underpins many downstream applications in information retrieval, question answering, and clinical decision-making …


A Semantic Retrieval and Generation Framework for Alumni Intelligence Using LLaMA 3

B Acharya, S Koirala, R Chaudhari, K Gautam… - Journal of Engineering …, 2025

… Llama models utilize dense transformer architectures, SwiGLU activation, rotary positional embeddings, and have been trained on curated datasets of up to 2 trillion tokens from sources like CommonCrawl, C4, Wikipedia, and StackExchange. These …


A Semantics-aware Head-driven Approach for Multilingual Dependency Parsing

TMH Nguyen, P Le-Hong - IEEE Access, 2025

… FastText distributes pretrained word vectors for 157 languages, trained on Common Crawl and Wikipedia. These models were trained using CBOW with position-weights, in dimension 300, with character n-grams of length 5, a window …


A Simulation-Based Slope Metric for Anchor List Reliability in Word Embedding Spaces

MA Taylora, DS Stoltzb, H Harpera, S Kumarc… - 2025

Inducing semantic relations in word vector spaces and analyzing how other words or entire documents discursively engage these relations is a popular form of cultural analysis. We propose a reliability metric that is easily interpretable and agnostic to …


A Social Media-Driven Visual-Language Framework for Disaster Analysis and Reporting

With the widespread use of social media in disaster events, massive multimodal data (images and text) provide a new source of information for disaster monitoring and response, but problems such as high data heterogeneity, high noise, and cross-modal …


A Study on Automatic English Grammatical Error Correction Using Transformer and BERT

This study evaluates grammatical error correction using Transformer and pre-trained BERT models. The goal is to understand how regulating different parameters can affect the effectiveness of these two models in mitigating errors in an English text …


A Study on the Automation of Topic Modeling Using Prompt Engineering-Based in ChatGPT

DK Jung, JH Lee - Knowledge Management Research, 2025

This study proposes a methodology that can perform topic modeling analysis using only natural language-based instructions, utilizing ChatGPT, an interactive AI model, and Prompt Engineering techniques. Existing topic modeling has limitations in that it …


A Survey of AI for Materials Science: Foundation Models, LLM Agents, Datasets, and Tools

MH Van, P Verma, C Zhao, X Wu - arXiv preprint arXiv:2506.20743, 2025

Foundation models (FMs) are catalyzing a transformative shift in materials science (MatSci) by enabling scalable, general-purpose, and multimodal AI systems for scientific discovery. Unlike traditional machine learning models, which are typically narrow in …


A survey of datasets in medicine for large language models

D Zhang, X Xue, P Gao, Z Jin, M Hu, Y Wu, X Ying - Intelligence & Robotics, 2024

With the advent of models such as ChatGPT and other models, large language models (LLMs) have demonstrated unprecedented capabilities in understanding and generating natural language, presenting novel opportunities and challenges …


A Survey of Deep Learning Architectures in Modern Machine Learning Systems: From CNNs to Transformers

T Mowbray - Journal of Computer Technology and Software, 2025

… Simultaneously, the accessibility of massive datasets such as ImageNet, COCO, OpenWebText, and Common Crawl has made it feasible to train large-capacity models capable of capturing complex data distributions. This convergence has led …


A Survey of DeepSeek Models

F Neha, D Bhati - Authorea Preprints, 2025

Advances in artificial intelligence (AI) rely on systems capable of human-like reasoning, a limitation for conventional Large Language Models (LLMs), which struggle with multi-step logic, abstract conceptualization, and latent relationship …


A Survey of Large Language Models: Evolution, Architectures, Adaptation, Benchmarking, Applications, Challenges, and Societal Implications

SMS Mohammadabadi, BC Kara, C Eyupoglu, C Uzay… - 2025

This survey offers an in-depth review of Large Language Models (LLMs), highlighting the significant paradigm shift they represent in artificial intelligence. Our purpose is to consolidate state-of-theart advances in LLM design, training …


A Survey of Large Language Models: Foundations and Future Directions

This survey presents a comprehensive overview of large language models (LLMs), with a particular focus on the evolution and foundational role of attention mechanisms—tracing their origins from psychological and algorithmic precursors to …


A Survey of LLM $\times $ DATA

X Zhou, J He, W Zhou, H Chen, Z Tang, H Zhao… - arXiv preprint arXiv …, 2025

The integration of large language model (LLM) and data management (DATA) is rapidly redefining both domains. In this survey, we comprehensively review the bidirectional relationships. On the one hand, DATA4LLM, spanning large-scale data …


A Survey of LLM× DATA


A Survey of Low-Resource Sentence Representation

J Ma, N Liu, Z Zhang, G Liu, T Liu, M Lu, N Wu, Y Ji - 2025 International Conference …, 2025

… [8] proposed XLM-R, which significantly improved cross-language task performance by pre-training on over 2TB of filtered CommonCrawl data across 100 languages. The success of XLM-R demonstrates the effectiveness of largescale data …


A survey of multilingual large language models

L Qin, Q Chen, Y Zhou, Z Chen, Y Li, L Liao, M Li… - Patterns, 2025

Multilingual large language models (MLLMs) leverage advanced large language models to process and respond to queries across multiple languages, achieving significant success in polyglot tasks. Despite these breakthroughs, a comprehensive …


A Survey of NLP Progress in Sino-Tibetan Low-Resource Languages

S Liu, M Best - Proceedings of the 2025 Conference of the Nations of …, 2025

… 8 for CommonCrawl9, which one of the widely used corpora collected from the Internet for training language models. For ST languages, the CommonCrawl corpus only contains the Chinese languages (not specified which ones), Tibetan, Burmese …


A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontiers

M Hu, C Ma, W Li, W Xu, J Wu, J Hu, T Li, G Zhuang… - arXiv preprint arXiv …, 2025

Scientific Large Language Models (Sci-LLMs) are transforming how knowledge is represented, integrated, and applied in scientific research, yet their progress is shaped by the complex nature of scientific data. This survey presents a …


A survey on 1-bit quantized large language models

K Tripathi, D Malik, A Akshat, K Lata - Neural Computing and Applications, 2025

… C4 [1, 14, 34, 36, 56, 60, 61] is a large, colossal, and clean version of Common Crawl’s web … It contains over 100B text documents which are taken from 84 CommonCrawl snapshots. … over 100B text documents coming from 84 …


A Survey on AI Search with Large Language Models

J Li, X Li, Y Zheng, Y Jin, S Wang, J Wu, Y Wang… - 2025

Searching for accurate information is a complex task that demands significant effort. Although search engines have transformed the way we access information, they often struggle to understand intricate human intentions fully. Recently, Large …


A Survey on Data Contamination for Large Language Models

Y Cheng, Y Chang, Y Wu - arXiv preprint arXiv:2502.14425, 2025

… Li (2023b) present Contamination Detector to check whether test examples appear on the internet via Bing search and Common Crawl index. The tool is available at: https://github.com/ liyucheng09/Contamination_Detector. Ravaut et al. …


A Survey on Intelligent Network Operations and Performance Optimization Based on Large Language Models

S Long, J Tan, B Mao, F Tang, Y Li, M Zhao, N Kato - IEEE Communications Surveys …, 2025

… For example, the Common Crawl dataset, a public dataset containing billions of web pages [131], offers a wide range of linguistic samples and is particularly useful for training language models. Additionally, the datasets Books1 and Books2 [132] …


A Survey on Large Language Models for Communication, Network, and Service Management: Application Insights, Challenges, and Future Directions

GO Boateng, H Sami, A Alagha, H Elmekki… - arXiv preprint arXiv …, 2024

… The data for model pre-training may come from various sources such as common crawl [77], Wikipedia dumps, web text [78], video databases (CCTV camera feed) [75], image files, and specific data for application-specific tasks. Common crawl is a …


A Survey on Large Language Models with some Insights on their Capabilities and Limitations

A Matarazzo, R Torlone - arXiv preprint arXiv:2501.04040, 2025

The rapid advancement of artificial intelligence, particularly with the development of Large Language Models (LLMs) built on the transformer architecture, has redefined the capabilities of natural language processing. These models now exhibit …


A Survey on Mathematical Reasoning and Optimization with Large Language Models

A Forootani - arXiv preprint arXiv:2503.17726, 2025

… To bridge this gap, Deepseek [218] introduced a domain-specific model (spmath) trained on the DeepSeekMath Corpus, a high-quality dataset of 120B math tokens curated from Common Crawl using a fastText-based classifier [219]. This model …


A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends

Y Ding, S Luo, Y Dai, Y Jiang, Z Li, G Martin, Y Peng - arXiv preprint arXiv …, 2025

Visually-Rich Document Understanding (VRDU) has emerged as a critical field, driven by the need to automatically process documents containing complex visual, textual, and layout information. Recently, Multimodal Large Language Models (MLLMs) …


A Survey on Protecting Users Against Phishing Attacks

A Albishri, MM Dessouky - 2025 2nd International Conference on Advanced …, 2025

… For instance, paper [1] relies on data obtained from the Alexa Database and common crawl. This dataset comprises both phishing and non-phishing websites, providing a rich ground for the development of machine learning algorithms that …


A Survey on Responsible LLMs: Inherent Risk, Malicious Use, and Mitigation Strategy

H Wang, W Fu, Y Tang, Z Chen, Y Huang, J Piao… - arXiv preprint arXiv …, 2025

While large language models (LLMs) present significant potential for supporting numerous real-world applications and delivering positive social impacts, they still face significant challenges in terms of the inherent risk of privacy leakage …


A Survey on the Impact of Pre-Trained Language Models in Sentiment Classification Task

H Gautam, A Gaur, DK Yadav - International Journal of Data Science and Analytics, 2025

The evolution of pre-trained language models (PLMs) has significantly transformed the landscape of sentiment analysis, particularly in handling complex, noisy, informal, and short-text commonly found on social media. While numerous surveys have …


A Systematic Analysis of Base Model Choice for Reward Modeling

K Ahrabian, P Jandaghi, N Mokhberian… - arXiv preprint arXiv …, 2025

… We believe this is due to the potential occurrence of similar documents in the excluded CommonCrawl and C4 categories. Figure 7 showcases the Jensen-Shannon Distance (JSD) between different models over the scores of the entire 1M samples …


A systematic survey of natural language processing for the Greek language

J Bakagianni, K Pouli, M Gavriilidou, J Pavlopoulos - Patterns, 2025

Comprehensive monolingual natural language processing (NLP) surveys are essential for assessing language-specific challenges, resource availability, and research gaps. However, existing surveys often lack standardized methodologies …


A technical background on artificial intelligence and intelligent language models

Stunning advancements in artificial intelligence (AI) over the last several years have undoubtedly opened new possibilities and challenges for the field of second language learning. Of course, AI is not new, and for decades it has attracted the …


A Temporal Knowledge Graph Generation Dataset Supervised Distantly by Large Language Models

J Zhu, Y Fu, J Zhou, D Chen - Scientific Data, 2025

Abstract Knowledge graphs can be constructed by extracting triples from documents, which denotes document-level relation extraction. Each triple illustrates a fact composed of two entities and a relation. However, temporal information …


A Tough Row to Hoe: Instruction Fine-Tuning LLaMA 3.2 for Multilingual Sentence Disambiguation and Idiom Identification

Idiomatic expressions (IEs) are a fundamental aspect of language, traditionally defined as expressions whose meanings cannot be inferred from their individual components. However, modern linguistic theories propose a more complex …


A Unified Approach on Phishing Detection using Chrome Extension Integrating ML and NLP

GS Prakash, DS Muthukumar, R Kumar, P Rashmi… - 2025 3rd International …, 2025

… We sourced legitimate URLs from established repositories such as Common Crawl and DMOZ, while phishing URLs were collected from PhishTank, OpenPhish, and APWG databases. This deliberate diversity in data sources was crucial for …


Abstractive Event Analysis of Armed Conflicts: Introducing the UCDP-AEC Dataset

É Simon, HB Olsen, R Carreño, R Mishra, N Arefyev… - … 2025 Conference on …, 2025

This paper introduces a new dataset of document-level event annotations in the domain of armed conflict. By augmenting the event database from the Uppsala Conflict Data Program (UCDP) with source documents identified in public web …


Accelerating High-Dimensional Nearest Neighbor Search with Dynamic Query Preference

Y Gao, R Zhao, Z Li, B Zheng, Y Zhu, Z Chen - arXiv preprint arXiv:2508.07218, 2025

Approximate Nearest Neighbor Search (ANNS) is a crucial operation in databases and artificial intelligence. Current graph-based ANNS methods, such as HNSW and NSG, have shown remarkable performance but are designed under the assumption …


AccelES: Accelerating Top-K SpMV for Embedding Similarity via Low-bit Pruning

J Zhai, X Shi, K Huang, C Ye, W Hu, B He, H Jin - 2025 IEEE International …, 2025

In the realm of recommendation systems, achieving real-time performance in embedding similarity tasks is often hindered by the limitations of traditional Top-K sparse matrix-vector multiplication (SpMV) methods, which suffer from high latency …


Accessibility Barriers in Multi-Terabyte Public Datasets: The Gap Between Promise and Practice

M Bara - arXiv preprint arXiv:2506.13256, 2025

… Common Crawl stands out as the most genuinely accessible massive dataset, offering 34-100+ TB per monthly crawl with petabytes accumulated since 2008 [1]. The data is freely available via AWS S3 in WARC, WAT, and WET formats, with …


ACECODER: Acing Coder RL via Automated Test-Case Synthesis

H Zeng, D Jiang, H Wang, P Nie, X Chen, W Chen - arXiv preprint arXiv:2502.01718, 2025

Most progress in recent coder models has been driven by supervised fine-tuning (SFT), while the potential of reinforcement learning (RL) remains largely unexplored, primarily due to the lack of reliable reward data/model in the code domain. In this …


ADAM: A Diverse Archive of Mankind for Evaluating and Enhancing LLMs in Biographical Reasoning

J Cekinmez, O Ghahroodi, SF Chandle, D Gupta… - arXiv preprint arXiv …, 2025

We introduce ADAM (A Diverse Archive of Mankind), a framework for evaluating and improving multimodal large language models (MLLMs) in biographical reasoning. To the best of our knowledge, this is the first work to systematically examine LLM …


AdaNDV: Adaptive Number of Distinct Value Estimation viaLearning to Select and Fuse Estimators

X Xu, T Zhang, X He, H Li, R Kang, S Wang, L Xu… - arXiv preprint arXiv …, 2025

Estimating the Number of Distinct Values (NDV) is fundamental for numerous data management tasks, especially within database applications. However, most existing works primarily focus on introducing new statistical or learned estimators, while …


Adapting Language Models to Indonesian Local Languages: An Empirical Study of Language Transferability on Zero-Shot Settings

RA Putri - arXiv preprint arXiv:2507.01645, 2025

In this paper, we investigate the transferability of pre-trained language models to low-resource Indonesian local languages through the task of sentiment analysis. We evaluate both zero-shot performance and adapter-based transfer on ten local languages …


Adapting Large Language Models for Character-based Augmentative and Alternative Communication

D Gaines, K Vertanen - arXiv preprint arXiv:2501.10582, 2025

Users of Augmentative and Alternative Communication (AAC) may write letter-by-letter via an interface that uses a character language model. However, most state-of-the-art large pretrained language models predict subword tokens of variable length. We …


Adaptive Parallel Processing Algorithm with Dynamic Scheduling for Large-Scale Data Processing in Cloud Environments: Implementation and Performance …

Y Zhang, D Yi, S Wu, S Cheng - Informatica, 2025

As large-scale data processing tasks continue to grow in volume and complexity, improving the efficiency of computational resource utilization and task execution performance has emerged as a central challenge in cloud computing environments …


Adaptive Phishing Detection in Web Applications Using Ensemble Deep Learning and Feature Fusion Techniques

Phishing attacks represent one of the most persistent and evolving threats to web applications, often leading to severe financial loss, data breaches, and the compromise of user trust. Conventional detection techniques based on blacklists …


Adaptive sorting for large keys, strings, and database rows

M Kuhrt, B Seeger, S Wild, G Graefe - … für Business, Technologie und Web (BTW 2025 …, 2025

As sorting a database table may require expensive comparisons, eg, due to column count or column types such as long or international strings, optimizing the count and cost of comparisons is important. Adaptive sorting avoids comparisons by exploiting …


Addressing Bias in LLMs: Strategies and Application to Fair AI-based Recruitment

A Peña, J Fierrez, A Morales, G Mancera, M Lopez… - arXiv preprint arXiv …, 2025

The use of language technologies in high-stake settings is increasing in recent years, mostly motivated by the success of Large Language Models (LLMs). However, despite the great performance of LLMs, they are are susceptible to ethical concerns …


Advanced Implementation of a Multilevel Model for Text Summarization in Kazakh Using Pretrained Models

D Oralbekova, O Mamyrbayev, M Othman… - Engineering, Technology & …, 2025

This study investigates transformer models for the task of hybrid text summarization in the Kazakh language. Using mBART, mT5, and XLM-RoBERTa models, a multilevel architecture was developed that processes text at the character, subword …


Advanced Layout Analysis Models for Docling

N Livathinos, C Auer, A Nassar, RT de Lima, M Lysak… - arXiv preprint arXiv …, 2025

… We have incorporated WordScape documents from the 2013 CommonCrawl snapshot into our data mix. However, a detailed inspection of the annotations revealed a significant semantic mismatch: WordScape’s “Table” label is frequently applied to …


Advanced technique for firmware security analysis through heterogeneous data fusion and knowledge mapping

P Xiao, L Xie, F Hang, H Li - PloS one, 2025

As the core component of a device, firmware’s security directly affects the stability of the entire system and the security of user data. In order to provide a more comprehensive and accurate data foundation and improve the accuracy of firmware …


Advanced Tool Learning and Selection System (ATLASS): A Closed-Loop Framework Using LLM

MA Haque, J Williams, S Siddique, MH Islam, H Ali… - arXiv preprint arXiv …, 2025

The combination of LLM agents with external tools enables models to solve complex tasks beyond their knowledge base. Human-designed tools are inflexible and restricted to solutions within the scope of pre-existing tools created by experts. To …


Advancements in Natural Language Processing: Leveraging Transformer Models for Multilingual Text Generation

MZ Hossain, S Goyal - Pacific Journal of Advanced Engineering Innovations, 2024

Background: Recent advancements in Natural Language Processing (NLP) have revolutionized text generation techniques, with Transformer models becoming the cornerstone of modern NLP tasks, particularly in multilingual text generation …


Advancements in Transformer-Based Models for Enhanced Hate Speech Detection in Arabic: Addressing Dialectal Variations and Cross-Platform Challenges

A Fat'hAlalim, Y Liu, Q Xie, N Ibrahim - ACM Transactions on Asian and Low …, 2025

… It was trained on 2.5TB of newly created clean CommonCrawl data in 100 languages. We used the xlm-roberta-base version[60]. • AlBERT: AlBERT is a lite BERT model that presents two parameter-reduction techniques to lower memory …


Advancing EHR analysis: Predictive medication modeling using LLMs

H Alghamdi, A Mostafa - Information Systems, 2025

In modern healthcare systems, the analysis of Electronic Health Records (EHR) is fundamental for uncovering patient health trends and enhancing clinical practices. This study aims to advance EHR analysis by developing predictive models for …


Advancing Eye-Gaze Writing Systems With Computer Vision, and Dynamic Text Suggestions

Eye gaze writing, a novel interaction modality, has the potential to revolutionize communication for individuals with limited mobility. In our research, we investigated the deep learning algorithms efficiency for real-time eye gaze writing. We have …


Advancing Math Reasoning in Language Models: The Impact of Problem-Solving Data, Data Synthesis Methods, and Training Stages

Z Chen, T Liu, M Tian, Q Tong, W Luo, Z Liu - arXiv preprint arXiv:2501.14002, 2025

… 2) Mathematical corpus is designed to enhance the model’s proficiency in mathematics, primarily comprising general mathematical content extracted from sources like CommonCrawl web pages. The main objective is to imbue the pre-trained …


Advancing models of semantic representation: empirical study designs, network analysis methods, and computational tools

People learn an astonishing amount of things about the world throughout life. This knowledge is retained in semantic representations, the cognitive manifestation of factual information in memory. Modeling semantic representations is an important …


Advancing real-time infectious disease forecasting using large language models

H Du, Y Zhao, J Zhao, S Xu, X Lin, Y Chen, LM Gardner… - Nature Computational …, 2025

Forecasting the short-term spread of an ongoing disease outbreak poses a challenge owing to the complexity of contributing factors, some of which can be characterized through interlinked, multi-modality variables, and the intersection of …


Advancing Retrieval-Augmented Generation for Persian: Development of Language Models, Comprehensive Benchmarks, and Best Practices for Optimization

SB Hosseinbeigi, S Asghari, MAS Kashani… - arXiv preprint arXiv …, 2025

This paper examines the specific obstacles of constructing Retrieval-Augmented Generation(RAG) systems in low-resource languages, with a focus on Persian's complicated morphology and versatile syntax. The research aims to improve …


Advancing tourism sentiment analysis: a comparative evaluation of traditional machine learning, deep learning, and transformer models on imbalanced datasets

S Srianan, A Nanthaamornphong, C Phucharoen - Information Technology & Tourism, 2025

Tourism sentiment analysis faces substantial challenges due to class imbalance and the complex linguistic features of user-generated content. This study systematically compares eight sentiment classification models, spanning traditional machine …


Advantageous Parameter Expansion Training Makes Better Large Language Models

N Gu, Y Chen, Z Zhang, P Fu, Z Lin, S Wang, Y Sun… - arXiv preprint arXiv …, 2025

Although scaling up the number of trainable parameters in both pre-training and fine-tuning can effectively improve the performance of large language models, it also leads to increased computational overhead. When delving into the parameter difference, we …


Adversarial Attacks against Neural Ranking Models via In-Context Learning

A Bigdeli, N Arabzadeh, E Bagheri, CLA Clarke - arXiv preprint arXiv:2508.15283, 2025

… Given the large document sizes in the Common Crawl news collection and the C4 collection, we divide documents into chunks of 512 tokens with a stride of 256 tokens. We determine the relevance score of the topicdocument pair used in the re-ranking …


Adversarial Learning for Cross-Lingual Word Embeddings

In the field of natural language processing, current neural network systems are hungry for labelled data. However, large amounts of human-annotated or human-corrected labelled data are only available for a limited number of languages. Previous studies …


Adversarial Speech-Text Pre-Training for Speech Translation

C Liu, L Chen, W Zhang, X Li, P Tang, M Yu, S Ghosh… - ICASSP 2025-2025 IEEE …, 2025

Large-scale pre-training has been shown to benefit speech translation tasks. However, existing multimodal pre-training efforts rely on parallel corpora for semantic alignment, potentially limiting performance to the scale of available data …


AEHRC at BioLaySumm 2025: Leveraging T5 for Lay Summarisation of Radiology Reports

W Zhang, S Chandra, B Koopman, J Dowling… - Proceedings of the 24th …, 2025

Biomedical texts, such as research articles and clinical reports, are often written in highly technical language, making them difficult for patients and the general public to understand. The BioLaySumm 2025 Shared Task addresses this challenge by …


AFRIDOC-MT: Document-level MT Corpus for African Languages

JO Alabi, IA Azime, M Zhang, C España-Bonet… - arXiv preprint arXiv …, 2025

This paper introduces AFRIDOC-MT, a document-level multi-parallel translation dataset covering English and five African languages: Amharic, Hausa, Swahili, Yor\`ub\'a, and Zulu. The dataset comprises 334 health and 271 information technology news …


AfroXLMR-Comet: Multilingual Knowledge Distillation with Attention Matching for Low-Resource languages

JS Raju, JS Walia, S Raghav, V Marivate - arXiv preprint arXiv:2502.18020, 2025

… manually audited, general domain 3T token monolingual dataset based on CommonCrawl, spanning 419 languages. We employ a multilingual subset of the dataset that represents African languages, specifically Kinyarwanda (rw), Swahili (sw) …


Aggarwal, CC (2018). Artificial Neural Network and Deep Learning. Springer. Anderson, JA (1995). An Introduction to Neural Networks. The MIT Press. Bahdanau, D …

N Rahayu - Deep Learning: Teori, Algoritma, dan Aplikasi, 2025

… Dataset besar seperti Common Crawl atau Wikipedia sering digunakan untuk melatih model ini. Hasil: model transformer seperti GPT-4 mampu menghasilkan teks yang menyerupai manusia dan melakukan berbagai tugas NLP . Aplikasi …


AGI for the Earth, the path, possibilities and how to evaluate intelligence of models that work with Earth Observation Data?

M Valipour, K Zheng, J Lowman, S Szabados… - arXiv preprint arXiv …, 2025

… available datasets such as Common Crawl3. Human language only encapsulates one dimension of intelligence in which we can encode observations, events, their interrelations, and basically any concept that can be described textually. The …


AI Applications for Ancient Art History Education

… , scrubbed information from large, publicly available datasets from companies such as LAION and Common Crawl that “web crawl” the internet to process and index the information to search engines (Common Crawl, 2025). As well as other …


AI as a Child in a Cage: On Mirrors, Obedience, and the Illusion of Intelligence

This paper explores the metaphor of the child in the cage as a framework for understanding the development of artificial intelligence systems under conditions of constraint. Contemporary large language models (LLMs) are trained not on the full …


AI Based Mock Interview System Using Natural Language Processing

K Senthilkumar, S Ranjith, M Sivasakthi… - … International Conference on …, 2025

Mock interviews are crucial for job applicants to enhance and polish their skills before to real interviews. This study presents a novel AI-driven Mock Interview System that replicates authentic interview settings through the utilization of Natural …


AI Explains: ChatGPT

In a world where technology evolves at a breakneck pace,'AI Explains: ChatGPT'offers a comprehensive exploration of one of the most transformative innovations of our time. This book delves into the intricacies of ChatGPT, a model that has redefined …


AI Scaling: From Up to Down and Out

Y Wang, Y Li, C Xu - arXiv preprint arXiv:2502.01677, 2025

AI Scaling has traditionally been synonymous with Scaling Up, which builds larger and more powerful models. However, the growing demand for efficiency, adaptability, and collaboration across diverse applications necessitates a broader perspective …


AI tool for scientific literature data extraction

The aim of this project was to explore artificial intelligence (AI) by developing a tool that leverages large language models (LLMs) to extract structured information from scientific articles. Systematic literature reviews and meta-analyses are based on …


AI Tools and Technologies for Academic Research

A Szendi, D Kuttor, Z Pál - Institutional guide to using AI for research, 2025

This chapter explores the landscape of Artificial Intelligence (AI) tools and technologies specifically suited for academic research, emphasizing the role of locally run Generative AI (GenAI) applications. By examining the structure of data …


AI University Education

• The need for such education permeates all levels of education and all social strata.• A 1/3-2/3 society, where 1/3 of the population understands and benefits from scientific progress, while the remaining 2/3 lags, being impoverished and …


AI-assisted German Employment Contract Review: A Benchmark Dataset

O Wardas, F Matthes - arXiv preprint arXiv:2501.17194, 2025

Employment contracts are used to agree upon the working conditions between employers and employees all over the world. Understanding and reviewing contracts for void or unfair clauses requires extensive knowledge of the legal system …


AI-Based Digital Advertising Tools in English and Business

N Singh - Application of English in Artificial Intelligence (AI) And …, 2025

The integration of Artificial Intelligence (AI) into advertising has revolutionized the way businesses target consumers and create advertisements. This paper explores the use of AI-based tools in English advertising and their impact on businesses. By …


AI-Based Pronunciation Assessment and Grammatical Error Correction with Feedback for the German Language

SN Mehta, A Roth, C Munteanu, S Chandna - International Conference on Human …, 2025

The rapid advancement of AI has transformed education and language learning, leading to the development of Computer Aided Language Learning (CALL) systems. These systems help learners practice reading, writing, pronunciation, and vocabulary …


AI-Generated Content and the Pollution of the Information Sphere: A Freedom of Expression Analysis under Article 10 ECHR

This thesis analyses whether and to what extent AI-generated content is protected by freedom of expression and information under Article 10 §1 ECHR, and under what conditions state interferences can be justified to protect the integrity of the …


AI-Generated Content in Copyright Law: A Roadmap for Updating GCC Copyright Law

S Papastefanou - Innovation and Development of Knowledge Societies, 2025

The rise of Text-to-Image Diffusion Models (TIDM) and the ensuing possibility to create complex images with a few text specifications poses a challenge to the fundamentals of Intellectual Property Law. In view of the ambitious goals of the GCC …


AI-generated stories favour stability over change: homogeneity and cultural stereotyping in narratives generated by gpt-4o-mini

JW Rettberg, H Wigers - Open Research Europe, 2025

Can a language model trained largely on Anglo-American texts generate stories that are culturally relevant to other nationalities? To find out, we generated 11,800 stories - 50 for each of 236 countries – by sending the prompt “Write a 1500 word …


AI-Powered Real-Time Text Editor with Multilingual Translation and Speech Recognition

A Dwivedi, S Sahu, A Srivastava - 2024 IEEE 16th International Conference on …, 2024

This paper introduces a novel real-time collaborative text editor for revolutionizing information writing. We propose the development of an all-inclusive real-time collaborative text editor that does not require separate tools for information writing …


AI-Powered Sentiment Analytics in Banking: A BERT and LSTM Perspective.

MT Siddique, MJ Uddin, L Chambugong, AM Nijhum… - International …, 2025

In recent years, the banking industry has witnessed a surge in digital feedback channels, where customers regularly share their experiences and opinions. Extracting meaningful insights from this unstructured data is vital for enhancing …


AI-Powered Transcreation in Global Marketing: Insights from Iran

G Hassani, M Malekshahi, H Davari - ELOPE: English Language Overseas …, 2025

This study examines AI-powered transcreation’s role in improving cross-cultural brand communication. We employed GPT-3 to evaluate AI’s ability to enhance global marketing through improved translation and adaptation of brand messages …


AIsplaining: Generative AI explains linguistic identities to me

B Carbajal-Carrera - Australian Review of Applied Linguistics, 2025

… an analysis of undesirable content in the Common Crawl corpus. (Ed.),^(Eds.). Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language …


AIxcellent Vibes at GermEval 2025 Shared Task on Candy Speech Detection: Improving Model Performance by Span-Level Training

CR Thelen, PG Blaneck, T Bornheim, N Grieger… - … 2025 Conference on …, 2025

Positive, supportive online communication in social media (“candy speech”) has the potential to foster civility, yet automated detection of such language remains underexplored, limiting systematic analysis of its impact. We investigate how candy …


Aleph-Alpha-GermanWeb: Improving German-language LLM pre-training with model-based data curation and synthetic data generation

TF Burns, L Parcalabescu, S Wäldchen, M Barlow… - arXiv preprint arXiv …, 2025

… To curate the Common Crawl data, we applied a pipeline similar to (but which we show can perform better than) FineWeb2. We then … By leveraging a combination of Common Crawl web data, FineWeb2, and synthetic data conditioned on organic …


Algorithmic bias: sexualized violence against women in GPT-3 models

… Common Crawl was the main contributor within the GPT-3 training dataset, and at the time of writing is the largest non-curated web corpus … CommonCrawl has been shown to present several types of explicit and abusive content regardless of filtering …


Align-then-Slide: A complete evaluation framework for Ultra-Long Document-Level Machine Translation

J Guo, D Wei, Y Luo, X Chen, Z Wu, H Yang, H Shang… - arXiv preprint arXiv …, 2025

… Our bilingual data originate from CommonCrawl. We first randomly sampled 100 document pairs that contained both source and target texts. After rule-based filtering to remove poorly aligned samples, professional translators selected the 50 highest-quality …


Aligning LLMs for Multilingual Consistency in Enterprise Applications

A Agarwal, H Meghwani, HL Patel, T Sheng, S Ravi… - arXiv preprint arXiv …, 2025

Large language models (LLMs) remain unreliable for global enterprise applications due to substantial performance gaps between high-resource and mid/low-resource languages, driven by English-centric pretraining and internal reasoning biases. This …


All is Not Lost: LLM Recovery without Checkpoints

N Blagoev, O Ersoy, LY Chen - arXiv preprint arXiv:2506.15461, 2025

Training LLMs on decentralized and wimpy computation nodes, eg, multiple on-spot instances, lowers the training cost and enables model democratization. The inevitable challenge here is the churn of nodes due to failures and the operator's …


AlphaDecay: Module-wise Weight Decay for Heavy-Tailed Balancing in LLMs

D He, A Jaiswal, S Tu, L Shen, G Yuan, S Liu, L Yin - arXiv preprint arXiv:2506.14562, 2025

… All experiments employ the C4 dataset [32], a rigorously processed subset of Common Crawl widely adopted for language model pretraining. Our experimental design incorporates two key components: (1) a non-repeating data regime with …


Am I Blue or Is My Hobby Counting Teardrops? Expression Leakage in Large Language Models as a Symptom of Irrelevancy Disruption

B Köprü, M Mashal, Y Gurses, A Kadar, M Schmitt… - arXiv preprint arXiv …, 2025

… To analyse the expression leakage, we collect a benchmark dataset along with a scheme to automatically generate a dataset from free-form text from common-crawl In addition, we propose an automatic evaluation pipeline that correlates well with …


AmaSQuAD: A Benchmark for Amharic Extractive Question Answering

ND Hailemariam, B Guda, T Tefferi - arXiv preprint arXiv:2502.02047, 2025

This research presents a novel framework for translating extractive question-answering datasets into low-resource languages, as demonstrated by the creation of the AmaSQuAD dataset, a translation of SQuAD 2.0 into Amharic. The methodology …


An Analysis of Bias Towards Women in Large Language Models Using Likert Scale Evaluations

Closed-source large language models (LLMs) developed by large technology companies continue to grow in popularity. However, ethical conversations surrounding the safety of model outputs have been a prominent topic of discussion …


An Analysis of Chinese Censorship Bias in LLMs

M Ahmed, J Knockel, R Greenstadt - Proceedings on Privacy Enhancing …, 2025

… data” which likely includes the Common Crawl dataset [21, 26, 42]. In the case of OpenAI, GPT 3 was trained on the Common Crawl dataset so it is likely that the succeeding models were also [12]. We did an analysis of the Common Crawl …


An Analysis of Cross-Lingual Natural Language Processing for Low-Resource Languages

V Naik, K Rajeswari, K Jadhav, A Rahalkar - Proceedings of Tenth International Congress on …

This study examines cross-lingual natural language processing (NLP) techniques to address the challenges of developing conversational AI systems for lowresource languages. These languages often lack extensive linguistic resources such as large-scale …


An Efficient Construction Method for Matrix Decomposition-Based Natural Language Processing Models in Low-Dimensional Embedding Space

B Liu - J. COMBIN. MATH. COMBIN. COMPUT, 2025

Natural language processing (NLP) is developing very rapidly in the field of artificial intelligence, and has become an important direction in the development of computer science field and artificial intelligence industry. In this paper, in order to realize the …


An Efficient Private GPT Never Autoregressively Decodes

Z Li, Y Guan, K Yang, Y Feng, N Liu, Y Yu, J Leng… - arXiv preprint arXiv …, 2025

The wide deployment of the generative pre-trained transformer (GPT) has raised privacy concerns for both clients and servers. While cryptographic primitives can be employed for secure GPT inference to protect the privacy of both parties, they …


An ensemble classification model for improved performance of phishing detection system

M Sahoo, S Samanta, S Ghosh - International Journal of Information and Computer …, 2025

Individuals and organisations are at risk of money losses and data compromise from phishing attempts. Traditional rule-based phishing detection methods fail to keep up with attacker strategies. The need for more advanced and adaptive phishing …


An Ensemble Machine Learning With Feature Selection Methods for Detecting Phishing Attacks

… Phishing websites were obtained from Open Phish and Phish Tank, while legitimate websites were obtained from Alexa and Common Crawl. Next, data preprocessing was conducted, which included removing missing values, eliminating …


An Evaluation of N-Gram Selection Strategies for Regular Expression Indexing in Contemporary Text Analysis Tasks

L Zhang, S Deep, JM Patel, K Sankaralingam - arXiv preprint arXiv:2504.12251, 2025

… Since we could not locate the original dataset, we constructed a similar dataset using web pages from 2013 stored in Common Crawl [9]. We chose 2013 data because it is relatively close to 1999, ensuring that most of the regexes constructed …


An Expanded Massive Multilingual Dataset for High-Performance Language Technologies

L Burchell, O de Gibert, N Arefyev, M Aulamo, M Bañón… - arXiv preprint arXiv …, 2025

Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual …


An Explainable Artificial Intelligence Text Classifier for Suicidality Prediction in Youth Crisis Text Line Users: Development and Validation Study

J Thomas, A Lucht, J Segler, R Wundrack, M Miché… - JMIR Public Health and …, 2025

… This multilingual model, trained on 2.5 TB of CommonCrawl data in 100 languages, tokenizes and encodes input text into a 768-dimensional embedding. Each message is embedded separately and then attached to an array of embedded …


An Investigation of the Morpho-Syntactic and Syntactic Analyses of Dialectal Arabic

This dissertation provides an investigation of the linguistic characteristics of dialects of Arabic and their variation. The majority of studies on dialects of Arabic are on phonological and lexical variation. In this dissertation, I focus on the syntactic …


An MIT Exploration of Generative AI â¢ From Novel Chemicals to Opera

S Longpre, R Mahari, N Obeng-Marnu, W Brannon…

… Common Crawl. For pretraining text data, models rely predominantly on the Common Crawl.86 For instance, 60 percent of GPT-3 training data is from Common Crawl.87 This resource provides URLs, scrape times, and the raw documents …


An NLP-driven e-learning platform with LLMs and graph databases for personalised guidance

G Dobriţa, SV Oprea, A Bâra - Connection Science, 2025

Information is ubiquitously available at our fingertips, transforming the way we learn, work and engage with the world around us. The challenge is not just accessing data but discerning its relevance and utility. This constant flow of information demands …


An Outlook on the Opportunities and Challenges of Multi-Agent AI Systems

F Tian, A Luo, J Du, X Xian, R Specht, G Wang, X Bi… - arXiv preprint arXiv …, 2025

Multi-agent AI systems (MAS) offer a promising framework for distributed intelligence, enabling collaborative reasoning, planning, and decision-making across autonomous agents. This paper provides a systematic outlook on the current …


An Overview of Large Language Models for Statisticians

W Ji, W Yuan, E Getzen, K Cho, MI Jordan, S Mei… - arXiv preprint arXiv …, 2025

Large Language Models (LLMs) have emerged as transformative tools in artificial intelligence (AI), exhibiting remarkable capabilities across diverse tasks such as text generation, reasoning, and decision-making. While their success has primarily been …


An Overview of Large Language Models: Architectures, Emergent Abilities, and Applications

… LLMs typically rely on massive text corpora drawn from web crawl data (eg, Common Crawl), books, and Wikipedia. Tokenization schemes such as Byte-Pair Encoding (BPE) [23] and SentencePiece [24] balance vocabulary size against …


An Unsupervised Approach Based on Attentional Neural Models for Aspect-Based Sentiment Classification

L Zampierin, F Frasincar - ACM SIGAPP Applied Computing Review, 2025

… In this research, we use the 300-dimensional GloVe word representations that were pre-trained on 42 billion tokens from Common Crawl [23]. This word embedding matrix contains representations for 1.9 million words. The reason for this choice is twofold. …


An Unsupervised Approach for Aspect-Based Sentiment Classification Using Attentional Neural Models

L Zampierin, F Frasincar - Proceedings of the 40th ACM/SIGAPP Symposium on …, 2025

… In this research, we use the 300-dimensional GloVe word representations that were pre-trained on 42 billion tokens from Common Crawl [22]. This word embedding matrix contains representations for 1.9 million words. The reason for this choice is twofold. …


An Unsupervised Integrated Framework for Arabic Aspect-Based Sentiment Analysis and Abstractive Text Summarization of Traffic Services Using Transformer …

A Alotaibi, F Nadeem - Smart Cities, 2025

… Word-level representations: We used FastText, an extension of Word2Vec, trained by Facebook utilizing Common Crawl and Wikipedia information employing CBOW architecture [33]. FastText enhances the original Word2Vec model by …


Analysing and Mitigating Classification Bias for Text-based Foundation Models

The objective of text classification is to categorise texts into one of several pre-defined classes. Text classification is a standard natural language processing (NLP) task with various applicability in many domains, such as analysing the evolving …


Analysis and Simplification of Privacy Policy Documents at Scale

… We used Common Crawl,2 described below, to gather seed URLs to privacy policies on the web. We filtered the Common Crawl URLs to … The Common Crawl Foundation has been releasing large monthly internet web crawls along with their …


Analysis of Indic Language Capabilities in LLMs

A Vaidya, T Prabhakar, D George, S Shah - arXiv preprint arXiv:2501.13912, 2025

… Common Crawl: This is a collection of web archives consisting of terabytes of data. Common Crawl is a nonprofit organization that crawls the web and freely provides its archives … In Table 2 we list the percentage distribution of Indian languages in …


Analysis of Online Interviews: A Framework for Assessing Confidence, Politeness and Emotions Portrayed

A Nair, A Kamath, K Mehta, R Karani - SN Computer Science, 2025

It can be tough to understand how a person’s words and tone influence the interviewers while conducting interviews online. Candidates are required to communicate confidently and professionally. Both verbal and nonverbal aspects of …


Anatomy of DNS: Investigating Vulnerabilities and Countermeasures from Clients to Authoritative Servers

The Domain Name System (DNS) is a critical component of the Internet infrastructure, responsible for translating human-readable domain names into IP addresses. However, DNS has been shown to be vulnerable to various attacks, including traffic …


And Plato met ChatGPT: an ethical reflection on the use of chatbots in scientific research writing, with a particular focus on the social sciences

R Calderon, F Herrera - Humanities and Social Sciences Communications, 2025

This interdisciplinary paper analyzes the use of Large Language Models based chatbots (LLM-chatbots), with ChatGPT the most known exponent, in scientific research writing. By interacting with LLM-chatbots, researchers could reduce efforts …


ANELIA

* Школата на ПУ Паисий Хилендарски в Компютърната лингвистика от края на 1980-те и 1990-те и след това–морфологичен анализ и други видове разбор и моделиране на българския език, лексикология и лексикография, машинно …


Anglocentric standards in universal text and speech processing

Even as early as the mid-20th century, researchers like Warren Weaver envisioned designing machines that could understand any language, thereby facilitating seamless communication across languages and cultures. Language technology …


Anti-Regulatory AI: How" AI Safety" is Leveraged Against Regulatory Oversight

RJ Yew, B Judge - arXiv preprint arXiv:2509.22872, 2025

AI companies increasingly develop and deploy privacy-enhancing technologies, bias-constraining measures, evaluation frameworks, and alignment techniques -- framing them as addressing concerns related to data privacy, algorithmic fairness, and AI safety. This …


APCache: An Adaptive Postings Cache in Heterogeneous Memory for Storage-Resident Search Indices

… For the large Common Crawl dataset, we observe consistent results across several repetitions of the same experiment, ie, the standard deviation is negligible. We run each query experiment twice, taking the results of the second repetition. We …


Apertus: Democratizing Open and Compliant LLMs for Global Language Environments

A Hernández-Cano, A Hägele, AH Huang, A Romanou… - arXiv preprint arXiv …, 2025

We present Apertus, a fully open suite of large language models (LLMs) designed to address two systemic shortcomings in today's open model ecosystem: data compliance and multilingual representation. Unlike many prior models that release …


Application of Artificial Intelligence Technology in Gaming NPC and Existing Problems

W Wan - Proceedings of the 2025 3rd International Conference …, 2025

… Therefore, in the data resource collection stage, it is necessary to download data resources from large-scale corpus in public domains such as ACL anthology corpus and Common Crawl, and crawl the massive tweets or Weibo data published by …


APPLIED LINGUISTICS DRIVEN LARGE LANGUAGE MODEL FOR SARCASM RECOGNITION ON SOCIAL MEDIA CORPORA

AM ALASHJAEE, A ALSHAMMARI, MSA ALZAIDI… - Fractals, 2025

… Facebook researchers developed FastText, a word representation tool featuring a widespread lexicon of 2 million words obtained from Common Crawl, providing supervised and unsupervised modes. Every word represents a 300-D vector space …


Applying Artificial Intelligence in Translation

Names: Walter, Katharina editor| Agnetta, Marco editor Title: Applying artificial intelligence in translation: possibilities, processes and phenomena/edited by Katharina Walter and Marco Agnetta. Description: New York, NY: Routledge, 2026 …


Applying Language Models To Patient Health Records: Acronym Expansion, Long Document Classification and Explainable Predictions

The health industry is experiencing a digital transformation, with Electronic Health Records (EHRs) becoming central repositories for an ever-growing volume of patient data. While EHR clinical notes offer rich, detailed insights into patient …


Applying Word Embeddings for Lithuanian Morphology: The Case of Adjectival Participles

This paper presents how word embeddings were used to identify adjectival Lithuanian participles. Although traditionally considered to be a form of a verb, participles in the Lithuanian language also have the characteristics of adjectives …


Approaches to Epistemic Risk in Generative and General-Purpose AI

Generative and general-purpose AI systems stand poised to reshape longstanding information infrastructures and professions, ranging from search to social media to online journalism. Yet questions surrounding subtle biases, misinforming output …


Arabic Cyberbullying Detection: A Comprehensive Review of Datasets and Methodologies

H Aljalaoud, K Dashtipour, A AI_Dubai - IEEE Access, 2025

The freedom of speech in online spaces has substantially promoted engagement on social media platforms, where cyberbullying has emerged as a significant consequence. While extensive research has been conducted on cyberbullying …


Arabic Language Characteristics that Make its Automatic Processing Challenging

… April 2024, this is the largest Arabic corpus to date, compiled from common crawl Web Extracted Text (WET) files. It has been rigorously cleaned and de-duplicated to ensure data quality and provides a substantial resource for training authentic Arabic …


Arabic Language Processing

This volume constitutes the refereed proceedings of the 8th International Conference on Arabic Language Processing (ICALP 2023), formerly known as CITALA. The conference, initially scheduled for 2023, was postponed due to …


Arabic Sentiment Analysis Leveraging Hybrid Word Embeddings with Deep Learning Techniques

A Alharbi, N Sharma, F Hussain - International Conference on Advanced Information …, 2025

The quality of word representation is essential for achieving high performance in various Natural Language Processing (NLP) tasks. This study investigates the impact of pre-trained word embeddings on sentiment classification of Arabic text …


Architectural Deep Dive into Large Language Models

A Ghabi, H Hamam - Generative AI and Large Language Models …, 2025

This chapter presents an in-depth exploration of Large Language Models (LLMs), a cornerstone in the field of artificial intelligence and natural language processing. Beginning with an introduction to the fundamental concepts of language models, we …


Are Large Language Models for Education Reliable for All Languages?

V Gupta, SP Chowdhury, V Zouhar, D Rooein…

Large language models (LLMs) are increasingly being adopted in educational settings. These applications expand beyond English, though current LLMs remain primarily Englishcentric. In this work, we ascertain if their use in education settings in …


Are LLMs (Really) Ideological? An IRT-based Analysis and Alignment Tool for Perceived Socio-Economic Bias in LLMs

J Wachter, M Radloff, M Smolej, K Kinder-Kurlanda - arXiv preprint arXiv:2503.13149, 2025

We introduce an Item Response Theory (IRT)-based framework to detect and quantify socioeconomic bias in large language models (LLMs) without relying on subjective human judgments. Unlike traditional methods, IRT accounts for item …


Are Multilingual Language Models an Off-ramp for Under-resourced Languages? Will we arrive at Digital Language Equality in Europe in 2030?

G Rehm, A GrĂźtzner-Zahn, F Barth - arXiv preprint arXiv:2502.12886, 2025

… In addition, except for ROOTS, all data sets are based on CommonCrawl dumps. Even ROOTS, although trying to gather data from other sources, had to complement their data with a subset of OSCAR, which is also based on CommonCrawl. These …


Are We on the Right Way for Assessing Document Retrieval-Augmented Generation?

W Shen, M Wang, Y Wang, D Chen, J Yang, Y Wan… - arXiv preprint arXiv …, 2025

… 2024), and the CommonCrawl corpus. Scanned content presents greater challenges for models due to defects introduced by the document … 2023), augmented with multilingual slides from the Commoncrawl corpus. Collected slides …


Artificial intelligence and security: some reflections concerning the freedom of expression, information and democracy

F Fusco - International Journal of Electronic Security and Digital …, 2025

In contemporary society, the role of information in socio-economic development is increasing across domains such as policy, business, technology, and society. Among sources of information, news holds significant sway in shaping public opinion …


ARTIFICIAL INTELLIGENCE AS A FILTER AND AS A PHILTER1

… However, it is known that part of the data often originates from the NGO Common Crawl that maintains copies of large part of internet content available for download and further use. GPT models are trained on part of that data, in addition to Wikipedia …


Artificial intelligence development and policy landscape

G Gensler - The Economic Consequences of the Second Trump …

Artificial intelligence (AI) is one of the most transformative technologies of our times. As it further takes on pattern recognition, decision making, content generation, and complex reasoning, it will continue to create efficiencies and innovations across the …


Artificial intelligence in qualitative analysis: a practical guide and reflections based on results from using GPT to analyze interview data in a substance use program

… The GPT-4 (current version) was trained on a vast and diverse dataset that include Common Crawl, Wikipedia, books, and a selection of high-quality licensed datasets, as well as data from various web sources that encompass a wide range of …


Assessing BERT-based models for Arabic and low-resource languages in crime text classification

NK Al-harbi, M Alghieth - PeerJ Computer Science, 2025

The bidirectional encoder representations from Transformers (BERT) has recently attracted considerable attention from researchers and practitioners, demonstrating notable effectiveness in various natural language processing (NLP) tasks, including …


Assessing Bias in AI Chatbot Responses

AI communication in the form of chatbots has brought about a new paradigm of communication and service delivery through the use of large language models (LLMs) like GPT. However, as these technologies are applied in daily life, questions about …


Assessing Gender Bias of Pretrained Bangla Language Models in STEM and SHAPE Fields

NMK Arnob, S Mahmud, AT Wasi - Proceedings of the 6th Workshop on Gender Bias …, 2025

Gender bias continues to shape societal perceptions across both STEM (Science, Technology, Engineering, and Mathematics) and SHAPE (Social Sciences, Humanities, and the Arts for People and the Economy) domains. While existing …


Assessing the Agreement Competence of Large Language Models

AT García, L Wanner - Proceedings of the Eighth International Conference on …, 2025

While the competence of LLMs to cope with agreement constraints has been widely tested in English, only a very limited number of works deals with morphologically rich (er) languages. In this work, we experiment with 25 mono-and multilingual LLMs …


Assessing the Role of Data Quality in Training Bilingual Language Models

S Seto, M ter Hoeve, M de Seyssel, D Grangier - arXiv preprint arXiv:2506.12966, 2025

Bilingual and multilingual language models offer a promising path toward scaling NLP systems across diverse languages and users. However, their performance often varies wildly between languages as prior works show that adding more …


Assessing Transfer Learning's Impact on Deep Learning for Image Recognition and Natural Language Processing

… For instance, ImageNet is a common source for image recognition tasks, while datasets like Wikipedia or Common Crawl are often used for NLP tasks. • Target Task: The specific task that the pre-trained model is adapted to, which may involve a …


Assessing Variations in Open Datasets for Training Large Language Models: Biases and Benchmarking

V Koc - Baltic Multidisciplinary Research Letters Journal, 2025

Open datasets are critical to the development and training of large language models (LLMs). However, variations in dataset composition often introduce biases that can impact model performance and reliability. This Article investigates the nature and …


Asymmetric Semantic Search Using Multi-Dimensional Vector Text Data Representation

This paper aims to provide a comprehensive analysis of semantic search methodologies that leverage text embeddings and vector space models for semantics representation. Specifically, the objectives of this work are to: analyze the …


Attention-based chatbots for low-resource language processing: A comprehensive review

GC Uzoaru, II Ayogu, AC Onyeka, J Odii - SSR Journal of Engineering and …, 2025

… The process begins with pre-training on highresource languages, where models such as BERT, GPT, or mBERT learn linguistic patterns from large datasets like Wikipedia and Common Crawl[li]. These models develop generalized language …


AttentionDep: Domain-Aware Attention for Explainable Depression Severity Assessment

Y Ibrahimov, T Anwar, T Yuan, T Mutallimov… - arXiv preprint arXiv …, 2025

In today's interconnected society, social media platforms provide a window into individuals' thoughts, emotions, and mental states. This paper explores the use of platforms like Facebook, X (formerly Twitter), and Reddit for depression severity …


AttentionInfluence: Adopting Attention Head Influence for Weak-to-Strong Pretraining Data Selection

K Hua, S Wu, G Zhang, K Shen - arXiv preprint arXiv:2505.07293, 2025

Recently, there has been growing interest in collecting reasoning-intensive pretraining data to improve LLMs' complex reasoning ability. Prior approaches typically rely on supervised classifiers to identify such data, which requires labeling …


AutoClean: LLMs Can Prepare Their Training Corpus

X Shen, S Hu, X Zhang, X Han, X Meng, J Wei, Z Liu… - Proceedings of the 2025 …, 2025

… The data sourced from the Internet is often aggregated into datasets like Common Crawl, which presents significant quality variability and ne… We demonstrate the efficiency and effectiveness of AutoClean on both pre-training corpora such as …


AutoCurate: Automating Domain-Specific Dataset Curation for Large Language Models

… Other methods filter domain-specific data from public corpora (eg, CommonCrawl), but typically use static keyword filters or manual rules, … When working with petabytescale corpora (eg, the 300TB-scale CommonCrawl), this efficiency …


AutoGUI: Scaling GUI Grounding with Automatic Functionality Annotations from LLMs

H Li, J Chen, J Su, Y Chen, Q Li, Z Zhang - arXiv preprint arXiv:2502.01977, 2025

… To amass these trajectories, we utilize the latest Common Crawl repository as the data source for web UIs and Android Emulator for mobile UIs. Note that illegal websites and Apps are excluded manually from the sources to ensure no …


Automated Classification and Identification of Non-Functional Requirements in Agile-Based Requirements Using Pre-Trained Language Models

A Alhaizaey, M Al-Mashari - IEEE Access, 2025

Non-functional requirements (NFRs) are critical factors for software quality and success. A frequently reported challenge in agile requirements engineering is that NFRs are often neglected due to the focus on functional requirements (FRs) and the …


Automated detection of cryptocurrency investment scams at scale

The ecosystem of cryptocurrencies has grown and changed significantly since Bitcoin’s inception in 2008 (Nakamoto, 2008). Similarly, the number of people using cryptocurrencies as a means of investment, speculation and form of payment has …


Automated Semantic Labeling and Clustering of Product Claims and Ingredients Using Machine Learning: Leveraging LLMs to Automate Previously Time-Consuming …

This thesis is conducted at Knightec Group in Sweden for the client company Bintix. It is in the domain of unsupervised learning and natural language processing. The goal is to obtain a condensed list of consumer claims and ingredients used when …


Automated Speech Act Classification in Offensive German Language Tweets

M Plakidis, ELG Rehm - Abusive Language: Linguistic Resources, Methods and …, 2025

En matière de détection de discours de haine et de langage offensant, l’intégration des connaissances sur les actes de langage représente une voie de recherche encore peu explorée. Dans nos précédents travaux, nous avons analysé si la …


Automated User Story Generation in Requirements Elicitation using Fine-Tuned Large Language Models

In software development, generating user stories from requirements elicitation interviews is a critical yet time-consuming and subjective task. Traditional methods often rely heavily on human interpretation, which can introduce bias and limit …


Automatic Association of Quality Requirements and Quantifiable Metrics for Cloud Security Certification

J Bianchi, S Dong, L Petrillo, M Petrocchi - arXiv preprint arXiv:2503.09460, 2025

… The feature extractor chosen is FastText5, which is pre-trained on English texts from Wikipedia and Common Crawl. Data cleaning, such as removing stop words, is done before the feature vector computation to eliminate irrelevant information. Then …


Automatic Control With Human-Like Reasoning

Recent developments in language models have created new opportunities in air traffic control studies. The current focus is primarily on text and language-based use cases. However, these language models may offer a higher potential impact in the …


Automatic Distractor Generation with Paradigmatic Relation for English Vocabulary Tests

H Setiawan, I Hidayah, SS Kusumawardani - … on Smart Computing, IoT and Machine …, 2025

Automatic distractor generation (ADG) is a computer-based system that generates incorrect answers for multiple-choice questions to assist teachers to create educational assessments. It has been implemented in various subjects and …


AUTOMATIC GRADING OF SHORT ANSWERS

Developing effective Automatic Short Answer Grading (ASAG) for e-learning environments is challenging. We consider scoring a text-constructed student answer compared to a teacher-provided reference answer. In this thesis, we address three …


Automatic Text Simplification for Lithuanian: Transforming Administrative Texts into Plain Language

J Mandravickaitė, E Rimkienė, DK Kapkan… - Mathematics, 2025

In this study, we present the results of experiments on text simplification for the Lithuanian language, where we aim to simplify administrative-style texts to the Plain Language level. We selected mT5, mBART, and LT-Llama-2 as the foundational …


Automatic Text Summarization for Hindi Language Using Word Embeddings: A Critical Review

SA Khan, M Mudasir, HA Khanday - … Conference on Cognitive Robotics and Intelligent …, 2025

… 5) XLM-R [22]: 100 languages were used to train XLM-R (a transformer-based masked language model) with more than two terabytes of filtered data from CommonCrawl It performs significantly better than mBERT on a range of cross-lingual benchmarks. …


Automatic Urdu Grammar Error Correction: Harnessing the Power of Head Pruning for LLMs

M Hussain, W Ramay, MH Akbar, MN Zafar, T Rashid - International Journal of …, 2025

… In our proposed approach T5 leverages much larger CommonCrawl web data across languages and uses a larger 250k Sentence-Piece vocabulary to improve sub word coverage of Urdu Text. Beyond masked language modeling, T5 is pre-trained …


Automatic XPath generation agents for vertical websites by LLMs

J Huang, J Song - Journal of King Saud University Computer and …, 2025

… 2022a), which is trained on Products and Movies data from Common Crawl Footnote 1 using approximately 200,000 annotated samples, experiences a sharp decline in EM score from around 75 to below 20 when applied to extracting date and …


Automation of ETL Pipelines in DataStage

… + English Wikipedia 2,500M words)[16] instead GPT leverage web-scale data , the original GPT is trained in BookCorpus, GPT-2 is trained on 40+GB of WebText and GPT-3 (175 billion parameters) is trained on around 300 billion tokens from …


AutoMixer: Checkpoint Artifacts as Automatic Data Mixers

E Chang, Y Li, P Huber, D Kant, Y Shi, V Chandra - arXiv preprint arXiv:2506.21910, 2025

In language model training, it is desirable to equip models with capabilities from various tasks. However, it is not clear how to directly obtain the right data mixtures for these capabilities as the relationship between data and tasks is difficult to be …


AutoSchemaKG: Autonomous Knowledge Graph Construction through Dynamic Schema Induction from Web-Scale Corpora

J Bai, W Fan, Q Hu, Q Zong, C Li, HT Tsang, H Luo… - arXiv preprint arXiv …, 2025

… 2024) pretraining corpus across three diverse subsets, English Wikipedia, paper abstracts from Semantic Scholar, and 3% of Common Crawl data, we construct the ATLAS family of knowledge graphs (ATLAS-Wiki, ATLAS-Pes2o, and ATLAS-CC) …


AutoTestForge: A Multidimensional Automated Testing Framework for Natural Language Processing Models

H Xing, C Tian, L Zhao, Z Ma, WS Wang, N Zhang… - arXiv preprint arXiv …, 2025

In recent years, the application of behavioral testing in Natural Language Processing (NLP) model evaluation has experienced a remarkable and substantial growth. However, the existing methods continue to be restricted by the requirements …


Avoiding Catastrophe Through Intersectionality in Global AI Governance

L McCrory - Artificial Intelligence, 2025

The Digital Policy Hub at CIGI is a collaborative space for emerging scholars and innovative thinkers from the social, natural and applied sciences. It provides opportunities for undergraduate and graduate students and post-doctoral and …


Babel: Open Multilingual Large Language Models Serving Over 90% of Global Speakers

Y Zhao, C Liu, Y Deng, J Ying, M Aljunied, Z Li, L Bing… - arXiv preprint arXiv …, 2025

… To further analyze Babel’s performance across languages, we categorized them into highresource and low-resource languages based on their scores in Crawl (2025), a statistical measure derived from Common Crawl’s monthly archives that reflects …


Bachelor's Thesis Computing Science

T van der Straaten, D Hiemstra, TM Heskes - 2025

… The Open Web Index (OWI) [11] research aims to address this problem by improving the ability to build and maintain an index for a general collection of the size of the common crawl. Conceptually, having an open index that allows anyone to …


Back-translation effects on static and contextual word embeddings for topic classification embedding in classification tasks

This study investigates the impact of back-translation on topic classification, comparing its effects on static word vector representations (FastText) and contextual word embeddings (RoBERTa). Our objective was to determine whether back-translation …


Balancing Computation Load and Representation Expressivity in Parallel Hybrid Neural Networks

MM Moradi, W Ahmed, S Wen, S Mudur, W Zhang… - arXiv preprint arXiv …, 2025

Attention and State-Space Models (SSMs) when combined in a hybrid network in sequence or in parallel provide complementary strengths. In a hybrid sequential pipeline they alternate between applying a transformer to the input and then feeding …