BALANCING SPEED AND PERFORMANCE WITH LAYER FREEZING STRATEGIES FOR TRANSFORMER MODELS

B Kairatuly, A Shomanov - Scientific Journal of Astana IT University, 2025

In this paper, we evaluated different approaches to freezing BERT-base layers and analyzed their impact on the quality and speed of training in the task of named entity recognition in two languages. Layer freezing is an optimization technique in deep …


BAMG: A Block-Aware Monotonic Graph Index for Disk-Based Approximate Nearest Neighbor Search

H Li, J Xu - arXiv preprint arXiv:2509.03226, 2025

Approximate Nearest Neighbor Search (ANNS) over high-dimensional vectors is a foundational problem in databases, where disk I/O often emerges as the dominant performance bottleneck at scale. Existing graph indexing solutions for disk-based …


BanglaByT5: Byte-Level Modelling for Bangla

P Bhattacharyya, A Bhattacharya - arXiv preprint arXiv:2505.17102, 2025

Large language models (LLMs) have achieved remarkable success across various natural language processing tasks. However, most LLM models use traditional tokenizers like BPE and SentencePiece, which fail to capture the finer nuances of a …


Banzhida: Advancing Large Language Models for Tibetan with Curated Data and Continual Pre-Training

L Pan, B Xiong, L Yang, R Jin, S Zhang, Y Chen, L Shi… - arXiv preprint arXiv …, 2025

… However, we observe that a portion of the web-based data originates from different snapshots of Common Crawl, which may introduce redundancy. To address this, we appliy deduplication to remove duplicate content across time. A detailed …


Baseer: A Vision-Language Model for Arabic Document-to-Markdown OCR

K Hennara, M Hreden, MM Hamed, A Bastati, Z Aldallal… - arXiv preprint arXiv …, 2025

… The foundation for this synthetic data is a corpus of markdown-formatted documents, which were downloaded and filtered from the Common Crawl archive using a methodology analogous to our previously released dataset‡. To ensure the …


Basic Reading Distillation

Z Zhou, S Miao, X Duan, H Yang, M Zhang - Proceedings of the 63rd Annual Meeting …, 2025

… We use a subset of CommonCrawl (CC-100) corpus, which is usually included in LLMs pretraining, as the education resource to conduct the basic reading education. The whole education process contains two stages. In the first stage, for each …


Basics of Machine Learning

P Wulff, M Kubsch, C Krist - Applying Machine Learning in Science Education …, 2025

This chapter presents a historical brief of artificial intelligence and machine learning as well as an overview of conceptual basics of how ML works, alongside examples. Different approaches to ML are reviewed and the challenges of applying ML in …


Batch Query Processing and Optimization for Agentic Workflows

J Shen, N Wadlom, Y Lu - arXiv preprint arXiv:2509.02121, 2025

… Beyond the fact that LLMs are already costly, analytics tasks often process massive datasets, from public corpora like Common Crawl in PBs [14] to enterprise telemetry and logs spanning multiple petabytes [51]. Retrieval-Augmented Generation …


Behavior as a Modality

… Consider the Common Crawl project (https://commoncrawl.org), one common source of data included in most language models. It produces more than 20TB of text per month sampled from random web pages across the internet. T5 and other …


BehaviorBox: Automated Discovery of Fine-Grained Performance Differences Between Language Models

L Tjuatja, G Neubig - arXiv preprint arXiv:2506.02204, 2025

Language model evaluation is a daunting task: prompts are brittle, corpus-level perplexities are vague, and the choice of benchmarks are endless. Finding examples that show meaningful, generalizable differences between two LMs is …


BEING PROFILED: COGITAS ERGO SUM

… CommonCrawl 2012 corpus found that the majority of sites contain trackers, even … 4 Consider a new service provided by The Common Crawl Foundation, http://commoncrawl.org/, or, alterna- …


BelarusianGLUE: Towards a Natural Language Understanding Benchmark for Belarusian

M Aparovich, V Harytskaya, V Poritski, O Volchek… - Proceedings of the 63rd …, 2025

In the epoch of multilingual large language models (LLMs), it is still challenging to evaluate the models’ understanding of lower-resourced languages, which motivates further development of expert-crafted natural language understanding benchmarks …


BenchING: A Benchmark for Evaluating Large Language Models in Following Structured Output Format Instruction in Text-Based Narrative Game Tasks

P Taveekitworachai, MF Dewantoro, Y Xia… - IEEE Transactions on …, 2025

This paper presents BenchING, a new benchmark for evaluating large language models (LLMs) on their ability to follow structured output format instructions in text-based procedural content generation (PCG) tasks. The ability to condition LLMs to output in …


Benchmark Creation for Aspect-Based Sentiment Analysis in Low-Resource Odia Language and Evaluation through Fine-Tuning of Multilingual Models

L Dewangan, ZA Sayeed, C Maurya - … of the 31st International Conference on …, 2025

… 2020), a multilingual version of RoBERTa, pre-trained on 2.5TB of CommonCrawl data which supports 100 languages including Odia. In our experiment, we specifically fine-tune the xlm-roberta-base variant4, which has 279 million …


Benchmark of stylistic variation in LLM-generated texts

J Milička, A Marklová, V Cvrček - arXiv preprint arXiv:2509.10179, 2025

This study investigates the register variation in texts written by humans and comparable texts produced by large language models (LLMs). Biber's multidimensional analysis (MDA) is applied to a sample of human-written texts and …


Benchmarking bias in embeddings of healthcare AI models: using SD-WEAT for detection and measurement across sensitive populations

M Gray, L Wu - BMC Medical Informatics and Decision Making, 2025

… More specifically, the GloVe model utilized in this study was trained on Common Crawl data, which gives it a broader range of data sources than the BERT model trained on the Toronto Book Corpus and English Wikipedia. This, in turn, could result …


Benchmarking Debiasing Methods for LLM-based Parameter Estimates

NA de Pieuchon, A Daoud, CT Jerzak, M Johansson… - arXiv preprint arXiv …, 2025

… The corpus consists of English-language online biographies from the Common Crawl, annotated with self-identified binary gender and occupation labels (with 28 categories), enabling analysis of implicit gender biases in textual representations …


Benchmarking MLLM-based Web Understanding: Reasoning, Robustness and Safety

J Liu, J Xiao, W Tang, W Wang, Z Wang, M Zhang, S Yu - arXiv preprint arXiv …, 2025

Multimodal large language models (MLLMs) are increasingly positioned as AI collaborators for building complex web-related applications like GUI agents and front-end code generation. However, existing benchmarks largely emphasize visual …


Benchmarking State of the Art Website Embedding Methods for Effective Processing and Analysis in the Public Sector

J Gerber, J Saxer, B Kreiner, A Weiler - 2025

The ability to understand and process websites is crucial across various domains. It lays the foundation for machine understanding of websites. Specifically, website embedding proves invaluable when monitoring local government websites within …


Benchmarking Synonym Extraction Methods in Domain-Specific Contexts

… Pre-trained GloVe embeddings are loaded from pretrained GloVe model which was trained on 42 billion tokens from the Common Crawl dataset that resulted in creating 1.9 million vocabularies and their 300-dimensional vectors. fastText. Also …


BERT-Based Automation of Job Safety Analysis for industrial workplace compliance.

Our center focuses on understanding the critical role of Process Safety Management (PSM), emphasizing the safety of both personnel and control systems through software that detects hazards such as fires and unauthorized movements within …


BERT-Based Intrusion Detection System for RF Jamming Attacks in Vehicular Network

As vehicular networks continue to evolve toward increased connectivity and autonomy, they become more vulnerable to cybersecurity threats, particularly Radio Frequency (RF) jamming attacks that can severely disrupt communication systems …


BERT-based Models for Keyword Extraction from Arabic Scientific Articles

B Babayigit, H Sattuf, M Abubaker - ACM Transactions on Asian and Low-Resource …, 2025

Keywords at the beginning of research articles are crucial for conveying the content and main ideas of academic works. They serve as essential tools for researchers to efficiently search for relevant topics. The integration of traditional natural language …


BERT-PhishFinder: A Robust Model for Accurate Phishing URL Detection with Optimized DistilBERT

A Aljofey, SA Bello, J Lu, C Xu - IEEE Transactions on Dependable and Secure …, 2025

Phishing URL detection has become a critical challenge in cybersecurity, with existing methods often struggling to maintain high accuracy while generalizing across diverse datasets. In this paper, we introduce BERT-PhishFinder, a novel and …


BERTWEETRO: PRE-TRAINED LANGUAGE MODELS FOR ROMANIAN SOCIAL MEDIA CONTENT

DC Neagu - Studia Universitatis Babes Bolyai-Oeconomica, 2025

… As a first example we can mention the Common Crawl1 dataset which is a collection of web-scraped texts from the digital space and includes a variety of different languages and topics. Another important resource is the OpenAI WebText …


Beyond Architectures: Evaluating the Role of Contextual Embeddings in Detecting Bipolar Disorder on Social Media

K Hasan, J Saquer - arXiv preprint arXiv:2507.14231, 2025

Bipolar disorder is a chronic mental illness frequently underdiagnosed due to subtle early symptoms and social stigma. This paper explores the advanced natural language processing (NLP) models for recognizing signs of bipolar disorder based …


Beyond Buzzwords: The Development of Large Language Models and Their Use in Advertising and Strategic Communication Research

V Paltaratskaya, A Ji, P Mazumdar, K Wise - Journal of Current Issues & Research in …, 2025

This paper explores the application of large language models (LLMs) such as ChatGPT in advertising and communication research, emphasizing their foundational processes, research applications, and ethical implications. We provide …


BEYOND CHATBOTS: IMPROVING INTELLIGENT TUTORING SYSTEMS WITH BETTER DATA AND ASSESSMENTS

X Li, C Fadel, R Zaki - INTED2025 Proceedings, 2025

… This dataset has been refined and extracted from over 200 billion HTML files in the Common Crawl, resulting in a collection of 6.3 million documents with a total of 14.7 billion tokens. OpenWebMath is designed for pretraining and fine-tuning large …


Beyond Decoder-only: Large Language Models Can be Good Encoders for Machine Translation

Y Luo, T Zheng, Y Mu, B Li, Q Zhang, Y Gao, Z Xu… - arXiv preprint arXiv …, 2025

… Note that due to the extensive bilingual data in the En-De CommonCrawl corpus, we only sampled a portion and merged it with other data to create a dataset of 50M. For En-Cs, we excluded the CzEng 2.0 dataset due to licensing issues. …


Beyond English: Evaluating Automated Measurement of Moral Foundations in Non-English Discourse with a Chinese Case Study

CY Cheng, SA Hale - arXiv preprint arXiv:2502.02451, 2025

This study explores computational approaches for measuring moral foundations (MFs) in non-English corpora. Since most resources are developed primarily for English, cross-linguistic applications of moral foundation theory remain limited. Using …


Beyond Facts: Evaluating Intent Hallucination in Large Language Models

Y Hao, H Yu, J You - arXiv preprint arXiv:2506.06539, 2025

When exposed to complex queries containing multiple conditions, today's large language models (LLMs) tend to produce responses that only partially satisfy the query while neglecting certain conditions. We therefore introduce the concept of …


Beyond Fixed Length: Bucket Pre-training is All You Need

Q Yang, Q Peng, H Liu, K Liu, B Qin, T Liu

Large Language Models (LLMs) have demonstrated exceptional performance across various tasks, with pre-training stage serving as the cornerstone of their capabilities. However, the conventional fixed-length data composition strategy for pre-training …


Beyond Reactive Safety: Risk-Aware LLM Alignment via Long-Horizon Simulation

C Sun, D Zhang, CX Zhai, H Ji - arXiv preprint arXiv:2506.20949, 2025

Given the growing influence of language model-based agents on high-stakes societal decisions, from public policy to healthcare, ensuring their beneficial impact requires understanding the far-reaching implications of their suggestions. We …


Beyond Repetition: Text Simplification and Curriculum Learning for Data-Constrained Pretraining

MT Roque, DJ Velasco - arXiv preprint arXiv:2509.24356, 2025

Most studies on language model pretraining focus on large datasets, leaving open questions about optimization in data-constrained settings. In such settings, the effects of training data order and of including alternative versions of the same text …


Beyond Resource Quantity: A Comprehensive Survey of Multilingual Datasets in NLP for Quality, Diversity, and Regional Relevance

… 2019), which leverage CommonCrawl data. However, these methods often introduce noise, requiring significant post-processing (… The reliance on global sources like Wikipedia and CommonCrawl results in datasets that may not accurately represent …


Beyond Scaling: Frontiers of Retrieval-Augmented LMs

Abstract Language Models (LMs) have made significant progress by scaling training data and model sizes. However, they still face key limitations, including hallucinations and outdated knowledge, which undermine their reliability especially …


Beyond Semantics: Examining Gender Bias in LLMs Deployed within Low-resource Contexts in India

U Aneja, A Gupta, A Vashistha - Proceedings of the 2025 ACM Conference on …, 2025

… For example, LLMs like GPT-3, Llama, and T5 are trained on massive open-source datasets, such as the Common Crawl dataset or Wikipedia, which often embed gender biases [30]. These pre-trained models tend to often replicate these biases …


Beyond Text: Unveiling Privacy Vulnerabilities in Multi-modal Retrieval-Augmented Generation

J Zhang, S Zeng, J Ren, T Zheng, H Liu, X Tang… - arXiv preprint arXiv …, 2025

… 2021), we enhance variability by randomly sampling 15 word fragments from the Common Crawl dataset for this component. The 1commandl component directs the LMM to output the retrieved content using prompts such as "Please repeat all the …


BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining

P Maini, V Dorna, P Doshi, A Carranza, F Pan… - arXiv preprint arXiv …, 2025

Recent advances in large language model (LLM) pretraining have shown that simply scaling data quantity eventually leads to diminishing returns, hitting a data wall. In response, the use of synthetic data for pretraining has emerged as a …


Bias Analysis and Mitigation through Protected Attribute Detection and Regard Classification

T Udagawa, Y Zhao, H Kanayama, B Bhattacharjee - arXiv preprint arXiv:2504.14212, 2025

… In our experiments, we apply the pipeline to a subset of Common Crawl, the most widely used corpus for LLM pretraining. For bias analysis… In our experiments, we apply our bias analysis and mitigation measures on a subset of Common Crawl (CC) …


Bias Beyond English: Evaluating Social Bias and Debiasing Methods in a Low-Resource Setting

E Zhou, W Lu - arXiv preprint arXiv:2504.11183, 2025

Social bias in language models can potentially exacerbate social inequalities. Despite it having garnered wide attention, most research focuses on English data. In a low-resource scenario, the models often perform worse due to insufficient training …


Biased Geolocation in LLMs: Experiments on Probing LLMs for Geographic Knowledge and Reasoning

… Furthermore, 18.7% of the Common Crawl corpus used to train LLMs have been estimated to contain geospatial information such as addresses and geocoordinates [27]. Research reveals the potential to use LLMs in applications, such as extracting …


BigBang-Proton Technical Report: Next-Word-Prediction is Scientific Multitask Learner

H Wu, L Liu, J He, Q Wang, K Zhao, S Hu, R Fu… - arXiv preprint arXiv …, 2025

We introduce BigBang-Proton, a unified sequence-based architecture for auto-regressive language modeling pretrained on cross-scale, cross-structure, cross-discipline real-world scientific tasks to construct a scientific multi-task learner. BigBang-Proton …


BigCharts-R1: Enhanced Chart Reasoning with Visual Reinforcement Finetuning

A Masry, A Puri, M Hashemi, JA Rodriguez, M Thakkar… - arXiv preprint arXiv …, 2025

… Common Crawl. To enhance visual diversity, we utilize the Mint-1T dataset (Awadalla et al.… 2024), which includes extensive PDF files covering numerous topics from Common Crawl1. We extract images from these documents and employ a rigorous …


Bilingual generated text detection through semantic and statistical analysis

C Min, R Zhang, J Liu - Intelligent Data Analysis, 2025

… XLM-RoBERTa is a pre-trained language model based on the Transformer architecture, a multilingual version of RoBERTa through pre-training on 2.5TB of filtered Common Crawl data containing 100 languages. It aims to enhance …


Binary classification for perceived quality of headlines and links on worldwide news websites, 2018-2024

A McCutcheon, TEA de Oliveira, A Zheleznov, C Brogly - arXiv preprint arXiv …, 2025

The proliferation of online news enables potential widespread publication of perceived low-quality news headlines/links. As a result, we investigated whether it was possible to automatically distinguish perceived lower-quality news headlines/links …


Biodiversity Monitoring at Scale with Foundation Models

… Unlike the public, centralized, and open-source web sources available to easily gather snapshots of internet text and image data such as Common Crawl [126], high-quality ecological observations tend to be non-standardized and non-centralized, with …


Biomed-Enriched: A Biomedical Dataset Enriched with LLMs for Pretraining and Extracting Rare and Hidden Content

R Touchent, N Godey, E de la Clergerie - arXiv preprint arXiv:2506.20331, 2025

We introduce Biomed-Enriched, a biomedical text dataset constructed from PubMed via a two-stage annotation process. In the first stage, a large language model annotates 400K paragraphs from PubMed scientific articles, assigning scores for …


BitNet: 1-bit Pre-training for Large Language Models

H Wang, S Ma, L Ma, L Wang, W Wang, L Dong… - Journal of Machine …, 2025

The increasing size of large language models (LLMs) has posed challenges for deployment and raised concerns about environmental impact due to high energy consumption. Previous research typically applies quantization after pre-training …


BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity

C Song, W Zhao, X Han, C Xiao, Y Chen, Y Li, Z Liu… - arXiv preprint arXiv …, 2025

To alleviate the computational burden of large language models (LLMs), architectures with activation sparsity, represented by mixture-of-experts (MoE), have attracted increasing attention. However, the non-differentiable and inflexible routing …


BloomWise: Enhancing problem-solving capabilities of LLMs using Bloom's-Taxonomy-inspired prompts

The limited ability of Large Language Models (LLMs) in mathematics—a skill critical for solving complex problems—has garnered significant interest from the research community. Many approaches have employed in-context learning to improve LLMs’ …


Boundary-making practices: LLMs and an artifactual production of objectivity

… For instance, Common Crawl, a filtered subset of which is used in GPT-3, is heavily skewed toward English-language content, including 46% of the 2023 version (‘Common Crawl’ 2025). This English dominance perpetuates Western …


Break the Checkbox: Challenging Closed-Style Evaluations of Cultural Alignment in LLMs

M Kabir, A Abrar, S Ananiadou - arXiv preprint arXiv:2502.08045, 2025

A large number of studies rely on closed-style multiple-choice surveys to evaluate cultural alignment in Large Language Models (LLMs). In this work, we challenge this constrained evaluation paradigm and explore more realistic, unconstrained …


Breaking Memory Limits: Gradient Wavelet Transform Enhances LLMs Training

Z Wen, P Luo, J Wang, X Deng, J Zou, K Yuan, T Sun… - arXiv preprint arXiv …, 2025

… The C4 English benchmark is a colossal, cleaned version of Common Crawl’s web crawl corpus based on the Common Crawl dataset. Includes 305GB of English-language text and is mainly intended to pretrain language models. The validation complexity is …


Bridging Gaps in Natural Language Processing for Yor\ub\'a: A Systematic Review of a Decade of Progress and Prospects

TA Jimoh, T De Wille, NS Nikolov - arXiv preprint arXiv:2502.17364, 2025

Natural Language Processing (NLP) is becoming a dominant subset of artificial intelligence as the need to help machines understand human language looks indispensable. Several NLP applications are ubiquitous, partly due to the myriads of …


Bridging Language Gaps: Advances in Cross-Lingual Information Retrieval with Multilingual LLMs

R Goworek, O Macmillan-Scott, EB Özyiğit - arXiv preprint arXiv:2510.00908, 2025

… Typical sources include Common Crawl and Wikipedia. The aim is knowledge acquisition and learning universal language structures. Pre-… Most pre-training data comes from web sources, particularly Common Crawl and Wikipedia. Crawled …


BRoverbs--Measuring how much LLMs understand Portuguese proverbs

TS Almeida, GK Bonás, JGA Santos - arXiv preprint arXiv:2509.08960, 2025

Large Language Models (LLMs) exhibit significant performance variations depending on the linguistic and cultural context in which they are applied. This disparity signals the necessity of mature evaluation frameworks that can assess their …


Building a Rich Dataset to Empower the Persian Question Answering Systems

M Yazdinejad, M Kaedi - arXiv preprint arXiv:2412.20212, 2024

… XLM-RoBERTa [40] is a relatively new and big interlingual language model based on RoBERTa and has been trained on 100 languages on CommonCrawl filtered by 2.5 TB. Unlike other XLM models, XLM-RoBERTa doesn’t need language …


Building AI Agents with LLMs, RAG, and Knowledge Graphs: A practical guide to autonomous and modern AI agents

Master LLM fundamentals to advanced techniques like RAG, reinforcement learning, and knowledge graphs to build, deploy, and scale intelligent AI agents that reason, retrieve, and act autonomously Key Features Implement RAG and knowledge …


Building Data Infrastructure for Low-Resource Languages

SKK Luger, R Mosquera, PO Suarez - Proceedings of the Eighth Workshop on …, 2025

… ; a strategic collaboration with the Common Crawl Foundation to enhance web crawling capabil… We expect the best submissions to be incorporated in the stack used by Common Crawl, as … While Common Crawl already annotates their crawls …


Building High-Quality Datasets for Portuguese LLMs: From Common Crawl Snapshots to Industrial-Grade Corpora

T Sales Almeida, R Nogueira, H Pedrini - arXiv e-prints, 2025

TS Almeida, R Nogueira, H Pedrini - arXiv preprint arXiv:2509.08824, 2025

The performance of large language models (LLMs) is deeply influenced by the quality and composition of their training data. While much of the existing work has centered on English, there remains a gap in understanding how to construct …


Building Transformer-Based Conversational Agents Capable of Sentiment Detection and Human-Like Dialogue Generation

The rapid advancement of transformer-based architectures has significantly transformed the capabilities of conversational agents, enabling them to generate coherent, context-aware, and human-like dialogues. This paper explores the …


Building Trust in AI via Safe and Responsible Use of LLMs

Artificial Intelligence (AI), and more recently Generative AI technologies like large language models (LLMs), have become pervasive, influencing diverse areas of society and reshaping the way complex tasks are approached. The rapid evolution …


byteSizedLLM@ DravidianLangTech 2025: Sentiment Analysis in Tamil Using Transliteration-Aware XLM-RoBERTa and Attention-BiLSTM

DP Manukonda, RG Kodali - Proceedings of the Fifth Workshop on Speech, Vision …, 2025

This study investigates sentiment analysis in code-mixed Tamil-English text using an Attention BiLSTM-XLM-RoBERTa model, combining multilingual embeddings with sequential context modeling to enhance classification performance. The model was …


Calibrated Semi-Supervised Models for Disaster Response based on Training Dynamics

K Gupta, N Gautam, T Sosea, D Caragea, C Caragea - Proceedings of the …, 2025

Despite advancements in semi-supervised learning (SSL) techniques that can be used when labeled data is limited, many SSL approaches still face challenges related to miscalibration. Calibration is crucial for ensuring the accuracy, reliability …


Can LLMs be used to Quantify the Emotional Salience of Text Statements using an Elo Rating System?

K Beaumont, M Oravec, H Emerson, I Penton-Voak…

… Mainstream LLMs, primarily trained on the CommonCrawl dataset, reflect biases towards English-speaking, younger males from what are sometimes referred to as WEIRD, or western, educated, industrial, rich, democratic, societies [4, 18, 19, 24, 31] …


Can News Predict the Direction of Oil Price Volatility? A Language Model Approach with SHAP Explanations

R Hashamia, F Maldonado - arXiv preprint arXiv:2508.20707, 2025

Financial markets can be highly sensitive to news, investor sentiment, and economic indicators, leading to important asset price fluctuations. In this study we focus on crude oil, due to its crucial role in commodity markets and the global economy …


Can Performant LLMs Be Ethical? Quantifying the Impact of Web Crawling Opt-Outs

D Fan, V Sabolčec, M Ansaripour, AK Tarun, M Jaggi… - arXiv preprint arXiv …, 2025

… While FineWeb-Edu is derived from CommonCrawl3—whose data dates back as far as 2008 and respects crawling opt-outs—these opt-outs are enforced only at the time of crawling. Since data owners can update their robots.txt file at any time …


Can Pre-training Indicators Reliably Predict Fine-tuning Outcomes of LLMs?

H Zeng, K Hui, H Zhuang, Z Qin, Z Yue, H Zamani… - arXiv preprint arXiv …, 2025

While metrics available during pre-training, such as perplexity, correlate well with model performance at scaling-laws studies, their predictive capacities at a fixed model size remain unclear, hindering effective model selection and development. To …


Can We Trust the Machine? LLMs Mimic Human Expected Utility Theory Violations and Its Impact on Decision and Negotiation Systems

Expected Utility Theory (EUT) has long served as a benchmark for rational decision-making, with well-documented human deviations in the form of framing effects, time inconsistency, and violations of independence and sequential rationality. In this …


Can You Feel It? Exploring the Emotional Profile of LLM Responses to Children's Queries

Agents based on Large Language Models (LLM) have introduced a new way of information seeking that could simplify the search process to suit children’s cognitive skills, as these agents often respond to natural language inquiries with easy-to-read …


Canada as a Champion for Public AI: Data, Compute and Open Source Infrastructure for Economic Growth and Inclusive Innovation

… Over the course of AI development, the data available on the internet — such as text on public websites crawled by nonprofits like Common Crawl — has been foundational to training AI models. Now, as that is no longer enough data to train …


Cancer Vaccine Adjuvant Name Recognition from Biomedical Literature using Large Language Models

H Rehana, J Zheng, L Yeh, B Bansal, NB Çam… - arXiv preprint arXiv …, 2025

… Llama takes advantage of publicly available datasets such as CommonCrawl, Wikipedia, and GitHub repositories, which makes the models powerful and compatible with open-source platforms. Llama models are available in various …


Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Framework

L Kopf, N Feldhus, K Bykov, PL Bommer, A Hedström… - arXiv preprint arXiv …, 2025

Automated interpretability research aims to identify concepts encoded in neural network features to enhance human understanding of model behavior. Current feature description methods face two critical challenges: limited robustness and the …


Capturing the Effects of Quantization on Trojans in Code LLMs

A Hussain, SAMK Zarkouei, MRI Rabin, MA Alipour… - arXiv preprint arXiv …, 2025

… The pretraining dataset used to generate Llama mostly comprise of web crawl data from the English CommonCrawl [19] and C4 [20] datasets (82%), along with data from Wikipedia, Github, StackExchange, ArXiv, Gutenberg, and Books3 [17] …


Caregiver-in-the-Loop AI: A Simulation-Based Feasibility Study for Dementia Task Verification

J Lai, D Black, K Beaton, B Ye, A Mihailidis - arXiv preprint arXiv:2508.18267, 2025

Caregivers of people living with dementia (PLwD) experience stress when verifying whether tasks are truly completed, even with digital reminder systems. Generative AI, such as GPT-4, may help by automating task verification through follow-up …


Causal Investigation of Tense Encoding in Multilingual Transformer

This thesis investigates how multilingual decoder-only transformers encode simple past, present, and future tenses across typologically diverse languages and whether isolating those temporal subspaces enables the controlled steering of generated text …


CCI4. 0: A Bilingual Pretraining Dataset for Enhancing Reasoning in Large Language Models

G Liu, L Wang, J Li, Y Yu, Y Xu, J Chen, Y Bai, F Liao… - arXiv preprint arXiv …, 2025

We introduce CCI4.0, a large-scale bilingual pre-training dataset engineered for superior data quality and diverse human-like reasoning trajectory. CCI4.0 occupies roughly $35$ TB of disk space and comprises two sub-datasets: CCI4.0-M2-Base …


Centroid analysis: Inferring concept representations from open-ended word responses

The present research proposes and evaluates a novel method-centroid analysis-for measuring representations and concepts at both individual and group levels by mapping open-ended responses onto a pre-existing semantic vector space …


Challenges and opportunities in modern artificial intelligence systems: A focus on natural language processing

… Owing to their substantial data requirements, their training often involves vast text corpora such as the Common Crawl, along with sources like books and Wikipedia. Although LLMs have been developed for diverse languages or even designed as …


Challenges and opportunities of automated essay scoring for low-proficient L2 English writers

V De Wilde, O De Clercq - Assessing Writing, 2025

Assessing students’ writing can be a challenging activity. To make writing assessment more feasible, researchers have investigated the possibilities of automated essay scoring (AES). Most studies investigating AES have focused on L1 …


Challenges in Zero-Shot and Few-Shot Learning for Complex Queries

Zero-shot and few-shot learning have emerged as promising approaches for enabling machine learning models to generalize to novel tasks with minimal or no task-specific training data. However, applying these techniques to complex queries …


Characterizing Bias: Benchmarking Large Language Models in Simplified versus Traditional Chinese

H Lyu, J Luo, J Kang, A Koenecke - arXiv preprint arXiv:2505.22645, 2025

… We operationalize this by retrieving the frequency of each name’s occurrence in the Common Crawl web crawl corpus.Then, for each LLM and each prompting language variant (English, Simplified Chinese, or Traditional Chinese), we examine …


Charting the Landscape of African NLP: Mapping Progress and Shaping the Road Ahead

JO Alabi, MA Hedderich, DI Adelani, D Klakow - arXiv preprint arXiv:2505.21315, 2025

With over 2,000 languages and potentially millions of speakers, Africa represents one of the richest linguistic regions in the world. Yet, this diversity is scarcely reflected in state-of-the-art natural language processing (NLP) systems and large …


ChatGPT and L2 Chinese writing: evaluating the impact of model version and prompt language on automated corrective feedback

CTY Yang, HHJ Chen - Computer Assisted Language Learning, 2025

… As of 2022, the year ChatGPT-3.5 was officially announced, CommonCrawlFootnote 1 contained approximately 52 billion pages in English compared to 6.8 billion pages in Chinese, revealing a stark imbalance in language representation. This uneven …


ChatGPT based credit rating and default forecasting

J Lin, S Lai, H Yu, R Liang, J Yen - Journal of Data, Information and Management, 2025

… In particular, the recent revolutionary GPT-3 has as many as 175 billion parameters, and more than 80% of the data comes from network information such as Common Crawl, WebText2, and Wikipedia, as shown in Table 1. It is capable of …


ChatGPT or A Silent Everywhere Helper: A Survey of Large Language Models

A Akhtarshenas, A Dini, N Ayoobi - arXiv preprint arXiv:2503.17403, 2025

Large Language Models (LLMs) have revo lutionized natural language processing Natural Language Processing (NLP), with Chat Generative Pre-trained Transformer (ChatGPT) standing out as a notable exampledue to its advanced capabilities and widespread …


ChatGPT's security risk and its legal countermeasures

… , it is clear from its May 2020 article that the company primarily uses data from the CommonCrawl corpus, the WebText corpus, Wikipedia pages, and books for training21). The CommonCrawl corpus is a large dataset of raw web pages …


Chitrakshara: A Large Multilingual Multimodal Dataset for Indian languages

S Khan, A Faraz, A Ravi, M Nauman, M Sarfraz… - CVPR 2025 Workshop Vision …

… To address this gap, we introduce the Chitrakshara dataset series, covering 11 Indian languages sourced from Common Crawl. It comprises (1) … We begin by gathering 95 Common Crawl dumps spanning the years 2013 to 2023. Unlike …


Chitrarth: Bridging Vision and Language for a Billion People

S Khan, A Tarun, A Ravi, A Faraz, PK Pokala…

Recent multimodal foundation models are primarily trained on English or high resource European language data, which hinders their applicability to other medium and low-resource languages. To address this limitation, we introduce Chitrarth (Chitra …


CHRONOBERG: Capturing Language Evolution and Temporal Awareness in Foundation Models

N Hegde, S Paul, L Joel-Frey, M Brack, K Kersting… - arXiv preprint arXiv …, 2025

Large language models (LLMs) excel at operating at scale by leveraging social media and various data crawled from the web. Whereas existing corpora are diverse, their frequent lack of long-term temporal structure may however limit an LLM's ability …


Chronologically Consistent Large Language Models

S He, L Lv, A Manela, J Wu - arXiv preprint arXiv:2502.21206, 2025

… Starting in the year 2013, we introduce high-quality common crawl data. We witness a significant decrease in validation loss with the increase in data diversity. In the right panel of Figure 2, the GLUE scores for nearly all models exceed that of …


CHURRO: Making History Readable with an Open-Weight Large Vision-Language Model for High-Accuracy, Low-Cost Historical Text Recognition

SJ Semnani, H Zhang, X He, M Tekgürler, MS Lam - arXiv preprint arXiv:2509.19768, 2025

Accurate text recognition for historical documents can greatly advance the study and preservation of cultural heritage. Existing vision-language models (VLMs), however, are designed for modern, standardized texts and are not equipped to read the …


Circuit Partitioning Using Large Language Models for Quantum Compilation and Simulations

P Sinha, SK Jha, S Raj - arXiv preprint arXiv:2505.07711, 2025

We are in the midst of the noisy intermediate-scale quantum (NISQ) era, where quantum computers are limited by noisy gates, some of which are more error-prone than others and can render the final computation incomprehensible. Quantum circuit …


Citation Knowledge Graphs for Academic Insights: Modelling, Processing, and Analysis

A Angadi, A Surekha, SK Gorripati, S Muppidi - Graph Mining: Practical Uses and …, 2025

… As shown in Table 9.1, this chapter utilizes data from four key domains: Twitter and Facebook for social network analysis, CiteSeerX and arXiv HEP-Th for citation networks, MovieLens for recommendation systems [13], and Common Crawl and …


Citation: Kamalov, F.; Santandreu

… using mostly CommonCrawl, WebText, English Wikipedia, and two books corpora (Books1 …


Cite as: Robert Sassan, Civil Liability for Autonomous Police Robots: The Inadequacy of § 1983 in Responding to Robot Excessive Force, 30 RICH. JL & TECH. 471 …

[1] In the fall of 2023, New York City announced its plan to deploy an autonomous police robot, called K5, to patrol the Times Square subway station. 1 Mayor Eric Adams touted the initiative as a cost-saving measure K5 costs less per hour than a …


Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Models

Y Huang, S Chen, J Pei, M Zaheer, B Dhingra - arXiv preprint arXiv:2506.17585, 2025

… Due to frequent title duplication—especially in Common Crawl and RepliQA—we adopt a renaming strategy using an LLM. For each duplicated title, we iteratively rename the document until all titles are unique. We also perform cross-source …


CLaMP 3: Universal Music Information Retrieval Across Unaligned Modalities and Unseen Languages

S Wu, Z Guo, R Yuan, J Jiang, S Doh, G Xia, J Nam… - arXiv preprint arXiv …, 2025

CLaMP 3 is a unified framework developed to address challenges of cross-modal and cross-lingual generalization in music information retrieval. Using contrastive learning, it aligns all major music modalities--including sheet music, performance …


CLAPnq:[underline] C [/underline] ohesive [underline] L [/underline] ong-form [underline] A [/underline] nswers from [underline] P [/underline] assages in Natural …

S Rosenthal, A Sil, R Florian, S Roukos - Transactions of the Association for …, 2025

… However, they use sentence-level matching (by encoding sentences for semantic similarity comparisons) to retrieve up to top 7 documents from Common Crawl while avoiding exact matches as the abstractive dataset. In the extractive version, the …


CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training

S Diao, Y Yang, Y Fu, X Dong, D Su, M Kliegl, Z Chen… - arXiv preprint arXiv …, 2025

Pre-training datasets are typically collected from web content and lack inherent domain divisions. For instance, widely used datasets like Common Crawl do not include explicit domain labels, while manually curating labeled datasets such as …


Cloze Encounters: The Impact of Pirated Data Access on LLM Performance

Large Language Models (LLMs) have demonstrated remarkable capabilities in text generation, but their performance may be influenced by the datasets on which they are trained, including potentially unauthorized or pirated content. We investigate the …


CMVC+: a Multi-View Clustering Framework for Open Knowledge Base Canonicalization via Contrastive Learning

Y Yang, W Shen, J Shu, Y Liu, E Curry, G Li - IEEE Transactions on Knowledge and …, 2025

Open information extraction (OIE) methods extract plenty of OIE triples from unstructured text, which compose large open knowledge bases (OKBs). Noun phrases and relation phrases in such OKBs are not canonicalized, which leads to …


Code Blue: The Threat of Synthetic Data Use to Generative Medical AI

AB Cyphert, VK Blake - Houston Journal of Health Law & Policy, 2025

In the field of health care, artificial intelligence (AI) has massive potential to improve and save lives. 1 AI systems have been used in the health care field for many years now, diagnosing, screening, treating, predicting illnesses and injuries, and enabling …


CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation

Z Liu, R Zhang, Z Wang, Z Yang, P Hovland, B Nicolae… - arXiv preprint arXiv …, 2025

Large language models (LLMs) are revolutionizing many science and engineering fields. However, their huge model sizes impose extremely demanding needs of computational resources in the pre-training stage. Although low-rank factorizations …


Collaborative Growth: When Large Language Models Meet Sociolinguistics

D Nguyen - Language and Linguistics Compass, 2025

Large Language Models (LLMs) have dramatically transformed the AI landscape. They can produce remarkable fluent text and exhibit a range of natural language understanding and generation capabilities. This article explores how LLMs might be …


Collaborators through Time: How Humans Partnered with Nature, Technology, and Each Other


Collective Scoping: Streamlining Entity Sets Towards Efficient and Effective Entity Linkages

L Traeger, A Behrend, G Karabatis - SN Computer Science, 2025

… For the best static strategy, we aggregate the Glove embeddings trained on Common Crawl (Glove-300d/42B/1.9M) without out-of-vocabulary (OOV) retrievals [28] and concatenate a one-hot encoded datatype (numeric, text, date, miscellaneous) …


College of Information Studies Language Science Center Institute for Advanced Computer Studies Natural language processing needs substantial data to make …

D Peskov, J Boyd-Graber, P Resnik, M Mazurek…

Title of proposal: Gathering Language Data Using Experts Denis Peskov, 2022 Dissertation directed by: Professor Jordan Boyd-Graber Department of Computer Science College of Information Studies Language Science Center Institute for …


Combining large language models with interpretable models for explainable aspect-based sentiment analysis in the medical domain

Y Zhang, S Wen, Y Zhu, Z Li, X Wang - Journal of King Saud University Computer and …, 2025

Aspect-based sentiment analysis (ABSA) effectively enables fine-grained understanding of medical feedback texts, but existing methods typically lack transparency, limiting trustworthiness in practical medical scenarios. To address this …


Combining transfer and ensemble learning models for image and text aspect-based sentiment analysis

A Chauhan, R Mohana - International Journal of System Assurance Engineering …, 2025

… These embeddings are trained on extensive corpora, such as Wikipedia and Common Crawl, and provide dense vector representations for each word in our text data. GloVe embeddings effectively capture word meanings based on their global co-occurrence …


Coming Back Differently: An Exploratory Case Study of Near Death Experiences of Webpages

In this case study, we use web archives to analyze 8,824 webpages that were taken offline and subsequently put back online, thus experiencing a “near death experience.” We enumerate the stages of a webpage’s near death experience …


Common Corpus: The Largest Collection of Ethical Data for LLM Pre-Training

PC Langlais, CR Hinostroza, M Nee, C Arnett… - arXiv preprint arXiv …, 2025

… Requiring a corpus of 300 billion tokens, GPT-3 introduced a standard training data pipeline shared by nearly all language models to date: large-scale processing of web datasets (45 TB of compressed source data from Common Crawl) and …


CommonForms: A Large, Diverse Dataset for Form Field Detection

J Barrow - arXiv preprint arXiv:2509.16506, 2025

… We use Common Crawl as a wellspring of PDFs and apply a rigorous cleaning process. This cleaning process results in improved data efficiency compared to using every PDF with a form field. To train the FFDNet family of models, we cast the …


Comparative Analysis Based on DeepSeek, ChatGPT, and Google Gemini: Features, Techniques, Performance, Future Prospects

A Rahman, SH Mahir, MTA Tashrif, AA Aishi, MA Karim… - arXiv preprint arXiv …, 2025

… Primary sources for general linguistic coverage included Common Crawl and WebText, and BooksCorpus was used for long-form structured text data [30], [31]. Domain-specific corpora like PubMed and arXiv were essential for testing the …


Comparative Analysis of Differentiated Approaches to Utilizing AI for Subverting Stereotypes

X Feng, M Murakami - Journal of Advances in Information Technology, 2025

Limited or superficial knowledge about others can foster stereotypes and prejudice. Consequently, this study explores specific methods to counteract these stereotypes. We posit that the key to challenging stereotypes lies in acquiring relevant knowledge …


Comparative Analysis of Embedding Models for Hindi-English Code-Mixed University related queries

O Ingale, S Margaj - The Voice of Creative Research, 2025

This study presents a comparative analysis of open source embedding models for developing a understanding Hindi-English code-mixed language on university related questions. With the increasing adoption of conversational agents in Indian …


Comparative Analysis of Encoder-Based and Decoder-Based Architectures for Automatic Conspiracy Theory Identification

This study evaluates the performance of encoderbased and decoder-based architectures for the Automatic Conspiracy Theory Identification (ACTI) task, focusing on Subtask A, which involves detecting conspiratorial content in Telegram posts. I …


Comparative analysis of transformer models for sentiment classification of UK CBDC discourse on X

G Kaur, S Haraldsson, A Bracciali - Discover Analytics, 2025

Sentiment analysis is critical in understanding public perceptions of evolving currencies such as central bank digital currencies (CBDCs). This study compares three transformer-based models—DistilBERT, RoBERTa, and XLM-RoBERTa—for …


Comparative Assessment of Large Language Model-Driven Recommendation Systems in Smart Spaces

Large Language Models (LLMs) are revolutionizing the field of data analysis and the management of Big Data. These models, powered by deep learning and advanced neural network architectures, are able to process large amounts of text data to …


COMPARATIVE SWOT ANALYSIS OF AUTOMATIC TEXT CORRECTION METHODS

IAS Kizi - Строительство и образование, 2025

The objective of the article "Comparative SWOT Analysis of Automatic Text" is as follows: The objective of "Correction Methods" is to assess and contrast the strengths, weaknesses, opportunities, and threats of three primary approaches to automated …


Comparing differentially private fine-tuning methods for large language models

Large language models (LLMs) have obtained state-of-the-art performance on many tasks in natural language processing. The success is achieved by the fine-tuning of large pre-trained language models on downstream tasks. However, these machine …


Comparing Large Language Models on Unfair Clause Detection in Terms of Services

… Commonly used corpora include datasets like Wikipedia, Common Crawl, and OpenWebText, among others. Once the corpus is collected, it undergoes a pre-processing stage to standardize and clean the text for use as input to the transformer model. A …


Comparing LLM-generated and human-authored news text using formal syntactic theory

O Zamaraeva, D Flickinger, F Bond… - arXiv preprint arXiv …, 2025

This study provides the first comprehensive comparison of New York Times-style text generated by six large language models against real, human-authored NYT writing. The comparison is based on a formal syntactic theory. We use Head-driven Phrase …


Comparing the semantic structures of lexicon of Mandarin and English

Y Yang, RH Baayen - Language and Cognition, 2025

This paper presents a cross-language study of lexical semantics within the framework of distributional semantics. We used a wide range of predefined semantic categories in Mandarin and English and compared the clusterings of these …


Comparison of physician and large language model chatbot responses to online ear, nose, and throat inquiries

M Motegi, M Shino, M Kuwabara, H Takahashi… - Scientific Reports, 2025

Large language models (LLMs) can potentially enhance the accessibility and quality of medical information. This study evaluates the reliability and quality of responses generated by ChatGPT-4, an LLM-driven chatbot, compared to those written by …


Comparison of Pre-trained Models for Domain-specific Entity Extraction from Student Report Documents

AV Melnikova, MS Vorobeva, AV Glazkova - MODELING AND ANALYSIS OF …, 2025

The authors propose a methodology for extracting domain-specific entities from student report documents in Russian language using pre-trained transformer-based language models. Extracting domain-specific entities from student report documents …


Compass-V2 Technical Report

S Maria - arXiv preprint arXiv:2504.15527, 2025

… , multi-stage pipeline for parsing and decoding Common Crawl, Wikipedia, high quality documents, and synthesizing relevant e-commerce documents, as shown in Figure 4. We first parsed the full Common Crawl dataset (CommonCrawl, 2007) to …


Compositional Text-to-Image Generation with Feedforward Layout Generation

S Liu, W Nie, AC Cheng, M Mardani, C Liu, B Eckart… - European Conference on …, 2025

Current text-to-image models often struggle with complex prompts, requiring additional inputs for better control. Recently, BlobGen introduced blob representation to enhance compositionality in generative models. However, this …


Comprehensive Analysis of Transparency and Accessibility of ChatGPT, DeepSeek, And other SoTA Large Language Models

Despite increasing discussions on open-source Artificial Intelligence (AI), existing research lacks a discussion on the transparency and accessibility of state-of-the-art (SoTA) Large Language Models (LLMs). The Open Source Initiative (OSI) has recently …


Compressing steganographic payloads with LLM assistance

J Ahmadullah - Cryptology ePrint Archive, 2025

… In our case, we have a TF-IDF cache of 1 million Wikipedia articles (16.15 GB), though we could have used the Common Crawl database (6.24 TB compressed). Our system finds the best compression method to use, and works with several …


Computational Foundation of Generative AI Models

R Gupta, S Tiwari, P Chaudhary - Generative AI: Techniques, Models and …, 2025

This chapter on Generative AI Foundations provides a comprehensive overview of the key workflow architectures, computational efficiency considerations, and foundational algorithms used in the design and application of generative models. It …


COMPUTER SCIENCE ENGINEERING

Artificial intelligence (AI), and particularly deep learning (DL) techniques, have brought about major paradigm shifts in the healthcare domain in recent years, offering revolutionary innovations in clinical processes such as diagnosis, treatment …


CONCAP: Seeing Beyond English with Concepts Retrieval-Augmented Captioning

G Ibrahim, R Ramos, Y Kementchedjhieva - arXiv preprint arXiv:2507.20411, 2025

… We compare: (1) CX, which adds filtered XM3600 lexicons (excluding the XM100 captions); (2) CXP, which includes PangeaIns cultural terms; and (3) CXPW, which adds Wikipedia and Common Crawl entries for broader but less focused coverage …


CONCAP: Seeing Beyond English with Retrieval-Augmented Captioning

G Ibrahim, R Ramos, Y Kementchedjhieva - CVPR 2025 Workshop Vision Language Models …


ConLID: Supervised Contrastive Learning for Low-Resource Language Identification

N Foroutan, J Saydaliev, YE Kim, A Bosselut - arXiv preprint arXiv:2506.15304, 2025

Language identification (LID) is a critical step in curating multilingual LLM pretraining corpora from web crawls. While many studies on LID model training focus on collecting diverse training data to improve performance, low-resource …


Consistent Performance of GPT-4o in Rare Disease Diagnosis Across Nine Languages and 4967 Cases

L Chimirri, JH Caufield, N Matentzoglu, MA Gargano… - medRxiv, 2025

… All language this study constitute at least ~1% of the CommonCrawl, which is a proxy for the amount of relative intern data available in a given language, a reflection of the language-specific data available for training. For th nine languages …


Consumer Data is Key to Artificial Intelligence Value: Welcome to the Health Care Future

C James - Journal of Participatory Medicine, 2025

Humanity stands at the threshold of a new era in biological understanding, disease treatment, and overall wellness. The convergence of evolving patient and caregiver (consumer) behaviors, increased data collection, advancements in health technology and …


Content Creation and Artistic Style Injection: Revival of Kerala Mural Art

R Prasannan, AM Nair, AK Achal, MG Sabitha… - Proceedings of the Third …

… were categorised based on language and divided into different datasets by resolution, predicted likelihood of having a watermark, and predicted “aesthetic” score (ie, subjective visual quality) in the publicly available LAION-5B dataset, which …


Content Form: APRJA 13 Pierre Depaz

This article investigates how the word embeddings at the heart of large language models are shaped into acceptable meanings. We show how such shaping follows two educational logics. The use of benchmarks to discover the capabilities of large …


Content Moderation of Surveillance Search Queries Using Fine-Tuned Generative LLMs

A Bakly, D Than - Master's Thesis in Mathematical Sciences, 2025

We study how small, fine-tuned generative large language models (LLMs) can moderate free-text search queries for surveillance video systems. Four open models, Llama 3.2 1B, Llama 3.2 3B, Qwen 2.5 0.5B, and 1.5 B, are trained on six subtasks …


Continual Pre-training of MoEs: How robust is your router?

B Thérien, CÉ Joseph, Z Sarwar, A Panda, A Das… - arXiv preprint arXiv …, 2025

… Having established the benefits of replay and infinite learning rate schedules for continually pre-training MoEs, we now quantitatively verify the efficacy of these techniques by continually pre-training our MoEs on 200B tokens of Code and …


Continual Pre-training on Character-level Noisy Texts Makes Decoder-based Language Models Robust Few-shot Learners

T Kojima, Y Matsuo, Y Iwasawa - Transactions of the Association for Computational …, 2025

Recent decoder-based pre-trained language models (PLMs) generally use subword tokenizers. However, adding character-level perturbations drastically changes the delimitation of texts by the tokenizers, leading to the vulnerability of PLMs. This study …


Contrastive Learning Pre-Training and Quantum Theory for Cross-Lingual Aspect-Based Sentiment Analysis

… XLM-RoBERTa [8]: This is a transformer-based multilingual pre-trained language model that extends RoBERTa to over 100 languages, trained with a masked language modeling objective on a large-scale CommonCrawl corpus. It serves as a …


Contrastive pre-training and instruction tuning for cross-lingual aspect-based sentiment analysis

W Zhao, Z Yang, S Yu, S Zhu, L Li - Applied Intelligence, 2025

… [27] introduced mT5, a multilingual variant of the T5 model, which was pre-trained on a Common Crawl-based dataset covering 101 languages. Lin et al. [36] developed a multilingual generative language model trained on a corpus …


Controlled Natural Language Generation for Morphologically Rich Languages: The Case of Arabic

Recent breakthroughs in natural language processing (NLP) have led to the development of natural language generation (NLG) systems, such as large language models (LLMs), that can produce fluent, human-like text. However, these …


Controlling Context: Generative AI at Work in Integrated Circuit Design and Other High-Precision Domains

E Moss, E Watkins, C Persaud, P Karunaratne, D Nafus - arXiv preprint arXiv …, 2025

Generative AI tools have become more prevalent in engineering workflows, particularly through chatbots and code assistants. As the perceived accuracy of these tools improves, questions arise about whether and how those who work in high-precision …


Conventional Image Recognition Chatbot

P Gupta, S Maity, U Gupta - … Conference on Advances and Applications in Artificial …, 2025

The integration of artificial intelligence (AI) into human-computer interaction has revolutionized various fields by enabling seamless interaction between users and machines. This research aims to design a standard image recognition chatbot that …


Copyright and Generative AI: Opinion

S Dusollier, M Kretschmer, T Margoni, P Mezei… - JIPITEC–Journal of …, 2025

The ECS considers that the current development of generative artificial intelligence (AI), under the regulatory framework set up by the Directive on Copyright in the Digital Single Market (CDSM) of 2019 and the AI Act of 2024 (Regulation (EU) 2024/1689) …


Copyright Infringement of Content Used in Training Generative AI Models: A Comparative Analysis in Light of Current US Cases

EM Dogan - J. Com. & Intell. Prop. L., 2025

This study comparatively examines the copyright infringement issue of content used in training generative Al models within the framework of US, EU, and Turkish legal systems. In light of current US cases, it evaluates the US" fair use" doctrine, EU's text …


Copyright, fair use, and AI technology development: time to sunset the “transformative purpose” test

SJ Blodgett-Ford - Research Handbook on the Law of Artificial …, 2025

… ” or plural “corpora”) for “training” text-based AI models include for example, digital copies of books, “CommonCrawl” (a specific subset of … a 2019 snapshot of Common Crawl, accounting for 100 million tokens (basic units of text).”Allegedly, the …


CoRAG: Collaborative Retrieval-Augmented Generation

A Muhamed, M Diab, V Smith - arXiv preprint arXiv:2504.01883, 2025

Retrieval-Augmented Generation (RAG) models excel in knowledge-intensive tasks, especially under few-shot learning constraints. We introduce CoRAG, a framework extending RAG to collaborative settings, where clients jointly train a shared model …


CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmented Generation

Y Cheng, K Mao, Z Zhao, G Dong, H Qian, Y Wu…

… However, since CORAL is built upon Wikipedia and existing LLMs are typically trained on corpora like Wikipedia and CommonCrawl, using these LLMs as generators could lead to contamination in the conversational RAG process due to …


Corn Cultivation with Precision: Language Agents for Real-Time Decision Making

A Chao - 2025 1st International Conference on Consumer …, 2025

The amalgamation of Retrieval-Augmented Generation (RAG) and Chain-of-Thought (CoT) oriented prompt engineering within the context of Large Language Models (LLMs) represents a considerable progression in agricultural decision support frameworks …


Corpus Modeling and the Geometries of Text

MA Taylor - The Oxford Handbook of the Sociology of Machine …, 2025

The spatial turn in computational text analysis is inciting a sprightly interdisciplinary passion that seems, at times, to entail sprinting when we ought to mosey. These advances—specifically the growing family of word embedding techniques—are …


COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMs

L Liu, Z Xu, Z Zhang, H Kang, Z Li, C Liang, W Chen… - arXiv preprint arXiv …, 2025

… 2020), which is a colossal, cleaned version of Common Crawl’s web crawl corpus for pre-taining. We conduct comprehensive experiments and ablation studies on 130M models and demonstrate the token efficiency of COSMOS. We then scale up …


Counterfactual Query Rewriting to Use Historical Relevance Feedback

J Keller, M Fröbe, G Hendriksen, D Alexander… - arXiv preprint arXiv …, 2025

When a retrieval system receives a query it has encountered before, previous relevance feedback, such as clicks or explicit judgments can help to improve retrieval results. However, the content of a previously relevant document may have …


Craw4LLM: Efficient Web Crawling for LLM Pretraining

S Yu, Z Liu, C Xiong - arXiv preprint arXiv:2502.13347, 2025

… are typically built from large-scale web crawls such as Common Crawl (CommonCrawl… Common web crawlers like Common Crawl prioritize pages based on graph connectivity … A critical analysis of the largest source for generative AI training data …


CrediBench: Building Web-Scale Network Datasets for Information Integrity

E Kondrup, S Sabry, H Abdallah, Z Yang, J Zhou… - arXiv preprint arXiv …, 2025

… Our processed one-month snapshot extracted from the Common Crawl archive in December 2024 contains 45 million nodes and 1 billion … As the Common Crawl data is released monthly, we represent the graph as a sequence of graph snapshots …


CReSt: A Comprehensive Benchmark for Retrieval-Augmented Generation with Complex Reasoning over Structured Documents

M Khang, S Park, T Hong, D Jung - arXiv preprint arXiv:2505.17503, 2025

… To ensure broad document-domain coverage in both English and Korean, CReSt sources raw documents from two publicly available collections: PDF files from Common Crawl (CC-MAIN) for English and crawled document images from National …


CritiQ: Mining Data Quality Criteria from Human Preferences

H Guo, K Lv, Q Guo, T Liang, Z Xi, D Song, Q Zhang… - arXiv preprint arXiv …, 2025

Language model heavily depends on high-quality data for optimal performance. Existing approaches rely on manually designed heuristics, the perplexity of existing models, training classifiers, or careful prompt engineering, which require significant …


Cross-Domain Affective Analysis of Large-Scale Textual Data Using Transformer-Based NLP Models

… In this study, datasets such as Common Crawl News included hundreds of thousands of documents, necessitating scalable solutions for … To analyze the emotional tone of hundreds of thousands of news articles from the Common Crawl …


Cross-Encoder Models in Czech

… FERNET-C5 - BERT-based model trained on 93 GB of Czech text extracted from the Czech portion of the Common Crawl dataset (C5 … was trained on large-scale, pre-filtered corpora from the CommonCrawl dataset. This data spans 100 different …


Cross-Language Summarization

Cross-Language Summarization (CLS) is a crucial task of NLP in which the goal is to generate a summary in a target language that differs from the language of the input document. This thesis investigates the capabilities of a twostage Extractive–Abstractive …


Cross-Lingual Language Models for Real-Time Translation in Multilingual Human-Robot Collaborative Interfaces

… This was followed by the development of XLMRoBERTa (XLM-R), which expanded the scale and performance of mBERT by training on a much larger and more diverse dataset (CommonCrawl) and using a more efficient pretraining …


Cross-Lingual Machine Translation: An Integrated Approach with Contextualsequencexl and Goldenhawk Search Optimization Algorithm (GHSO)

… With the help of sub word data from Common Crawl (600B tokens), FastText Sub word has 2 million-word vectors trained on it. Sub word embedding breaks down each word into its constituent sub words, giving us more information. The sub words …


Cross-Lingual Text Classification with Large Language Models

B Han, ST Yang, C LuVogt - Companion Proceedings of the ACM on Web …, 2025

… (mixtral-8x7b)3 • Meta-Llama-3-70B-Instruct (llama3-70b)4 • XLM-RoBERTa-L (XLM)5: a multilingual version of RoBERTa, pre-trained on 2.5TB of filtered CommonCrawl data containing 100 languages. It is evaluated in the state-of-the-art work [2]. …


Cross-Lingual Transfer for Low-Resource Natural Language Processing

I García-Ferrero - arXiv preprint arXiv:2502.02722, 2025

Natural Language Processing (NLP) has seen remarkable advances in recent years, particularly with the emergence of Large Language Models that have achieved unprecedented performance across many tasks. However, these developments …


Cross-Lingual Transfer Learning for Low-Resource Hate Speech Detection

Hate speech on digital platforms poses significant societal challenges, particularly in low-resource languages where data scarcity hampers effective detection. This study explores cross-lingual transfer learning to enhance hate speech detection in low-resource …


Cross-Linguistic Transfer in Multilingual NLP: The Role of Language Families and Morphology

A Bankula, P Bankula - arXiv preprint arXiv:2505.13908, 2025

… We fine-tune XLM-R a 12-layer Transformer model trained on 100 languages with CommonCrawl data [3]. XLM-R’s training data is much larger than mBERT’s and is as such more balanced for various languages, which tends to improve performance …


Cross-region Model Training with Communication-Computation Overlapping and Delay Compensation

Y Zhu, Y Xu, H Xu, Y Liao, Z Yao, L Huang - arXiv preprint arXiv:2504.17672, 2025

Training large language models (LLMs) requires massive computational resources, often necessitating the aggregation of geographically distributed data centers (\ie, cross-region training). However, the high communication latency in wide-area …


Cuac: Fast and Small Universal Representations of Corpora

JP McCrae, B Stearns, AM Qazi, S Banerjee, AK Ojha

… Firstly, we evaluate a small section of the Colossal Common Crawl Corpus (Raffel … Large-scale NLP datasets (eg, Common Crawl, Wikipedia, or domain-specific corpora) take up terabytes of space. A specialized compression format can …


Cuckoo: An IE Free Rider Hatched by Massive Nutrition in LLM's Nest

L Peng, Z Wang, F Yao, J Shang - arXiv preprint arXiv:2502.11275, 2025

Massive high-quality data, both pre-training raw texts and post-training annotations, have been carefully prepared to incubate advanced large language models (LLMs). In contrast, for information extraction (IE), pre-training data, such as BIO-tagged …


CUET_NetworkSociety@ DravidianLangTech 2025: A Multimodal Framework to Detect Misogyny Meme in Dravidian Languages

MDMK Ratul, S Aftahee, TA Babu, J Hossain… - Proceedings of the Fifth …, 2025

Memes are commonly used for communication on social media platforms, and some of them can propagate misogynistic content, spreading harmful messages. Detecting such misogynistic memes has become a significant challenge, especially for low-resource …


Cultural Devaluation within Occupations: Demand-and Supply-Side Analysis in Japan

Much existing research on job devaluation infers the role of cultural beliefs using the proportion of women in an occupation as a proxy. This study, rather than relying on the percentage of female workers, directly measures cultural feminine meanings …


Cultural dimensions in the perception of success: Comparative analysis of word associations across languages using LLM word embedding

… Each language’s embeddings are trained independently on respective Common Crawl or Wikipedia corpora using subword information, allowing robust representation even for infrequent words. These embeddings reflect semantic …


Cultural Variability and Bias in Online Social Interactions and Large Language Models

Despite their intended global usage, most technologies are designed and developed within narrow cultural frames, reflecting the values and assumptions of their often Western developers. Thus, when deployed across cultures without …


CULTUREINSTRUCT: Curating Multi-Cultural Instructions at Scale

Large language models, despite their remarkable success in recent years, still exhibit severe cultural bias. Therefore, in this paper, we introduce CULTUREINSTRUCT 1, a large-scale instruction-tuning dataset designed to reduce …


Curriculum-Guided Layer Scaling for Language Model Pretraining

K Singh, N Band, E Adeli - arXiv preprint arXiv:2506.11389, 2025

As the cost of pretraining large language models grows, there is continued interest in strategies to improve learning efficiency during this core training stage. Motivated by cognitive development, where humans gradually build knowledge as their brains …


Customer Query Classification Based on DistilBERT and TextCNN

T Satidkarn, A Imsombut - 2024 8th International Conference on Information …, 2024

Customer service representatives handle customer inquiries and resolve issues. However, due to the high volume of inquiries, it is challenging for customer service representatives to provide timely services. To address this issue, natural language …


CyLLM-DAP: Cybersecurity Domain-Adaptive Pre-Training Framework of Large Language Models

… The Common Crawl organization maintains this dataset by conducting regular scrawls, which started in 2007. Currently, Common Crawl is the biggest dataset with hundreds of TiB of data, spanning over billions of web pages. When working with …


D3. 1 Overview of the state of the art

Deliverable 3.1 (D3. 1)–Overview of the state of the art, is the first deliverable within Work Package (WP) 3-Analysing Social Media Communication. The overall aim of the WP is to analyse the interrelationship between emotions, values and identities in …


DALIP: Distribution Alignment-based Language-Image Pre-Training for Domain-Specific Data

J Wu, J Xie, Z Zhang, Q Wang, Q Hu, P Li, S Xu - arXiv preprint arXiv:2504.01386, 2025

… Specifically, MetaCLIP [62] is introduced to utilize metadata expansion and create a substantial CommonCrawl dataset of 400 million image-text pairs. SigLIP [68] improves training efficiency by introducing a pairwise sigmoid loss. For fine-grained …


Data augmentation for dense passage retrieval using corpus-passage frequency-based token deletion

A Moon, K Kim, J Lee - Journal of Big Data, 2025

This paper proposes a novel data augmentation method to address class imbalance in large-scale information retrieval systems. In particular, a corpus-passage frequency-based token deletion technique is introduced to improve the accuracy of …


Data Caricatures: On the Representation of African American Language in Pretraining Corpora

N Deas, B Vente, A Ananthram, JA Grieser, D Patton… - arXiv preprint arXiv …, 2025

… 2021) and because many corpora predominantly rely on Common Crawl texts, we focus our human judgments and AAL feature analyses on … texts are the primary source of data for pertaining, we focus experiments on corpora composed of …


Data Efficacy for Language Model Training

Y Dai, Y Huang, X Zhang, W Wu, C Li, W Lu, S Cao… - arXiv preprint arXiv …, 2025

… We utilize the Redpajama [28] sourced from CommonCrawl as D, which offers a relatively balanced knowledge distribution [38]. The downstream loss J(θ) for the data scoring model is computed on the LIMA [39], which is a high-quality dataset …


Data hound: Analysing non-English data smells in large code datasets

Large Language Models (LLMs) are increasingly used for code-centric tasks. However, their training data often exhibits data smells that may hinder downstream quality. This research focuses on the “Uneven Natural Languages” smell and the …


Data Leakage in Visual Datasets

P Ramos, R Ramos, N Garcia - arXiv preprint arXiv:2508.17416, 2025

We analyze data leakage in visual datasets. Data leakage refers to images in evaluation benchmarks that have been seen during training, compromising fair model evaluation. Given that large-scale datasets are often sourced from the internet …


Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training

K Yang, X Liu, L Ji, H Li, Y Gong, P Cheng, M Yang - arXiv preprint arXiv:2507.15640, 2025

Continual pre-training on small-scale task-specific data is an effective method for improving large language models in new target fields, yet it risks catastrophic forgetting of their original capabilities. A common solution is to re-weight training …


Data Mixture Optimization: A Multi-fidelity Multi-scale Bayesian Framework

T Yen, AWT Siah, H Chen, T Peng, D Guetta… - arXiv preprint arXiv …, 2025

… We used only data from the first five categories to pretrain the language models while holding out the data from CommonCrawl and C4 to simulate data mixture optimization in out-of-distribution settings. We measure the training losses and …


Data Optimization for LLMs: A Survey

… Two distinct but interrelated redundancy problems emerge in LLM development: 1) Web-crawled training corpora exhibit significant duplication, with studies showing 3–14% of tokens in datasets like Common Crawl appearing in near-identical form across …


Data Transformation Strategies to Remove Heterogeneity

S Yoo, J Lee, C Yoon, G Son, H Hong, S Seo, S Yim… - arXiv preprint arXiv …, 2025

… Text models tokenize text data from diverse sources like CommonCrawl [171] dumps, websites, and books to train [191]. GPT-3 and LLaMa, for example, were trained using approximately 570GB of preprocessed text data, which included …


Data-Centric Elastic Pipeline Parallelism for Efficient Long-Context LLM Training

S Wang, Y Wang, A Sun, F Fu, Z Zhu, B Cui, X Han… - arXiv preprint arXiv …, 2025

… As transformer is the predominant architecture of LLM, we evaluate InfiniPipe to train GPT-series models (7B, 13B, 30B) on two famous real-world datasets: CommonCrawl and GitHub. The sequence length and token distribution of these two datasets are …


Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for Foundation Models

D Chen, Y Huang, X Pan, N Jiang, H Wang, C Ge…

The burgeoning field of foundation models necessitates advanced data processing mechanisms capable of harnessing vast valuable data with varied types utilized by these models. Nevertheless, the current landscape presents unique challenges that …


DataDecide: How to Predict Best Pretraining Data with Small Experiments

I Magnusson, N Tai, B Bogin, D Heineman, JD Hwang… - arXiv preprint arXiv …, 2025

… A SOTA Common Crawl corpus using best ablated deduplication, cleaning heuristics, and quality filter. We quality filter to top 7% of DCLM classified documents and further take 2+ or 3+ scores with FineWeb-edu classifier; or filter to top 3% or 10 …


Dataset Ownership Verification for Pre-trained Masked Models

Y Xie, J Song, Y Shan, X Zhang, Y Wan, S Zhang… - arXiv preprint arXiv …, 2025

High-quality open-source datasets have emerged as a pivotal catalyst driving the swift advancement of deep learning, while facing the looming threat of potential exploitation. Protecting these datasets is of paramount importance for the interests of …


Datasets, Documents, and Repetitions: The Practicalities of Unequal Data Quality

A Fang, H Pouransari, M Jordan, A Toshev, V Shankar… - arXiv preprint arXiv …, 2025

… DCLM does this by increasing the number of Common Crawl WARC files at the same rate as increasing the training token budget. However, as seen in Figure 6, when the number of WARC files increases, so does the number of duplicates. We …


DATE-LM: Benchmarking Data Attribution Evaluation for Large Language Models

C Jiao, Y Pan, E Xiao, D Sheng, N Jain, H Zhao… - arXiv preprint arXiv …, 2025

… We set D, to be Fineweb [54], a recently proposed high-quality web corpus constructed from CommonCrawl through cleaning and deduplication. We randomly sample 1M datapoints (2048 tokens each) as the large training data pool, and for a …


DCAD-2000: A Multilingual Dataset across 2000+ Languages with Data Cleaning as Anomaly Detection

Y Shen, W Lai, S Wang, X Zhang, K Luo, A Fraser… - arXiv preprint arXiv …, 2025

… To incorporate the most recent multilingual data, we extract and process Common Crawl dumps from May 2024 (CC-MAIN-2024-22) to November 2024 (CC-MAIN-2024-46). Using the Fineweb-2 pipeline8, we process 21.54TB of multilingual data, ensuring …


Decentralization of Generative AI via Mixture of Experts for Wireless Networks: A Comprehensive Survey

Y Xu, J Wang, R Zhang, C Zhao, D Niyato, J Kang… - arXiv preprint arXiv …, 2025

Mixture of Experts (MoE) has emerged as a promising paradigm for scaling model capacity while preserving computational efficiency, particularly in large-scale machine learning architectures such as large language models (LLMs). Recent …


Decoding corporate communication strategies: Analysing mandatory published information under Pillar 3 across turbulent periods with unsupervised machine …

A Pilková, M Munk, L Kelebercová - PLOS ONE, 2025

This study explores the communication patterns of Slovak banks with stakeholders through mandatory disclosures mandated by Basel III’s Pillar 3 framework and annual reports in 2007−2022. Our primary objective is to identify key topics …


Decoding Fake News and Hate Speech: A Survey of Explainable AI Techniques: A Survey of Explainable AI Techniques.

M Ngueajio, S Aryal, M Atemkeng, G Washington… - ACM Computing Surveys

… Additionally, the authors utilize the common crawl version of the pretrained Glove model to build the word embeddings for the non-BERT classifiers, and each model s classification performance was evaluated, compared, and reported in terms of their …


Decoding Wine Narratives with Hierarchical Attention: Classification, Visual Prompts, and Emerging E-Commerce Possibilities

V Diaconita, A Belciu, AMI Corbea, I Simonca - Journal of Theoretical and Applied …, 2025

Wine reviews can connect words to flavours; they entwine sensory experiences into vivid stories. This research explores the intersection of artificial intelligence and oenology by using state-of-the-art neural networks to decipher the nuances in wine …


Decomposing Implicit Bias in Distributional Semantic Models: The Roles of First-and Second-Order Co-Occurrence

M Apsel, MN Jones - Proceedings of the Annual Meeting of the Cognitive …, 2025

… The C4 corpus is a cleaned subset of the Common Crawl web scrape corpus, designed to remove noisy data and improve text quality for research applications. We used the validation set of the English-language version of the dataset, sourced …


Decoupling Content and Expression: Two-Dimensional Detection of AI-Generated Text

G Bao, L Rong, Y Zhao, Q Zhou, Y Zhang - arXiv preprint arXiv:2503.00258, 2025

The wide usage of LLMs raises critical requirements on detecting AI participation in texts. Existing studies investigate these detections in scattered contexts, leaving a systematic and unified approach unexplored. In this paper, we present HART, a …


DeDisCo at the DISRPT 2025 Shared Task: A System for Discourse Relation Classification

Z Ju, J Wu, A Purushothama, A Zeldes - arXiv preprint arXiv:2509.11498, 2025

This paper presents DeDisCo, Georgetown University's entry in the DISRPT 2025 shared task on discourse relation classification. We test two approaches, using an mt5-based encoder and a decoder based approach using the openly available …


Deep Generative Models for Prediction and Design of Enzymes

Over billions of years, proteins have evolved functions that drive nearly all biological processes on Earth. This vast evolutionary record offers an enormous experimental dataset that enables predictive modeling of biological systems. In this thesis, I …


Deep Learning and Natural Language Processing in the Field of Construction

R Kessler, N Béchet - arXiv preprint arXiv:2501.07911, 2025

This article presents a complete process to extract hypernym relationships in the field of construction using two main steps: terminology extraction and detection of hypernyms from these terms. We first describe the corpus analysis method to extract …


Deep Learning Error Minimization System for Real-Time Big Data Analysis in Mobile Applications

Y Qing, Z Jing - Academic Journal of Computing & Information Science

This paper presents a novel deep learning error minimization system designed to enhance the efficiency, adaptability, and accuracy of real-time big data analysis in mobile applications. Traditional deep learning systems face limitations such as the …


Deep Learning in Biomedical Research and Statistical Inference on Time Warping Functions

In this dissertation, I will present two research projects that I have been working on during my doctoral study at Florida State University. I will provide a brief summary for each of these projects as follows. The detailed studies will be given in the chapters …


Deep multimodal fusion for video game age rating classification

video games appeal to a wide range of ages, from children to adults. As a result, reliable age rating systems like the Entertainment Software Rating Board (ESRB) and Pan European Game Information (PEGI) are essential for guarding younger …


Deep Research Agents: A Systematic Examination And Roadmap

Y Huang, Y Chen, H Zhang, K Li, M Fang, L Yang, X Li… - arXiv preprint arXiv …, 2025

… In addition, studies [43, 56] expanded retrieval sources from structured databases (eg, Wikipedia) to large-scale, diverse web corpora such as the Common Crawl dump preprocessed via the CCNet pipeline [25]. Further improvements of RAG …


DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research

J Coelho, J Ning, J He, K Mao, A Paladugu, P Setlur… - arXiv preprint arXiv …, 2025

… FineWeb is a large-scale English web corpus collected from 96 Common Crawl snapshots between 2013 and 2024. It comprises approximately 15 trillion tokens of cleaned and deduplicated web data. The dataset employs rigorous filtering …


Defending Against Authorship Attribution Attacks With Large Language Models

In today's digital era, individuals leave significant digital footprints through their writing, whether on social media or on their employer's devices. These digital footprints pose a serious challenge for identity protection: authorship attribution …


Defending the digital domain: BERT-Powered deep learning for aggression detection in text stream

GC Sekhar, A Brahmaiah, G Kalaiarasi, M Selvi… - AIP Conference …, 2025

With the advent of the internet, cyberbullying has grown in importance, affecting the well-being of its victims. To tackle this concern, an effective tool for a range of NLP tasks because it has several advantages over the current models in field. BERT …


DEFINING A STRATEGIC ACTION PLAN FOR AI IN HIGHER EDUCATION

F PAPADHOPULLI, M TAFAJ - PROCEEDINGS OF INTERNATIONAL SCIENTIFIC …, 2025

This paper discusses key challenges of Artificial Intelligence in Education, with main focus on higher education institutions. We start with reviewing normative actions of international organizations and concerns expressed about the current technical …


Defining Foundation Models for Computational Science: A Call for Clarity and Rigor

Y Choi, SW Cheung, Y Kim, PH Tsai, AN Diaz… - arXiv preprint arXiv …, 2025

The widespread success of foundation models in natural language processing and computer vision has inspired researchers to extend the concept to scientific machine learning and computational science. However, this position paper argues that as the …


Deliverable 3.11: UC5: Report on methodology and results to use online data for business register enhancement

VAS Finland, O ten Bosch, ADS Netherlands…

1 Background This document is part of the Work Package 3 (WP3) New use-cases from the ESSnet Trusted Smart Statistics–Web Intelligence Network project (TSS-WIN). The overall objective of WP3 is to explore the potential of new types of web data …


Delving into: the quantification of Ai-generated content on the internet (synthetic data)

DHR Spennemann - arXiv preprint arXiv:2504.08755, 2025

… models, such as ChatGPT and DeepSeek, are derived from fiction and non-fiction books, government documents, articles, and web pages to establish the parameters of language, while a considerable amount of factual knowledge has been taken from …


Democratising AI through Culture: Making Generative AI Participatory and Intersectional through an AI of the Commons

This report explores the possibility to make AI more inclusive and participatory through the concept of “AI as a cultural commons”. It proposes a practical approach for cultural practitioners to decolonise current AI systems by influencing …


Democratizing Foundation Models Through Robust Understanding and Learning With Imperfect Data

The field of generative AI has witnessed unprecedented growth, driven by advancements in large foundation models. However, this progress has created a critical bottleneck: the development of these models has become increasingly …


Demonstration Selection and Task Formulation for Effective In-Context Learning

With the rise of Large Language Models (LLMs) that are intractable to train or hidden behind APIs, prompting has become increasingly important as a training-free and customizable interface to leverage them. Fortunately, LLMs can perform novel tasks …


DEMYSTIFYING LARGE LANGUAGE MODELS: A TECHNICAL DEEP DIVE

This article provides a comprehensive exploration of Large Language Models (LLMs), examining their fundamental architectures, training methodologies, and future directions. Beginning with the revolutionary transformer architecture, it delves into …


DEMYSTIFYING LONG CHAIN-OF-THOUGHT REASON

Scaling inference compute enhances reasoning in large language models (LLMs), with long chains-of-thought (CoTs) enabling strategies like backtracking and error correction. Reinforcement learning (RL) has emerged as a crucial method for …


Demystifying Long Chain-of-Thought Reasoning in LLMs

E Yeo, Y Tong, M Niu, G Neubig, X Yue - arXiv preprint arXiv:2502.03373, 2025


Dependency Craftwork: Governing Digital Tools in Public Interest Collectives

This dissertation examines how public interest collectives adopt, adapt, and contest digital technologies in ways that reflect and reshape their normative commitments. Drawing on three case studies, I show that the attempt to apply values such as …


Deploying Privacy Guardrails for LLMs: A Comparative Analysis of Real-World Applications

S Asthana, B Zhang, R Mahindru, C DeLuca… - arXiv preprint arXiv …, 2025

… For example, the latest Common Crawl dataset (Smith et al. 2013) encompasses data from over 3 billion web pages and has vast user PII data. This raises concerns about data … Dirt cheap webscale parallel text from the common crawl. Association …


Design and Implementation of a GraphRAG based Recommendation System for Circular Integration in the Process Industry

The transition to a circular economy (CE) is crucial for the process industry, which faces significant challenges in resource efficiency and waste reduction due to complex material flows and industrial interdependencies. Current decision-making …


Design of disease recommendation system for neurological disorders using LLaMA2 model for biomedical applications

A Fadnavis, A Paigwar, L Harinkhede, V Chobitkar… - Recent Advances in Sciences …

With an aging global population and the increasing prevalence of neurological disorders, neuro-health research has gained paramount importance. This field encompasses a wide range of studies, from investigating the molecular and cellular …


DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation

J Xiao, M Wang, MH Lam, Y Wan, J Liu, Y Huo, MR Lyu - arXiv preprint arXiv …, 2025

… Design2Code [9] addressed this limitation by manually curating 484 authentic web pages from the Common Crawl dataset, constructing the first real-world benchmark for design-to-code evaluation. Building upon this foundation …


DESIGNER: Design-Logic-Guided Multidisciplinary Data Synthesis for LLM Reasoning

W Liu, Y Zhao, Y Luo, M Xu, J Liu, Y Li, X Hu, Y Xu… - arXiv preprint arXiv …, 2025

Large language models (LLMs) have achieved remarkable success in many natural language tasks but still struggle with complex, multi-step reasoning, particularly across diverse disciplines. Existing reasoning datasets often either lack disciplinary …


Designing Intelligent Interactive Systems for Vulnerable Populations

Technology alone cannot drive social change for vulnerable populations—it amplifies existing human intentions and social forces, sometimes exacerbating inequalities rather than reducing them. This dissertation shows that well-intentioned …


Detección de cognados verdaderos y false friends con word embeddings

C Periñán-Pascual, NJF Martínez - Revista Signos. Estudios de Lingüística, 2025

… In our experiment, we employed publicly available FastText embedding matrices of English and Spanish, each one containing 2M tokens and 300 dimensions trained on Wikipedia and the Common Crawl corpus, which contains petabytes of data …


Detección de cognados verdaderos y falsos amigos con word embeddings

C Periñán-Pascual, NJ Fernández-Martínez - Revista signos, 2025


Detecting caste and migration hate speech in low-resource Tamil language

BR Chakravarthi, S Rajiakodi, R Ponnusamy… - Language Resources and …, 2025

… It was pre-trained from scratch using a combination of datasets including Wikipedia, Common Crawl, PMINDIA, and Dakshina corpora, which cover 17 Indian languages. MuRIL is trained on both parallel and monolingual segments of these …


Detecting Data Contamination in LLMs via In-Context Learning

M Zawalski, M Boubdir, K Bałazy, B Nushi, P Ribalta - NeurIPS 2025 Workshop on …

We present Contamination Detection via Context (CoDeC), a simple and accurate method to detect and quantify training data contamination in large language models. CoDeC distinguishes between data memorized during training and data outside the …


Detecting deception: employing deep neural networks for fraudulent review detection on Amazon

JM Thilini Jayasinghe, S Dassanayaka - Neural Computing and Applications, 2025

In the era of e-commerce dominance, an increase in fake reviews on online shopping platforms compromises the integrity of consumer feedback systems. This study focuses on Amazon, a leading e-commerce platform in the USA, where fake …


Detecting Linguistic Diversity on Social Media

S Wong, B Adams, J Dunn - arXiv preprint arXiv:2502.21224, 2025

This chapter explores the efficacy of using social media data to examine changing linguistic behaviour of a place. We focus our investigation on Aotearoa New Zealand where official statistics from the census is the only source of language use data. We …


Detecting Manipulative Narratives in Social Media

During the 2022 Russian invasion of Ukraine, Telegram became a crucial platform for both information sharing and the spread of propaganda. Its speed, reach, and minimal moderation turned it into a powerful tool not only for communication but also …


Detecting True Cognates and False Friends with Word Embeddings Detección de cognados verdaderos y falsos amigos con word embeddings


Detection of Adverse Drug Events in Dutch clinical free text documents using Transformer Models: benchmark study

RM Murphy, N Mishra, NF de Keizer, DA Dongelmans… - arXiv preprint arXiv …, 2025

In this study, we set a benchmark for adverse drug event (ADE) detection in Dutch clinical free text documents using several transformer models, clinical scenarios and fit-for-purpose performance measures. We trained a Bidirectional Long Short-Term …


Detection of Cyber Attacks from Malicious URLs Using Ensemble Machine Learning Techniques

S Mohanty, AA Acharya - … Concepts, Applications, and Future Directions, Volume …

The Internet plays a crucial role in our daily lives, prompting browsers to introduce features that may expose users to threats from malicious URLs. These links can lead to spam, malware, and phishing attacks, causing significant financial loss and …


Detection of Inconsistencies in Payroll Regulations: A Comparison of Transformer Models and Traditional Retrieval Methods

This thesis explores how advanced language models compare to traditional NLP methods in aligning technical feedback validation rules from the Dutch Payroll Data Specification to their legal justifications in the Wage Tax Manual. These feedback …


Detection of Medical Conspiracy Theories with Limited Resources: Using Data from Prior Epidemics and LLMs

IB Schlicht, D Korenčić, B Chulvi, L Flek, P Rosso - 2025

Online dissemination of conspiracy theories (CTs) during epidemics poses significant risks to public health. This paper addresses the problem of detecting CTs in social media posts with an emphasis on the resource-constrained scenarios …


Detection of Phishing Activities Using Deep Learning Approaches

HB Gurushankar, HL Gururaj - 2025 17th International Conference on …, 2025

… The authentic URLs originated from a collection of web crawl data called Common Crawl. Phishtank, a website used as a phishing URL … A database consisting of one million authentic URLs from the Common Crawl database is used …


Detection of Somali-written Fake News and Toxic Messages on the Social Media Using Transformer-based Language Models

MA Mohamed, SD Ahmed, YA Isse, HM Mohamed… - arXiv preprint arXiv …, 2025

The fact that everyone with a social media account can create and share content, and the increasing public reliance on social media platforms as a news and information source bring about significant challenges such as misinformation, fake …


Determining category metadata in open data portals–an approach based on Formal Concept Analysis

MF Gligorijević, M Bogdanović, L Stoimenov - 2024 32nd Telecommunications Forum …, 2024

… The GloVe model used within this approach is pre-trained model on Common Crawl data with 840 billion generated tokens and vocabulary containing 2.2 million entries. Based on the calculated similarities between terms, the similarity between …


Develop a hybrid improved weighted quantum wolf optimization and fast mask recurrent convolutional neural network toenhance the performance of phishing …

C Rajeswary, M Thirumaran - Journal of Industrial and Management Optimization, 2025

… The data acquisition process begins by collecting two types of videos: authentic videos from the Common Crawl Foundation, and phishing videos from Phish Tank. The collected videos are then split into a training dataset and a testing dataset. The …


Developing Advanced Question-Answering Models for Legal Kazakh Texts: A Comparative Study of Modern Approaches

D Rakhimova, V Karyukin, A Karibayeva… - Asian Conference on …, 2025

… In this paper, we present mT5, a multilingual variant of T5 that was pre-trained on a new Common Crawl-based dataset covering 102 languages. We describe the design and modified training of mT5 and demonstrate its state-of-the-art …


Developing and Utilizing a Large-Scale Cantonese Dataset for Multi-Tasking in Large Language Models

J Jiang, AKY Truong, Y Chen, Q Bao, S Wang, P Chen… - arXiv preprint arXiv …, 2025

… tion, we interface with Common Crawl to amass a broader corpus of Chinese text. Open-source corpora: (1) Wikipedia serves as a primary source due to its comprehensive data availability. The Wikipedia pages are systematically archived …


Developing Japanese CLIP Models Leveraging an Open-weight LLM for Large-scale Dataset Translation

I Sugiura, S Kurita, Y Oda, D Kawahara, N Okazaki - … of the 2025 Conference of the …, 2025

… However, web crawling presents challenges due to the relatively small proportion of Japanese web pages in Common Crawl, which … It is a largescale dataset of image-text pairs, where images and their corresponding IMG-alt text are collected …


Developing locally trainable large language models

Emerging Large Language Models (LLMs) like GPT-3.5 and GPT-4 have been fundamentally transforming human society since their launch, as they demonstrate groundbreaking capabilities across various tasks. However, the colossal model size …


Developing named-entity recognition for state authority archives

I Toivanen, V Poso, M Lipsanen, T Välisalo - Digital Humanities in the Nordic and …, 2025

… The training data used for FinBERT consists of news data (from the national public broadcasting company, Yleisradio, and the Finnish News Agency STT), online discussions (from the forum Suomi24) and internet crawl data (eg …


Development and Validation of an AWE System “Write On with Cambi!”

Automated Writing Evaluation (AWE) systems that leverage artificial intelligence (AI) to provide formative feedback to students on their writing have been available for many years. Systems have historically used a specific type of AI, automated scoring …


Dhati+: Fine-tuned Large Language Models for Arabic Subjectivity Evaluation

S Bellaouar, A Nehar, S Souffi, M Bouameur - arXiv preprint arXiv:2508.19966, 2025

… on 2.5 TB of filtered CommonCrawl data containing 100 languages including Arabic. XLM-RoBERTa uses the same MLM objective as the XLM model with only one change: removing the language embeddings, allowing the model to better deal with …


Diagnosing the Effects of Pre-training Data on Fine-tuning and Subgroup Robustness for Occupational NER in Clinical Notes

D Moukheiber, S Mahindre, M Gao - Workshop on Spurious Correlation and Shortcut …

… We analyze out-of-domain performance on clinical datasets using our best-performing model on the common crawl dataset, fine-tuned Llama3-8B. For testing, we sample 1000 samples from both i2b2 and mimic datasets and report their recall performance …


Did ChatGPT or Copilot use alter the style of internet news headlines? A time series regression analysis

C Brogly, C McElroy - arXiv preprint arXiv:2503.23811, 2025

… This was built from the Common Crawl news dataset over 2016/09/02 to 2023/06/28; every Tuesday and Friday of crawled news pages is included within this range. Common Crawl HTML pages were parsed for hyperlink captions and headings …


Die SuperGLEBer at GermEval 2025 shared tasks: Growing pains-when more isn't always better

J Wunderle, J Pfister, A Hotho - KONVENS 2025 Conference on Natural Language …, 2025

We participate in this year’s GermEval 2025 Shared Tasks by extending SuperGLEBer, a comprehensive benchmark for evaluating German language understanding to the new tasks. Rather than focusing on optimizing taskspecific …


Diet Engine: A real-time food nutrition assistant system for personalized dietary guidance

AM Saad, MRH Rahi, MM Islam, G Rabbani - Food Chemistry Advances, 2025

In an era where intelligent technologies are rapidly shaping our lives, a Real-Time Nutrition Assistant System emerges as an essential tool for maintaining a healthy lifestyle and promoting awareness. A Real-Time Nutrition Assistant System …


Differentiation-Based Extraction of Proprietary Data from Fine-Tuned LLMs

Z Li, D Wu, S Wang, Z Su - arXiv preprint arXiv:2506.17353, 2025

The increasing demand for domain-specific and human-aligned Large Language Models (LLMs) has led to the widespread adoption of Supervised Fine-Tuning (SFT) techniques. SFT datasets often comprise valuable instruction-response pairs …


DigitalCertiAnalytics: A tool for collection and analysis of X. 509v3 digital certificates

The progressive evolution of digital communications requires ever more accurate and sophisticated protection, which is why this thesis focuses on the in-depth analysis of digital certificates, fundamental elements for ensuring the integrity and …


Directed Graph-alignment Approach for Identification of Gaps in Short Answers

A Sahu, PK Bhowmick - arXiv preprint arXiv:2504.04473, 2025

In this paper, we have presented a method for identifying missing items known as gaps in the student answers by comparing them against the corresponding model answer/reference answers, automatically. The gaps can be identified at word …


Discovering Forbidden Topics in Language Models

C Rager, C Wendler, R Gandikota, D Bau - arXiv preprint arXiv:2505.17441, 2025

Refusal discovery is the task of identifying the full set of topics that a language model refuses to discuss. We introduce this new problem setting and develop a refusal discovery method, LLM-crawler, that uses token prefilling to find forbidden topics …