Pre-trained BERT Model Retrieval: Inference-Based No-Learning Approach using k-Nearest Neighbour Algorithm

HL PHAM, R MIBAYASHI, T YAMAMOTO, MP KATO… - IEICE Transactions on …, 2025

In this study, we propose a method to efficiently retrieve BERT pre-trained models that achieve good performance on a specific document classification task. In natural language processing problems, the common practice involves fine-tuning existing …


Pre-trained language model for code-mixed text in Indonesian, Javanese, and English using transformer

AF Hidayatullah, RA Apong, DTC Lai, A Qazi - Social Network Analysis and Mining, 2025

Pre-trained language models (PLMs) have become increasingly popular due to their ability to achieve state-of-the-art performance on various natural language processing tasks with less training data and time. However, they struggle when …


Pre-training under infinite compute

K Kim, S Kotha, P Liang, T Hashimoto - arXiv preprint arXiv:2509.14786, 2025

Since compute grows much faster than web text available for language model pre-training, we ask how one should approach pre-training under fixed data and no compute constraints. We first show that existing data-constrained approaches of increasing …


Predicting LLM Reasoning Performance with Small Proxy Model

W Koh, J Suk, S Han, SY Yun, J Shin - arXiv preprint arXiv:2509.21013, 2025

Given the prohibitive cost of pre-training large language models, it is essential to leverage smaller proxy models to optimize datasets before scaling up. However, this approach becomes challenging for reasoning capabilities, which exhibit emergent …


Predictive Data Selection: The Data That Predicts Is the Data That Teaches

K Shum, Y Huang, H Zou, D Qi, Y Liao, X Chen, Q Liu… - arXiv preprint arXiv …, 2025

Language model pretraining involves training on extensive corpora, where data quality plays a pivotal role. In this work, we aim to directly estimate the contribution of data during pretraining and select pretraining data in an efficient manner. Specifically …


Preference Curriculum: LLMs Should Always Be Pretrained on Their Preferred Data

X Zhang, L Xu, F Duan, Y Zhou, S Wang, J Wang, X Cai - arXiv preprint arXiv …, 2025

Current large language models (LLMs) generally utilize a consistent data distribution throughout the entire pretraining process. However, as the model's ability improves, it intuitively should be pretrained with differentiated data. To …


Preprint: Did I Just Browse A Website Written by LLMs?

R Govindan, HV Madhyastha - arXiv e-prints, 2025

S He, R Govindan, HV Madhyastha - arXiv preprint arXiv:2507.13933, 2025

… Common Crawl. To understand the historical trend, we analyzed 10,479 random sites from Common Crawl archives from 2020 to 2025 (284,523 pages). Overall, only 451 sites (4.30%) are detected as LLM(-dominant), much lower than the 9.84 …


Prepublication Draft

… Whether trained on a highly curated photo collection or the billion web pages of the Common Crawl, large language models start by atomizing content in the archive and then compressing it into an engine that can produce new artefacts derived from …


Pretraining GPT-style models in Hungarian

K Szentmihályi, DM Nemeskey, AM Szekeres…

… We compiled our web text corpus from all Common Crawl dumps until the end of 2023. We followed the procedure outlined in [25] with a … We compiled our web text corpus from all Common Crawl dumps until the end of 2023. We followed the …


Primus: A Pioneering Collection of Open-Source Datasets for Cybersecurity LLM Training

YC Yu, TH Chiang, CW Tsai, CM Huang, WK Tsao - arXiv preprint arXiv:2502.11191, 2025

Large Language Models (LLMs) have shown remarkable advancements in specialized fields such as finance, law, and medicine. However, in cybersecurity, we have noticed a lack of open-source datasets, with a particular lack of high-quality …


PRIMUS: A Pioneering Collection of Open-Source Datasets for


Prior-based Noisy Text Data Filtering: Fast and Strong Alternative For Perplexity

Y Seo, G Kim, J Kim, J Yeo - arXiv preprint arXiv:2509.18577, 2025

… Among these, Common Crawl accounts for the major portion (74.5%) of the corpus. This makes it a particularly suitable environment for evaluating filtering methods, as it contains a high proportion of noisy web content that must be …


Privacy Ripple Effects from Adding or Removing Personal Information in Language Model Training

J Borkar, M Jagielski, K Lee, N Mireshghallah… - arXiv preprint arXiv …, 2025

Due to the sensitive nature of personally identifiable information (PII), its owners may have the authority to control its inclusion or request its removal from large-language model (LLM) training. Beyond this, PII may be added or removed from training …


Privacy-Preserving Transformers: SwiftKey's Differential Privacy Implementation

A Abouelenin, M Abdelrehim, R Fahim, A Hendy… - arXiv preprint arXiv …, 2025

In this paper we train a transformer using differential privacy (DP) for language modeling in SwiftKey. We run multiple experiments to balance the trade-off between the model size, run-time speed and accuracy. We show that we get small and …


Probabilistic Orthogonal Decay for Gradient Alignment Modulation in Large Language Model Pretraining

J Harrison, A Delta, R Green, C Simpson, A Scolto…

… The pretraining dataset consisted of a 1.1 trillion token corpus drawn from publicly available Common Crawl and academic benchmarks, filtered for deduplication, language coverage, and quality through a tiered ranking system based on perplexity …


Procedural history

In 2012, same-sex couple Charlie Craig and David Mullins from Colorado made plans to be lawfully married in Massachusetts and return to Colorado to celebrate with their family and friends. At that time the state constitution prohibited same-sex …


Profiling and optimization of multi-card GPU machine learning jobs

M Lawenda, K Khloponin, K Samborski, Ł Szustak - arXiv preprint arXiv:2505.22905, 2025

The effectiveness and efficiency of machine learning methodologies are crucial, especially with respect to the quality of results and computational cost. This paper discusses different model optimization techniques, providing a comprehensive …


Progress in the Application of Artificial Intelligence in English Corpus Pattern Recognition

Y Song, G Shan - 2025 3rd International Conference on Data Science …, 2025

… The model adopts a Transformer-based bidirectional encoding architecture, learns context representation based on the 1.2TB Common Crawl dataset in the pre-training phase, introduces a domain adaptation layer in the fine-tuning phase to handle …


Progressive Depth Up-scaling via Optimal Transport

M Cao, X Wang, N Aletras - arXiv preprint arXiv:2508.08011, 2025

Scaling Large Language Models (LLMs) yields performance gains but incurs substantial training costs. Depth up-scaling offers training efficiency by adding new layers to pre-trained models. However, most existing methods copy or average …


Prompt-Based Out-of-Distribution Intent Detection

R Chow, AYS Lam - IEEE Transactions on Emerging Topics in …, 2025

… While one could augment with other public datasets, the coverage is still nowhere as broad as the coverage of the pre-training corpus (ie the entire Wikipedia and common crawl). A by-product is that this method can be more easily adapted to other …


Propagating machine translation traits to predict potential impact on the target language

N Aranberri, JA Pascual - Natural Language Processing

… The texts are collected from web data released by the Common Crawl project.a We acknowledge that this collection might include both original and translated texts for any of the languages involved. In any case, we can argue that it is a rather …


Pruning Weights but Not Truth: Safeguarding Truthfulness While Pruning LLMs

Y Fu, R Li, X Long, H Yu, X Han, Y Yin, P Li - arXiv preprint arXiv:2509.00096, 2025

Neural network pruning has emerged as a promising approach for deploying LLMs in low-resource scenarios while preserving downstream task performance. However, for the first time, we reveal that such pruning disrupts LLMs' internal activation …


PsOCR: Benchmarking Large Multimodal Models for Optical Character Recognition in Low-resource Pashto Language

I Haq, Y Zhang, IA Khan - arXiv preprint arXiv:2505.10055, 2025

This paper evaluates the performance of Large Multimodal Models (LMMs) on Optical Character Recognition (OCR) in the low-resource Pashto language. Natural Language Processing (NLP) in Pashto faces several challenges due to the cursive …


Psy-Insight: Explainable Multi-turn Bilingual Dataset for Mental Health Counseling

K Chen, Z Sun, Y Wen, H Lian, Y Gao, Y Li - arXiv preprint arXiv:2503.03607, 2025

… We collect datasets from crawled blogs and books and also extract conversation from raw common crawl datasets including the book3 and Massive Never-ending BT Vast Chinese corpus project. All the copyright information and data sources are …


Pula: Training Large Language Models for Setswana

In this work we present Pula, a suite of bilingual language models proficient in both Setswana and English. Leveraging recent advancements in data availability and efficient fine-tuning, Pula 8B and Pula 14B outperform GPT-4o and Gemini 1.5 Pro …


Pushing the Boundaries of Large Language Models: Innovations and Limitations in NLP, Finance, and Mathematics

Large Language Models (LLMs) have emerged as transformative tools across a spectrum of domains, yet their practical deployment reveals a blend of remarkable potential and notable limitations. This research explores innovative methodologies …


Q-Adam-mini: Memory-Efficient 8-bit Quantized Optimizer for Large Language Model Training

Y Han, C Yang, C Chen, X Wang, R Sun - ES-FoMo III: 3rd Workshop on Efficient Systems for …

We propose $\textbf{Q-Adam-mini}$, a memory-efficient optimizer for Large Language Model (LLM) training that achieves $\textbf{8$\times$}$ reduction in GPU memory usage while maintaining performance parity with full-precision AdamW …


Quality Beyond A Glance: Revealing Large Quality Differences Between Web-Crawled Parallel Corpora

R Van Noord, M Esplà-Gomis, M Chichirău… - Proceedings of the 31st …, 2025

… CCAligned This corpus was created through URL-based document alignment on a collection of 68 Common Crawl Snapshots. Document … through sentence alignment on a collection of documents from 10 Common Crawl Snapshots. FastText …


Quality over Quantity: Boosting Data Efficiency Through Ensembled Multimodal Data Curation

J Xu, Y Song, D Wang, W Zhao, M Chen, K Chen, Q Li - arXiv preprint arXiv …, 2025

In an era overwhelmed by vast amounts of data, the effective curation of web-crawl datasets is essential for optimizing model performance. This paper tackles the challenges associated with the unstructured and heterogeneous nature of such …


Quantifying, Understanding, and Improving Generalization in Deep Learning

Generalization is a defining challenge of modern machine learning. Classical theory explains small supervised models but struggles with the surprising behavior of over-parameterized neural networks and with other paradigms such as reinforcement learning and large-scale …


Quantizing Large Language Models for Code Generation: A Differentiated Replication

A Giagnorio, A Mastropaolo, S Afrin, M Di Penta… - arXiv preprint arXiv …, 2025

Large Language Models (LLMs) have shown an impressive capability in code generation and, specifically, to automatically implement requirements described in natural language. The LLM effectiveness generally increases with its size: The …


Quantum leap in medical

S Chokkakula, S Chong, B Yang, H Jiang, J Yu, R Han… - 2025

… These models were trained on massive datasets, with GPT-3 using 45 terabytes of text data from various sources including Common Crawl, WebText2, Books1, Books2, and Wikipedia (22). This vast and multifarious dataset permit the models to …


Quantum-Enhanced Attention Mechanism in NLP: A Hybrid Classical-Quantum Approach

SM Tomal, AA Shafin, D Bhattacharjee, MD Amin… - arXiv preprint arXiv …, 2025

Transformer-based models have achieved remarkable results in natural language processing (NLP) tasks such as text classification and machine translation. However, their computational complexity and resource demands pose challenges for …


Query Details

Cyberbullying has emerged as a significant concern in the modern world. In Bangladesh receiving hate comments and bullying on social media platforms, particularly on Facebook, has unfortunately become a common occurrence. As a low-resource …


Query Smarter, Trust Better? Exploring Search Behaviours for Verifying News Accuracy

D Elsweiler, S Ateia, M Bink, G Donabauer, MF Pichel… - arXiv preprint arXiv …, 2025

While it is often assumed that searching for information to evaluate misinformation will help identify false claims, recent work suggests that search behaviours can instead reinforce belief in misleading news, particularly when users generate …


Quotegraph: A Social Network Extracted from Millions of News Quotations

M Čuljak, R West, A Spitz, A Arora - arXiv preprint arXiv:2507.17626, 2025

We introduce Quotegraph, a novel large-scale social network derived from speaker-attributed quotations in English news articles published between 2008 and 2020. Quotegraph consists of 528 thousand unique nodes and 8.63 million directed edges, pointing …


Qwen 2.5: A Comprehensive Review of the Leading Resource-Efficient LLM with potentioal to Surpass All Competitors

I Ahmed, S Islam, PP Datta, I Kabir, NUR Chowdhury…

The purpose of the review is to provide a comprehensive analysis of Qwen 2.5, highlighting its advancements in AI models. Key findings indicate that Qwen 2.5 features significant improvements in dataset size (expanding from 7 trillion to 18 …


Qwen2. 5-1M Technical Report

A Yang, B Yu, C Li, D Liu, F Huang, H Huang, J Jiang… - 2025

In this report, we introduce Qwen2. 5-1M, a series of models that extend the context length to 1 million tokens. Compared to the previous 128K version, the Qwen2. 5-1M series have significantly enhanced long-context capabilities through long-context …


RAG in the Wild: On the (In) effectiveness of LLMs with Mixture-of-Knowledge Retrieval Augmentation

R Xu, Y Zhuang, Y Yu, H Wang, W Shi, C Yang - arXiv preprint arXiv:2507.20059, 2025

… 2024) – a large-scale, multi-domain datastore that combines general web sources (eg, CommonCrawl) with specialized domains (eg, PubMed). We evaluate tasks spanning both general knowledge and domain-specific QA, where no prior …


RAG-sec bot: Orchestrating compliance and portability by leveraging localized LLMs in contextualized dialogue systems

A Pal, MG Mathew, AVG Moorthy, CVS Babu - AIP Conference Proceedings, 2025

The design objective of a conversational agent, is to mimic the colloquial comportment and discourse patterns exhibited by human interlocutors, primarily via textual modality. To maintain its relevance and efficacy, a chatbot necessitates …


Ranking Generated Answers

S Heineking, J Probst, D Steinbach, M Potthast…

… We therefore obtained only the original web documents from CommonCrawl, discarded those containing fewer than 50 characters in the HTML body, and extracted plain text using the Resiliparse library.We were able to restore 6,692 web …


ReaderLM-v2: Small Language Model for HTML to Markdown and JSON

F Wang, Z Shi, B Wang, N Wang, H Xiao - arXiv preprint arXiv:2503.01151, 2025

We present ReaderLM-v2, a compact 1.5 billion parameter language model designed for efficient web content extraction. Our model processes documents up to 512K tokens, transforming messy HTML into clean Markdown or JSON formats with …


Reading and Writing at a Distance: Integrating Corpus and AI Literacy in the Classroom

K Löser - International Conference on Artificial Intelligence in …, 2025

This paper presents a conceptual framework that fuses corpus-based analysis (distant reading) with generative AI practice (distant writing) to advance critical digital literacy in secondary and tertiary classrooms. Using accessible corpus tools—COCA, DWDS …


Real-TabPFN: Improving Tabular Foundation Models via Continued Pre-training With Real-World Data

A Garg, M Ali, N Hollmann, L Purucker, S Müller… - 1st ICML Workshop on Foundation …

… accuracy compared to using broader, potentially noisier corpora like CommonCrawl or GitTables. Our resulting model, Real-TabPFN, … The prevalence of smaller datasets in broad corpora like CommonCrawl and GitTable contrasts with …


Real-time Monitoring of Economic Shocks using Company Websites

M Koenig, J Rauch, M Woerter - arXiv preprint arXiv:2502.17161, 2025

… We use the CommonCrawl dataset to access historical information from company websites. CommonCrawl is an extensive and constantly updated collection of web data that covers a large part of the web content and allows access to the historical …


RealSyn: An Effective and Scalable Multimodal Interleaved Document Transformation Paradigm

T Gu, K Yang, C Zhang, Y Xie, X An, Z Feng, D Liu… - arXiv preprint arXiv …, 2025

… 2024) dataset uses a comprehensive filtering strategy and includes 141 million web pages, 353 million associated images, and 115 billion text tokens extracted from Common Crawl. However, due to data format constraints and training …


Reasoning Beyond Limits: Advances and Open Problems for LLMs

MA Ferrag, N Tihanyi, M Debbah - arXiv preprint arXiv:2503.22732, 2025

… 7B architecture, the model is further pre-trained on an extensive corpus of 120 billion math-related tokens extracted from Common Crawl, complemented by natural language and code data. As a result, DeepSeekMath 7B achieves an impressive …


Rebranding empire in the age of generative AI

D Lakshmi S - Frontiers in Communication, 2025

… These datasets are scraped from Wikipedia articles, Reddit forums, and Common Crawl archives. But whose knowledge is scraped? Which languages are missing? …


Recent Trends on Artificial Intelligence in Automated Hate Speech Detection

N Goyal, A Kumar, A Chaddha, D Lakshmi - Ethical AI Solutions for Addressing Social …, 2025

This study investigates the performance of AI in detecting HS in diverse cultural and contextual settings. Existing AI models, trained primarily on English datasets, struggle with regional dialects, idiomatic phrases, and cultural nuances. A …


Reconstructing Reason in AI: An ESSIM Model to Address Structural Failures in

Contemporary large language models (LLMs) have demonstrated unprecedented fluency in natural language generation, yet their foundational architectures suffer from structural epistemic failures. This manuscript identifies eight core deficiencies—ranging …


Recreating Neural Activity During Speech Production with Language and Speech Model Embeddings

OM Khanday, PRS Esteban, ZA Lone, M Ouellet… - arXiv preprint arXiv …, 2025

Understanding how neural activity encodes speech and language production is a fundamental challenge in neuroscience and artificial intelligence. This study investigates whether embeddings from large-scale, self-supervised language and …


Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models

T Nguyen, Y Li, O Golovneva, L Zettlemoyer, S Oh… - arXiv preprint arXiv …, 2025

… 2024), made publicly available by Common Crawl. … We start with web documents from Common Crawl that has undergone some filtering (ie, RefinedWeb heuristics (… 2024), Common Crawl data that has passed the initial rule-based …


RefineX: Learning to Refine Pre-training Data at Scale from Expert-Guided Programs

B Bi, S Liu, X Ren, D Liu, J Lin, Y Wang, L Mei, J Fang… - arXiv preprint arXiv …, 2025

… Since raw web data—often from Common Crawl—is noisy and inconsistent, most LLM pipelines apply extensive preprocessing (Touvron … A critical analysis of the largest source for generative ai training data: Common crawl. In Proceedings of the …


Refining Czech GEC: Insights from a Multi-Experiment Approach

P Pechman, M Straka, J Straková, J Náplava - arXiv preprint arXiv:2506.22402, 2025

… Tasks texts from Common Crawl [4], the SYN v4 corpus [7], the News 2019 corpus [1], and the Wikipedia corpus presented within DaMuEL [9]. … We attribute the lowest performance of Common Crawl to its relatively high noisiness, while the …


REFRAG: Rethinking RAG based Decoding

X Lin, A Ghosh, BKH Low, A Shrivastava, V Mohan - arXiv preprint arXiv:2509.01092, 2025

Large Language Models (LLMs) have demonstrated remarkable capabilities in leveraging extensive external knowledge to enhance responses in multi-turn and agentic applications, such as retrieval-augmented generation (RAG). However …


Reframing the performance and ethics of “empathic” AI: Wisdom of the crowd and placebos

Recently, claims have emerged that artificial intelligence (AI) is better at providing empathy than humans. These claims come paired with suggestions that people should use empathic AI to supplement human empathy. This paper critically …


Register Always Matters: Analysis of LLM Pretraining Data Through the Lens of Language Variation

A Myntti, E Henriksson, V Laippala, S Pyysalo - arXiv preprint arXiv:2504.01542, 2025

… (2020) validated the quality of their Pile datasets by evaluating models trained on the Pile, CommonCrawl, and CC-100. Likewise, Burchell et al. (2025… HPLT v2 datasets have been processed from a combination of Internet Archive and Common …


Reimagining Unit Test Generation with AI: A Journey from Evolutionary Models to Transformers

SZ Esubalew, BG Assefa - IEEE Access, 2025

The rapid evolution of software development demands efficient and scalable unit testing methodologies to ensure software reliability. Traditional manual test case generation is time-consuming and often inadequate for modern agile workflows …


Reinforced Disentangled HTML Representation Learning with Hard-Sample Mining for Phishing Webpage Detection

JH Yoon, SJ Buu, HJ Kim - Electronics, 2025

… The datasets used in this study include benign data from Common Crawl and phishing data from Phishtank and Mendeley Data, as summarized in Table 3. The benign dataset, collected in February 2023, contains 1,048,575 instances, providing …


Relationships between Urban Growth and Intercity Networks based on Toponym Co-occurrences

B Lee, H Shin, M Woo - Journal of the Korean Regional Science Association, 2025

It is crucial to comprehend the operational principles of intercity networks, as the hub-centered network model to enhance regional competitiveness is being advocated such as a megaregion plan. However, traditional urban network analysis as a factor of urban …


Reparameterized LLM Training via Orthogonal Equivalence Transformation

Z Qiu, S Buchholz, TZ Xiao, M Dax, B Schölkopf, W Liu - arXiv preprint arXiv …, 2025

While large language models (LLMs) are driving the rapid advancement of artificial intelligence, effectively and reliably training these large models remains one of the field's most significant challenges. To address this challenge, we propose POET, a …


Representation Learning for Tabular Data: A Comprehensive Survey

JP Jiang, SY Liu, HR Cai, Q Zhou, HJ Ye - arXiv preprint arXiv:2504.16109, 2025

Tabular data, structured as rows and columns, is among the most prevalent data types in machine learning classification and regression applications. Models for learning from tabular data have continuously evolved, with Deep Neural Networks (DNNs) …


Representation Learning Methods for Association Prediction Tasks in Drug Discovery

Abstract Representation learning is a key step in bridging machine learning and drug discovery. Understanding the interactions between drugs and various biological entities is critical for drug discovery. In this research, we explore advanced …


Representations, Retrieval, and Evaluation in Knowledge-Intensive Natural Language Processing

Several major advancements have recently been made within the field of Natural Language Processing (NLP). Nowadays, NLP systems based on language models (LMs) are readily available to the public in the form of chatbots, code assistants, writing …


Research Challenges and Opportunities for Open Generative Modeling

This dissertation develops methods to make generative modeling more accessible, reliable, and legally grounded across vision, biology, and language. I introduce CommonCanvas, an open latent diffusion pipeline trained solely on Creative-Commons-licensed …


Research on Key Methods for Extracting High-Quality Chinese Corpus Based on Common Crawl

L Xiao, Z Zhao, C Wang, J Zhang, F Liu - 2025 28th International Conference on …, 2025

… This paper introduces a method to extract high-quality Chinese corpora based on the Common Crawl dataset, enhancing existing quality filtering and deduplication methods. In terms of deduplication, this paper enhances the Simhash algorithm to …


ResearchQA: Evaluating Scholarly Question Answering at Scale Across 75 Fields with Survey-Mined Questions and Rubrics

LS Yifei, A Chang, C Malaviya, M Yatskar - arXiv preprint arXiv:2509.00496, 2025

Evaluating long-form responses to research queries heavily relies on expert annotators, restricting attention to areas like AI where researchers can conveniently enlist colleagues. Yet, research expertise is widespread: survey articles synthesize …


Responsible AI and AI Governance

T Duke, P Giudici - Responsible AI in Practice: A Practical Guide to Safe …, 2025

Responsible AI is a new and nascent field, and the term “responsible AI” has been used interchangeably with the term “ethical AI” in recent years. In this chapter, we’ll look at a brief history of responsible AI and the factors influencing its emergence as …


Responsible AI in Practice

… Sourced from the Common Crawl web index, LAION-5B is a popular open source training dataset containing over 5.8 billion images used for image generation. It was used to train Stable Diffusion, an image generator introduced by Stability AI (a UK-based …


Restoring Rhythm: Punctuation Restoration Using Transformer Models for Bangla, a Low-Resource Language

MO Mamun, MA Mamun, A Ahmad, MIH Emu - arXiv preprint arXiv:2507.18448, 2025

Punctuation restoration enhances the readability of text and is critical for post-processing tasks in Automatic Speech Recognition (ASR), especially for low-resource languages like Bangla. In this study, we explore the application of transformer-based …


Retention analysis of edited knowledge after fine-tuning

F Wen, S Zhang - arXiv preprint arXiv:2507.14198, 2025

… Specifically, for our experiments on the GPT-2 XL, which was pre-trained on webtext, we choose the Common Crawl dataset for fine-tuning. Since a small subset of the data suffices to demonstrate the influence, we sampled 60k data for our …


Rethinking Data Mixture for Large Language Models: A Comprehensive Survey and New Perspectives

Y Liu, C Chen, J Yang, R Sun - arXiv preprint arXiv:2505.21598, 2025

Training large language models with data collected from various domains can improve their performance on downstream tasks. However, given a fixed training budget, the sampling proportions of these different domains significantly impact the …


Rethinking Fingerprinting: An Assessment of Behavior-based Methods at Scale and Implications for Web Tracking

K Crichton, LF Cranor, N Christin - … on Privacy Enhancing Technologies YYYY (X)

Most common forms of web tracking fail to maintain the continuity of a user’s identity over long periods of time: cookies get deleted, IP addresses are reassigned, attributes used for browser fingerprinting change. These identity discontinuities help …


Rethinking Multilingual Continual Pretraining: Data Mixing for Adapting LLMs Across Languages and Resources

Z Li, S Ji, H Luo, J Tiedemann - arXiv preprint arXiv:2504.04152, 2025

… 2024), a large-scale multilingual dataset derived from Common Crawl5, covering 419 languages. Since web-crawled text does not inherently guarantee monolingual integrity, we employ GlotLID (Kargaran et al.… 5https://commoncrawl.org/ …


Retrieval-Augmented Purifier for Robust LLM-Empowered Recommendation

L Ning, W Fan, Q Li - arXiv preprint arXiv:2504.02458, 2025

… [66] introduced CCNet, an automated pipeline designed to efficiently extract vast amounts of high-quality monolingual datasets from the Common Crawl corpus across various languages. Beyond enhancing the quality of training data …


Retrieval-augmented visual parcel invoice understanding transformer for address correction

YB Jeong, H Seo, YH Kim, WY Kim - Engineering Applications of Artificial Intelligence, 2025

In automated postal and logistics operations, Visual Parcel Invoice Understanding technology (VPIU) addresses the challenging task of recognizing named entities such as addresses, names, and phone numbers. In particular, the VPIU performance …


Retrieving the spatial layout of medium-scale geographical maps through distributional semantics

G Anceresi, D Gatti, T Vecchi, M Marelli, L Rinaldi - Neuropsychologia, 2025

Recent evidence has indicated that spatial representations, such as large-scale geographical maps, can be retrieved from natural language alone through cognitively plausible distributional-semantic models, which capture word meanings …


Retrofitting Language Models with Dynamic Tokenisation

Large Language Models (LLMs) are the backbone of modern Natural Language Processing (NLP) applications. They typically rely on subword tokenisation, breaking text into pieces of words or entire words, for efficient processing. Although …


Revealing Depression through Social Media via Adaptive Gated Cross-Modal Fusion Augmented with Insights from Personality Traits

GA Pradnyana, W Anggraeni, EM Yuniarno… - IEEE Access, 2025

… Trained on general-domain corpora such as BookCorpus and Common Crawl, RoBERTa demonstrates strong performance in various natural language understanding tasks by learning robust semantic representations. Furthermore, we also incorporate …


Reverse Browser: Vector-Image-to-Code Generator

Z Toth-Czifra - arXiv preprint arXiv:2509.05394, 2025

Automating the conversion of user interface design into code (image-to-code or image-to-UI) is an active area of software engineering research. However, the state-of-the-art solutions do not achieve high fidelity to the original design, as evidenced by …


Review of LLMs Applications in Electrical Power & Energy Systems

F Amjad, T Korotko, A Rosin - IEEE Access, 2025

This paper presents a comprehensive review of the applications, challenges, and future directions of Large Language Models (LLMs) in the Electrical Power Domain (EPD). Leveraging transformer-based architectures such as GPT, BERT, and LLaMA, LLMs …


Revisiting Chain-of-Thought in Code Generation: Do Language Models Need to Learn Reasoning before Coding?

RB Liu, A Li, C Yang, H Sun, M Li - Forty-second International Conference on Machine …

Large Language Models (LLMs) have demonstrated exceptional performance in code generation, becoming increasingly vital for software engineering and development. Recently, Chain-of-Thought (CoT) has proven effective for complex …


REVISITING DATA MIXING THROUGH THE LENS OF MULTI-OBJECTIVE OPTIMIZATION

Effective pretraining of large language models (LLMs) relies significantly on the strategic composition of training data from various sources. Traditional domain weighting approaches often focus on minimizing either average empirical loss or …


Revisiting Language Models in Neural News Recommender Systems

Y Zhao, J Huang, D Vos, M de Rijke - arXiv preprint arXiv:2501.11391, 2025

… This may be due to GloVe’s pre-training on the Common Crawl web data [22], likely making it more suited to news content than BERT models trained on BookCorpus and Wikipedia [4]. The effectiveness of GloVe in representing news …


Revisiting Replay and Gradient Alignment for Continual Pre-Training of Large Language Models

I Abbes, G Subbaraj, M Riemer, N Islah, B Therien… - arXiv preprint arXiv …, 2025

… 2024): this is a high-quality English text corpus derived from the CommonCrawl We sampled 100B tokens out of the 240T token standardized corpus. Tasks B and C are French and German respectively drawn from a subset of the 166 language Open …


Revisiting Scaling Laws for Language Models: The Role of Data Quality and Training Strategies

Z Chen, S Wang, T Xiao, Y Wang, S Chen, X Cai, J He… - Proceedings of the 63rd …, 2025

… Figure 6: The relationship between number of samples and cluster ID in Common Crawl dataset, with cluster IDs sorted in descending order by the number of samples. Points are sampled for illustration. In the figure, "Raw data" refers to the original …


Rewriting Pre-Training Data Boosts LLM Performance in Math and Code

K Fujii, Y Tajima, S Mizuki, H Shimada, T Shiotani… - arXiv preprint arXiv …, 2025

The performance of large language models (LLMs) in program synthesis and mathematical reasoning is fundamentally limited by the quality of their pre-training corpora. We introduce two openly licensed datasets, released under the Llama 3.3 …


RIP: Better Models by Survival of the Fittest Prompts

P Yu, W Yuan, O Golovneva, T Wu, S Sukhbaatar… - arXiv preprint arXiv …, 2025

Training data quality is one of the most important drivers of final model quality. In this work, we introduce a method for evaluating data integrity based on the assumption that low-quality input prompts result in high variance and low quality responses. This …


Risk Analysis Techniques for Governed LLM-based Multi-Agent Systems

A Reid, S O'Callaghan, L Carroll, T Caetano - 2025

Organisations are starting to adopt AI agents based on large language models to automate complex tasks, with deployments evolving from single agents towards multi-agent systems. While this promises efficiency gains, multi-agent systems fundamentally …


Risk Assessment and Security Analysis of Large Language Models

X Zhang, D Lyu, X Li - arXiv preprint arXiv:2508.17329, 2025

… Initially, they are parameterised using massive datasets such as OpenWebText and Common Crawl. Subsequently, by continuously expanding parameter scales as seen in models like the GPT series, PaLM, and LLaMA, they progressively expand …


RiskHarvester: A Risk-based Tool to Prioritize Secret Removal Efforts in Software Artifacts

SK Basak, T Pardeshi, B Reaves, L Williams - arXiv preprint arXiv:2502.01020, 2025

… In our study, we used the pre-trained fastText model cc.en.300.bin, trained on Common Crawl and Wikipedia with 5-character n-grams, a window size of 5, and 10 negatives. We used the fasttext [10] package of Python to access the model and …


Robust Bias Detection in MLMs and its Application to Human Trait Ratings

I Shrestha, L Tay, P Srinivasan - arXiv preprint arXiv:2502.15600, 2025

… The MLMs assessed are trained on datasets up to 2019 from sources like Common Crawl, BookCorpus, and Wikipedia. These likely lack adequate … Key to note is that RoBERTa’s training corpus is 50% news data from Common Crawl (CC-News) …


Robust LLM Fingerprinting via Domain-Specific Watermarks

T Gloaguen, R Staab, N Jovanović, M Vechev - arXiv preprint arXiv:2505.16723, 2025

As open-source language models (OSMs) grow more capable and are widely shared and finetuned, ensuring model provenance, ie, identifying the origin of a given model instance, has become an increasingly important issue. At the same time …


Robust, efficient, and knowledge-augmented text generation with pre-trained language models

Pre-trained Language Models (PLMs) have significantly advanced the field of text generation. However, their practical application is often hindered by challenges related to systematic capability evaluation, high computational costs for training and …


RoFL: Robust Fingerprinting of Language Models

YY Tsai, C Guo, J Yang, L van der Maaten - arXiv preprint arXiv:2505.12682, 2025

… For instance, she can extract a likely query-response pair from the Common Crawl dataset that may match many model lineages. As §4 demonstrates, ROFL learns unique fingerprints for a model lineage. In fingerprint verification, ROFL …


Roles and Potential of Large Language Models in Healthcare: A Comprehensive Review

C Lin, CF Kuo - Biomedical Journal, 2025

Large Language Models (LLMs) are capable of transforming healthcare by demonstrating remarkable capabilities in language understanding and generation. They have matched or surpassed human performance in standardized medical …


RSTHFS: A Rough Set Theory-Based Hybrid Feature Selection Method for Phishing Website Classification

JH Setu, N Halder, A Islam, MA Amin - IEEE Access, 2025

Phishing is a pervasive form of cybercrime where malicious websites deceive users into revealing sensitive information, eg, passwords and credit card details. Despite advances in cybersecurity, accurately detecting phishing websites remains …


RUAccent: Advanced System for Stress Placement in Russian with Homograph Resolution

DA Petrov - Proceedings of the 31st International Conference on …, 2025

This paper presents a novel approach to the problem of stress placement in Russian text, with a particular focus on resolving homographs. We introduce a comprehensive system that combines morphological analysis, context-aware neural …


S-DAT: A Multilingual, GenAI-Driven Framework for Automated Divergent Thinking Assessment

J Haase, PHP Hanel, S Pokutta - arXiv preprint arXiv:2505.09068, 2025

This paper introduces S-DAT (Synthetic-Divergent Association Task), a scalable, multilingual framework for automated assessment of divergent thinking (DT) -a core component of human creativity. Traditional creativity assessments are often labor-intensive …


Safeguarding Patient Data: Machine Learning for Phishing URL Detection in Healthcare Systems

AA Mousa, SADH Hassan, MK Rashid, M Al-Saady

… Benign URLs for validation were sourced from a 2023 snapshot of the Common Crawl dataset, representing a broad spectrum of contemporary web content. Phishing URLs were aggregated from PhishTank (live phishing feed) and …


Safety and Security Analysis of Large Language Models: Risk Profile and Harm Potential

C Akiri, H Simpson, K Aryal, A Khanna, M Gupta - arXiv preprint arXiv:2509.10655, 2025

While the widespread deployment of Large Language Models (LLMs) holds great potential for society, their vulnerabilities to adversarial manipulation and exploitation can pose serious safety, security, and ethical risks. As new threats continue to …


Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs

L Dou, Q Liu, F Zhou, C Chen, Z Wang, Z Jin, Z Liu… - arXiv preprint arXiv …, 2025

… For SEA language data that provide local text and knowledge, we extract content from 96 CommonCrawl snapshots spanning from summer 2013 to April 2024. Additionally, to extract high-quality and professional text, we also leverage publicly …


Salamandra Technical Report

A Gonzalez-Agirre, M Pàmies, J Llop, I Baucells… - arXiv preprint arXiv …, 2025

… In order to deal with the heterogeneous and noisy nature of web data, the Ungoliant pipeline [2] was used to produce the Colossal OSCAR corpus for the OSCAR project, from which we include 20 CommonCrawl snapshots7, originally in …


SampleMix: A Sample-wise Pre-training Data Mixing Strategey by Coordinating Data Quality and Diversity

X Xi, D Kong, J Yang, J Yang, Z Chen, W Wang… - arXiv preprint arXiv …, 2025

… Our findings reveal substantial overlap between domains—nearly all clusters contain samples from both CommonCrawl and C4. Furthermore, manual inspection of the clustered samples confirms that data from different domains frequently share …


SaudiCulture: A Benchmark for Evaluating Large Language Models Cultural Competence within Saudi Arabia

L Ayash, H Alhuzali, A Alasmari, S Aloufi - arXiv preprint arXiv:2503.17485, 2025

Large Language Models (LLMs) have demonstrated remarkable capabilities in natural language processing; however, they often struggle to accurately capture and reflect cultural nuances. This research addresses this challenge by focusing on …


Scalability analysis of language model training for clinical domain in HPC

The explosive growth of Large Language Models (LLMs) and their computational demands have made High-Performance Computing (HPC) essential components for efficient training. This thesis analyzes the scalability of training the XLM-RoBERTa-base …


Scalability of Generative AI Models: Challenges and Opportunities in Large-Scale Data Generation and Training

Generative models of artificial intelligence have revolutionized many sectors of society, allowing machines to create content similar to humans in fields ranging from text, images and music to code. Many creative ones are possible based on deep …


Scalable and Interpretable Conjugate Gradient Techniques

The widespread use of neural networks and their increasing complexity necessitate effective training algorithms to optimize their performance. While second-order methods like Scaled Conjugate Gradient (SCG) offer potential benefits by utilizing …


Scalable Machine Learning for Healthcare: Techniques, Applications, and Collaborative Frameworks

AE Benhmida, H Sakly, R Guetari, N Kraiem - Scalable Artificial Intelligence for …, 2025

This chapter provides an in-depth examination of strategies for scaling machine learning (ML) models tailored for healthcare applications. With the increasing availability of large-scale medical datasets, training sophisticated ML models has …


Scalable Private Partition Selection via Adaptive Weighting

JY Chen, V Cohen-Addad, A Epasto… - arXiv preprint arXiv …, 2025

In the differentially private partition selection problem (aka private set union, private key discovery), users hold subsets of items from an unbounded universe. The goal is to output as many items as possible from the union of the users' sets while …


Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents

Y Jang, Y Song, S Sohn, L Logeswaran, T Luo, DK Kim… - arXiv preprint arXiv …, 2025

… Our data collection process begins with CommonCrawl web posts, specifically utilizing the C4 [40] and Dolma [46] datasets. These web posts represent actual user discussions and questions about mobile OS tasks, providing a natural distribution of …


Scaling Agents via Continual Pre-training

L Su, Z Zhang, G Li, Z Chen, C Wang, M Song, X Wang… - arXiv preprint arXiv …, 2025

Large language models (LLMs) have evolved into agentic systems capable of autonomous tool use and multi-step reasoning for complex problem-solving. However, post-training approaches building upon general-purpose foundation …


Scaling Embedding Layers in Language Models

D Yu, E Cohen, B Ghazi, Y Huang, P Kamath, R Kumar… - arXiv preprint arXiv …, 2025

We propose SCONE ($\textbf{S}$calable, $\textbf{C}$ontextualized, $\textbf{O}$ffloaded, $\textbf{N}$-gram $\textbf{E}$mbedding), a method for extending input embedding layers to enhance language model performance as layer size scales. To avoid …


Scaling Language-Free Visual Representation Learning

D Fan, S Tong, J Zhu, K Sinha, Z Liu, X Chen… - arXiv preprint arXiv …, 2025

Visual Self-Supervised Learning (SSL) currently underperforms Contrastive Language-Image Pretraining (CLIP) in multimodal settings such as Visual Question Answering (VQA). This multimodal gap is often attributed to the semantics …


Scaling Laws for Optimal Data Mixtures

M Shukor, L Bethune, D Busbridge, D Grangier, E Fini… - arXiv preprint arXiv …, 2025

Large foundation models are typically trained on data from multiple domains, with the data mixture--the proportion of each domain used--playing a critical role in model performance. The standard approach to selecting this mixture relies on trial …


Scaling Laws for Speculative Decoding

S Yan, M Zhu, G Jiang, J Wang, J Chen, W Zhang… - arXiv preprint arXiv …, 2025

The escalating demand for efficient decoding in large language models (LLMs) is particularly critical for reasoning-intensive architectures like OpenAI-o3 and DeepSeek-R1, which depend on extended chain-of-thought reasoning. This study …


Scaling Laws for Upcycling Mixture-of-Experts Language Models

SP Liew, T Kato, S Takase - arXiv preprint arXiv:2502.03009, 2025

Pretraining large language models (LLMs) is resource-intensive, often requiring months of training time even with high-end GPU clusters. There are two approaches of mitigating such computational demands: reusing smaller models to train larger …


Scaling laws in zero-shot gender classification using CLIP

LM Ceschini, GO Ramos, CR Jung - Proceedings of the Computer Vision and Pattern …, 2025

… LAION-400M [18] is an open-source dataset with over 400 million text/image pairs scraped from the Common Crawl [7] web dump, filtered … a new candidate pool of 12.8 billion image-text pairs from Common Crawl [7], which they called CommonPool …


Scaling Low-Resource MT via Synthetic Data Generation with LLMs

O de Gibert, J Attieh, T Vahtola, M Aulamo, Z Li… - arXiv preprint arXiv …, 2025

We investigate the potential of LLM-generated synthetic data for improving low-resource machine translation (MT). Focusing on seven diverse target languages, we construct a document-level synthetic corpus from English Europarl, and extend it via pivoting …


Scaling Multi-Document Event Summarization: Evaluating Compression vs. Full-Text Approaches

A Pratapa, T Mitamura - arXiv preprint arXiv:2502.06617, 2025

Automatically summarizing large text collections is a valuable tool for document research, with applications in journalism, academic research, legal work, and many other fields. In this work, we contrast two classes of systems for large-scale multi-document …


Scaling Pre-training to One Hundred Billion Data for Vision Language Models

X Wang, I Alabdulmohsin, D Salz, Z Li, K Rong, X Zhai - arXiv preprint arXiv …, 2025

We provide an empirical investigation of the potential of pre-training vision-language models on an unprecedented scale: 100 billion examples. We find that model performance tends to saturate at this scale on many common Western-centric …


Scaling Up With Integrity: Valid and Efficient Narrative Policy Framework Analyses Using Large Language Models

KL Anglin, A Bertrand, J Gottlieb, J Elefante - Policy Studies Journal, 2025

Given vast quantities of digital and online data—such as news articles, congressional testimony, and social media posts—the potential for large scale narrative analyses has dramatically increased. Narrative Policy Framework (NPF) …


SCAN: Semantic Document Layout Analysis for Textual and Visual Retrieval-Augmented Generation

Y Dong, N Ueda, KĂĄ Boros, D Ito, T Sera, M Oyamada - arXiv preprint arXiv …, 2025

With the increasing adoption of Large Language Models (LLMs) and Vision-Language Models (VLMs), rich document analysis technologies for applications like Retrieval-Augmented Generation (RAG) and visual RAG are gaining significant attention. Recent research …


Scout: Leveraging Large Language Models for Rapid Digital Evidence Discovery

S Murtuza - arXiv preprint arXiv:2507.18478, 2025

… The training corpus typically consists of vast amount of Internet website data such as Common Crawl [6] dataset that consists of billions of crawled web pages. These models can achieve general purpose language generation and exemplary natural …


Scrapers selectively respect robots. txt directives: evidence from a large-scale empirical study

T Kim, K Bock, C Luo, A Liswood, E Wenger - arXiv preprint arXiv:2505.21733, 2025

… To collect data at scale, model trainers can either scrape the data themselves or rely on pre-collected datasets like Common Crawl [3]. Additionally, there are some datasets, like LAION 5B, which only provide URLs, requiring prospective users to …


Scraping the Shadows: Deep Learning Breakthroughs in Dark Web Intelligence

I Bakermans, D De Pascale, G Marcelino, G Cascavilla… - arXiv preprint arXiv …, 2025

… shared task (wiki dump + common crawl) for each language included in the system. The multi-language ELMo-embedding system in this study is trained on 20 million words sampled from shared task data (wiki dump + common crawl) for each …


Script-Agnosticism and its Impact on Language Identification for Dravidian Languages

M Agarwal, J Otten, A Anastasopoulos - Proceedings of the 2025 Conference of the …, 2025

Abstract Language identification is used as the first step in many data collection and crawling efforts because it allows us to sort online text into language-specific buckets. However, many modern languages, such as Konkani, Kashmiri, Punjabi etc., are …


SEA-LION: Southeast Asian Languages in One Network

R Ng, TN Nguyen, Y Huang, NC Tai, WY Leong… - arXiv preprint arXiv …, 2025

… 2025), as well as documents from CommonCrawl (CommonCrawl… For SEA-LION-Pilev2, we filter CommonCrawl WARC data for documents in SEA languages (ie, Burmese, Simplified Chinese, Indonesian, Khmer, Lao, Malay, Filipino, Tamil, Thai and …


SearchLab: Exploring Conversational and Traditional Search Interfaces in Information Retrieval

S Zerhoudi, M Granitzer - Proceedings of the 2025 ACM SIGIR Conference on …, 2025

Large Language Models (LLMs) have increased the popularity of conversational search systems and traditional search engines (SERPs), making the study of user behavior and search queries across various search systems a critical research area …


Security Alignment of Large Language Models via Jailbreaking Attacks

… of their resource availability from CommonCrawl 2. A language is categorized as a high-resource language if its data ratio on CommonCrawl is above 1%. A language is categorized as a medium-resource language if its data ratio on …


Security and Privacy Challenges of AIGC in Metaverse: A Comprehensive Survey

S Zhang, H Li, K Sun, H Chen, Y Wang, S Li - ACM Computing Surveys, 2025

The Metaverse is a hybrid environment that integrates both physical and virtual realms. The Metaverse has been accessible due to many facilitating technologies. One of the essential technologies that contribute to the Metaverse is AIGC. It is …


Seed-Coder: Let the Code Model Curate Data for Itself

Y Zhang, J Su, Y Sun, C Xi, X Xiao, S Zheng, A Zhang… - arXiv preprint arXiv …, 2025

… We implemented text extraction procedures on Common Crawl and identified two distinct categories of raw data: 1) web pages with explicit code tags (such as <code>. . . </code>) in HTML that are readily extractable using standard rules, and 2) non-explicit …


Seeing What Tastes Good: Revisiting Multimodal Distributional Semantics in the Billion Parameter Era

D Oneata, D Elliott, S Frank - Second Workshop on Visual Concepts

D Oneață, D Elliott, S Frank - Findings of the Association for Computational …, 2025

Accurate understanding of a concept includes representing the common attributes and affordances of that concept across multiple modalities. We investigate the ability of pre-trained vision models to represent the semantic attributes of concrete object …


Selected Research Articles

L Zampierin, F Frasincar, Y Xu, M Hassani

… In this research, we use the 300-dimensional GloVe word representations that were pre-trained on 42 billion tokens from Common Crawl [23]. This word embedding matrix contains representations for 1.9 million words. The reason for this choice is twofold. …


Self-Supervised 3D Representation Learning with Asymmetric Dual Self-Distillation for Point Clouds

Recognizing tree species from 3D LiDAR scans remains a challenge in large-scale forest inventory systems, particularly due to the limited availability and diversity of annotated training data. This thesis explores how self-supervised representation …


Self-supervised Domain Adaptation of Language Models for the Process Industry

Incorporating additional knowledge into pre-trained language models (PLMs) has proven to be highly effective in improving their performance in specialized fields. Graph structures, in particular, allow models to capture domain-specific relationships …


Semantic alignment: A measure to quantify the degree of semantic equivalence for English–Chinese translation equivalents based on distributional semantics

Y Liu, S Chen, Y Yang - Behavior Research Methods, 2025

… These models were trained on the concatenation of Common Crawl and Wikipedia using the CBOW method, with 300 dimensions, character n-… The obtained word vector sizes from fastText derived from Chinese and English …


Semantic Annotation Model and Method Based on Internet Open Dataset

X Gao, Y Wang, F Wang, B Zhang, C Hu, J Wang, L Ma - International Journal of …, 2025

… This paper selects Common Crawl dataset to provide sufficient training samples; methods such as removing stop words and deduplication are used to preprocess data to improve data quality; a keyword extraction model based on heuristic rules …


Semantic Mastery: Enhancing LLMs with Advanced Natural Language Understanding

M Hariharan - arXiv preprint arXiv:2504.00409, 2025

Large language models (LLMs) have greatly improved their capability in performing NLP tasks. However, deeper semantic understanding, contextual coherence, and more subtle reasoning are still difficult to obtain. The paper discusses state-of-the-art …


Semantic verbal fluency assessment using computational analysis in the Czech language

J Pesek, H Horakova, M Vyhnalek… - Applied Neuropsychology …, 2025

… Vector word embeddings were derived from the Czech fasttext library, which is trained on Common Crawl and Wikipedia using fastText. This model was trained using CBOW with position-weights, in dimension 300, with character n-grams of …


Semantics of productively formed regular derivatives in contextual use: the case of the Latvian agentive suffix-tāj

Productive derivation is a generic means for satisfying specific naming needs that arise in concrete contexts and situations. The semantics of productively formed regular derivatives as context-free pairings of form and meaning (ie taken out of …


Semi-Supervised Multilingual Alignment with Lexical Memory for Massively Parallel Text Mining

W Zhang, P Tang, C Lin, S Naagar, Z Ye, J Liu - ICASSP 2025-2025 IEEE …, 2025

… preprocessing data obtained from CommonCrawl and compiling query seeds for the target languages. The CommonCrawl corpus covers over … First, we collect approximately 261.5 billion pages from the CommonCrawl corpus, specifically …


Sensitivity to Emotional Exploitation in Reasoning Models: Stereotypical Analysis

OM Çeldİr, G Dalkiliç - 2025 7th International Congress on Human-Computer …, 2025

In this study, a survey was conducted on synthetic participants with different stereotypes based on three different moral dilemma scenarios using GPT-4o vs. o1-mini. The tests conducted show that the reasoning model tends to give utilitarian answers …


Sentiment Analysis of Sinhala News Comments Using Transformers

I Bandaranayake, H Usoof - Proceedings of the First Workshop on Natural …, 2025

… XLM-R is a multilingual model pre-trained on filtered Common Crawl data containing more than 100 languages, including Sinhala. This … by language classification and filtering of the Common Crawl corpus using the Ungoliant …


Sentiment Analysis of Social Network Contents using Machine Learning Algorithms: A Review

M Omar, A Salah, M Mahdy - International Journal of Computers and Informatics …, 2025

The exponential growth of social media spaces has resulted in a previously unimaginable amount of user-generated content, which can be used to identify public opinion, sentiment, and trends. Sentiment analysis is an area of the natural …


Sentiment classification for telugu using transformed based approaches on a multi-domain dataset

K Chattu, KAN Reddy, SB Veesam, PS Chirumamilla… - Scientific Reports, 2025

… The sentence encoder is a cross-lingual model that has been trained on 2.5 terabytes of data from Common Crawl documents, covering 100 different languages. The primary enhancement of XLM-Roberta, in comparison to its initial iteration, is a …


Sentiment Classification in Code-Mixed Indo-Aryan Languages: A Transformer-Based Survey

S Roy, JR Saini - Intelligent System and Data Analysis

… It has been trained on large (2.5 TB) Common Crawl Data [30]. It has performed well for all multiple cross-lingual benchmarks. This model consists of 12 … Dirt cheap web-scale parallel text from the common crawl. In: Proceedings of the 51st …


SentimentFormer: A Transformer-Based Multi-Modal Fusion Framework for Enhanced Sentiment Analysis of Memes in Under-Resourced Bangla Language

FTJ Faria, LH Baniata, MH Baniata, MA Khair, AIB Ata… - 2025

… Unlike mBERT, it focuses exclusively on MLM during pre-training, using massive multilingual corpora such as CommonCrawl to predict masked words. This focused training enhances its cross-lingual generalization and makes it particularly adept at …


SENTRA: Selected-Next-Token Transformer for LLM Text Detection

M Plyler, Y Zhang, A Tuzhilin, S Khalifah, S Tian - arXiv preprint arXiv:2509.12385, 2025

… We pre-trained our model on a relatively small sample of Common Crawl data. The volume of data and the amount of compute used for pre… When deploying AI detection models in the wild, we found it useful to tune the threshold to a desired …


Seq vs Seq: An Open Suite of Paired Encoders and Decoders

O Weller, K Ricci, M Marone, A Chaffin, D Lawrie… - arXiv preprint arXiv …, 2025

… we drop the noisiest sections (older Dolma common crawl, CC News, general StackExchange) and include filtered DCLM, math, and StackExchange. We then train for 250B tokens and use an inverse square root learning rate schedule from the …


Sexism Identification in Social Networks using LLMs

This paper describes our participation in the EXIST 2025 shared task on sexism detection in social media. We developed a variety of systems for both Task 1.1 (binary classification of sexism) and Task 1.2 (fine-grained categorization), combining …


SHARP: Synthesizing High-quality Aligned Reasoning Problems for Large Reasoning Models Reinforcement Learning

XJ Wu, Z Zhang, ZJ Wen, Z Zhang, W Ren, L Shi… - arXiv preprint arXiv …, 2025

… On the other hand, we recall seed documents based on high-quality STEM textbooks, academic papers, Common Crawl, etc., and extract topics through the latest reasoning models such as Deepseek R1 and Qwen3 to obtain better topic …


SHARPMARK: Revisiting and Enhancing Set-of-Mark Prompting for Multimodal Web Navigation

… HTML-T5 leverages a vast amount of raw 462 HTML data from the CommonCrawl dataset with 463 approximately 3.41 million examples. This exten464 sive training corpus helps HTML-T5 achieve its 465 performance, but it requires significant pre-training …


Sherkala-Chat: Building a State-of-the-Art LLM for Kazakh in a Moderately Resourced Setting

F Koto, R Joshi, N Mukhituly, Y Wang, Z Xie, R Pal… - Second Conference on Language …

… 2020) and CommonCrawl, two extensive and diverse datasets widely used in large-scale language model training. The Pile consists of high-quality curated sources, including academic papers, books, Wikipedia, and web content, ensuring a …


ShizhenGPT: Towards Multimodal LLMs for Traditional Chinese Medicine

J Chen, Z Cai, Z Liu, Y Yang, R Wang, Q Xiao, X Feng… - arXiv preprint arXiv …, 2025

… term TCM lexicon to extract TCM documents from Common Crawl (2017-2023) and WeChat public articles, totaling 96.4GB. A two-step filtering process, … For online data, we constructed a 30K-term TCM lexicon to identify high-density TCM …


Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentation

Y Xing, X Luo, J Xie, L Gao, H Shen, J Song - arXiv preprint arXiv:2508.06426, 2025

Generalist robot policies trained on large-scale datasets such as Open X-Embodiment (OXE) demonstrate strong performance across a wide range of tasks. However, they often struggle to generalize beyond the distribution of their training data. In this paper …


Siamese Hybrid Network Approach for Sentence Similarity

DAA Deepal, A Bandara, PRS De Silva - Vidyodaya Journal of Science, 2024

This paper presents a novel Siamese Hybrid Network approach, namely Siamese Bidirectional Long Short Memory with Convolutional Neural Network (SiBiLConv), for evaluating the similarity in natural language. The model integrates a Siamese …


SIGIR 2025--LiveRAG Challenge Report

D Carmel, S Filice, G Horowitz, Y Maarek, O Somekh… - arXiv preprint arXiv …, 2025

… The Fineweb dataset [11] consists of cleaned and de-duplicated Web content from CommonCrawl6. While Fineweb is relatively cleaner than other web-scale datasets, it still contains some toxic or offensive material and non-English pages …


Sign Operator for Coping with Heavy-Tailed Noise in Non-Convex Optimization: High Probability Bounds Under (L0, L1)-Smoothness

N Kornilov, P Zmushko, M Yandex, A Semenov…

In recent years, non-convex optimization problems are more often described by generalized (L0, L1)-smoothness assumption rather than standard one. Meanwhile, severely corrupted data used in these problems has increased the demand for …


Sign Operator for Coping with Heavy-Tailed Noise: High Probability Convergence Bounds with Extensions to Distributed Optimization and Comparison Oracle

N Kornilov, P Zmushko, A Semenov, A Gasnikov… - arXiv preprint arXiv …, 2025

The growing popularity of AI optimization problems involving severely corrupted data has increased the demand for methods capable of handling heavy-tailed noise, ie, noise with bounded $\kappa$-th moment, $\kappa \in (1,2]$. For the widely used …


Sign Spotting Disambiguation using Large Language Models

JH Low, OM Sincan, R Bowden - arXiv preprint arXiv:2507.03703, 2025

Sign spotting, the task of identifying and localizing individual signs within continuous sign language video, plays a pivotal role in scaling dataset annotations and addressing the severe data scarcity issue in sign language translation. While …


Sign-SGD is the Golden Gate between Multi-Node to Single-Node Learning: Significant Boost via Parameter-Free Optimization

D Medyakov, S Stanko, G Molodtsov, P Zmushko… - arXiv preprint arXiv …, 2025

… 2020] — a cleaned and filtered version of Common Crawl data specifically curated for language model pre-training. See the detailed description of the experimental setup in Appendix A.2. We compare the following methods: Sign-SGD …


Simple Morphology, Complex Models: A Benchmark Study and Error Analysis of POS Tagging for Martinican Creole

L Mompelat - Proceedings of the 2025 CLASP Conference on …, 2025

Part-of-speech (POS) tagging is a foundational task in NLP pipelines, but its development for Creole languages remains limited due to sparse annotated data and structural divergence from high-resource languages. This paper presents the …


Simplification of German Narrative Documents with Longformer mBART

Transformer-models have become the most prominent method for solving a multitude of natural language processing (NLP) tasks since their introduction in 2017. Natural Language Generation (NLG) is one of these problems. In this thesis we …


SimRE: A Requirements Similarity Tool for Software Product Lines

MI Limaylla-Lunarejo, N Condori-Fernandez… - 2025

A Software Product Line (SPL) is a paradigm that effectively describes families of products based on reuse. Requirements engineering in this domain is a complex task, especially when new products are introduced. In this context, identifying …


Simulating Society: Leveraging Large Language Models as Citizen Agents to Study Urban Behavior

JMN García, LAA Pastor, AM Carrero - 2025

Large Language Models (LLMs) have revolutionized natural language generation, enabling machines to produce coherent, contextual, and expressive text. These advances open up new possibilities in domains such as agent-based modeling …


Simulating Subjects: The Promise and Peril of Artificial Intelligence Stand-Ins for Social Agents and Interactions

AC Kozlowski, J Evans - Sociological Methods & Research, 2025

Large language models (LLMs), through their exposure to massive collections of online text, learn to reproduce the perspectives and linguistic styles of diverse social and cultural groups. This capability suggests a powerful social scientific application—the …


SinLlama-A Large Language Model for Sinhala

HWK Aravinda, R Sirajudeen, S Karunathilake… - arXiv preprint arXiv …, 2025

Low-resource languages such as Sinhala are often overlooked by open-source Large Language Models (LLMs). In this research, we extend an existing multilingual LLM (Llama-3-8B) to better serve Sinhala. We enhance the LLM tokenizer with …


skLEP: A Slovak General Language Understanding Benchmark

M Šuppa, A Ridzik, D Hládek, T Javůrek, V Ondrejová… - arXiv preprint arXiv …, 2025

In this work, we introduce skLEP, the first comprehensive benchmark specifically designed for evaluating Slovak natural language understanding (NLU) models. We have compiled skLEP to encompass nine diverse tasks that span token-level …


Skrr: Skip and Re-use Text Encoder Layers for Memory Efficient Text-to-Image Generation

H Seo, W Jeong, J Seo, SY Chun - arXiv preprint arXiv:2502.08690, 2025

Large-scale text encoders in text-to-image (T2I) diffusion models have demonstrated exceptional performance in generating high-quality images from textual prompts. Unlike denoising modules that rely on multiple iterative steps, text encoders require …


SLA: Beyond Sparsity in Diffusion Transformers via Fine-Tunable Sparse-Linear Attention

J Zhang, H Wang, K Jiang, S Yang, K Zheng, H Xi… - arXiv preprint arXiv …, 2025

In Diffusion Transformer (DiT) models, particularly for video generation, attention latency is a major bottleneck due to the long sequence length and the quadratic complexity. We find that attention weights can be separated into two parts: a small …


SlimPack: Fine-Grained Asymmetric Packing for Balanced and Efficient Variable-Length LLM Training

Y Liu, G Wu, S Zhang, W Zhang, Q Zhu, Z Li, C Wang - arXiv preprint arXiv …, 2025

… We extended our analysis to Common Crawl and Wikipedia datasets, with results detailed in Figure 15 and Figure 16. Both results reaffirm the trends observed on the GitHub dataset. Across these diverse datasets, slice-level packing consistently …


Small Language Models (SLMs) Can Still Pack a Punch: A survey

S Subramanian, V Elango, M Gungor - arXiv preprint arXiv:2501.05465, 2025

… Pile [37], 825 GiB english corpus dataset created using common crawl technique from sources like PubMed Central, ArXiv, GitHub, the FreeLaw Project, Stack Exchange, the US Patent etc is used to train SLMs like Cerebras-GPT [27] family of …


Small-to-Large Generalization: Data Influences Models Consistently Across Scale

Choice of training data distribution greatly affects model behavior. Yet, in large-scale settings, precisely characterizing *how* changes in training data influence predictions is often difficult due to model training costs. Current practice is to instead …


SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment

Y Song, Z Xue, D Wei, F Chen, J Gao, J Liu, H Liang… - arXiv preprint arXiv …, 2025

While frontier large language models (LLMs) continue to push capability boundaries, their deployment remains confined to GPU-powered cloud infrastructure. We challenge this paradigm with SmallThinker, a family of LLMs natively designed - not …


SmartGuard: Support Vector Machine (SVM)-Powered Defense Mechanism for Phishing Prevention

YA Rambharat, DR Shelke, KA Khaleel, AA Arunkumar… - … International Conference on …, 2024

SmartGuard serves as a strong defense system against data breaches and financial risks from hackers by utilizing a range of machine learning algorithms, including Support Vector Machine (SVM). The proposed system involves a website that …


SMOL: Professionally translated parallel data for 115 under-represented languages

I Caswell, E Nielsen, J Luo, C Cherry, G Kovacs… - arXiv preprint arXiv …, 2025

… Researcher in the Loop (RITL) Despite its success in the ablation, Greedy Token Set Cover had several problems when we scaled it to select from among all the English sentences of CommonCrawl. Firstly, it is maximized by honeypots, or …


SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion

A Nassar, A Marafioti, M Omenetti, M Lysak… - arXiv preprint arXiv …, 2025

We introduce SmolDocling, an ultra-compact vision-language model targeting end-to-end document conversion. Our model comprehensively processes entire pages by generating DocTags, a new universal markup format that captures all page elements …


SmolLab_SEU at CheckThat! 2025: how well do multilingual transformers transfer across news domains for cross-lingual subjectivity detection

MA Rahman, MA Amin, MS Dewan, MJ Hasan… - Faggioli et al, 2025

Automated detection of subjectivity in news articles is an important problem for fighting against fake news and promoting journalistic accountability, but this is a challenging task in various linguistic settings. This paper shows our method on Task …


SmolLM2: When Smol Goes Big--Data-Centric Training of a Small Language Model

LB Allal, A Lozhkov, E Bakouch, GM Blázquez… - arXiv preprint arXiv …, 2025

… We began by extracting text from Common Crawl WARC files using Resiliparse, focusing on all 5.8B unique URLs from the FineWeb dataset (a subset of Common Crawl’s 75B unique URLs). We then employed the FineWeb-Edu filtering approach …


SmolLM2: When Smol Goes Big—Data-Centric Training of a Fully Open Small Language Model

A Lozhkov, E Bakouch, GM Blazquez, G Penedo… - Second Conference on Language …

… Common Crawl WARC files using Resiliparse, focusing on all 5.8B unique URLs from the FineWeb dataset (a subset of Common Crawl’s … From the Common Crawl index, we retrieved a total of 7.7B URLs belonging to this list of domains: 5.7B …


SNAP: A Benchmark for Testing the Effects of Capture Conditions on Fundamental Vision Tasks

I Kotseruba, JK Tsotsos - arXiv preprint arXiv:2505.15628, 2025

… Overall, metadata in the datasets is distributed highly unevenly; datasets originating from the Common Crawl generally contain much fewer images with metadata (1-6%) than datasets from curated resources, such as Flickr and Wikipedia (30-60%). …


Society and Bias: Uncovering Automated Prejudices in Sociotechnical Natural Language Processing Systems

As artificial intelligence expands into diverse sectors like finance and healthcare, AI systems increasingly shape our social interactions. However, these systems often perpetuate human-like biases, particularly in natural language processing (NLP) …


SoftMatcha: A Soft and Fast Pattern Matcher for Billion-Scale Corpus Searches

H Deguchi, G Kamoda, Y Matsushita, C Taguchi… - arXiv preprint arXiv …, 2025

Researchers and practitioners in natural language processing and computational linguistics frequently observe and analyze the real language usage in large-scale corpora. For that purpose, they often employ off-the-shelf pattern-matching tools …


SoK: Advances and Open Problems in Web Tracking

Y Vekaria, Y Beugin, S Munir, G Acar, N Bielova… - arXiv preprint arXiv …, 2025

Web tracking is a pervasive and opaque practice that enables personalized advertising, retargeting, and conversion tracking. Over time, it has evolved into a sophisticated and invasive ecosystem, employing increasingly complex techniques …


SoK: Data Minimization in Machine Learning

R Staab, N Jovanović, K Mai, P Ganesh, M Vechev… - arXiv preprint arXiv …, 2025

Data minimization (DM) describes the principle of collecting only the data strictly necessary for a given task. It is a foundational principle across major data protection regulations like GDPR and CPRA. Violations of this principle have substantial real-world …


Speculating LLMs' Chinese Training Data Pollution from Their Tokens

Q Zhang, D Wang, H Qian, L Yan, T Zhang, K Xu, Q Li… - arXiv preprint arXiv …, 2025

… corpus by mixing the related webpages from CommonCrawl8 of 200 normal Chinese tokens … the polluted webpages containing “波*野 结衣” within CommonCrawl and compute its pres- … related to “波*野结衣” from CommonCrawl …


SSMT-PANBERT: A single-stage multitask model for phenotype extraction and assertion negation detection in unstructured clinical text

NE Zekaoui, M Rhanoui, S Yousfi, M Mikram - Computers in Biology and Medicine, 2025

Automatic phenotype extraction and assertion negation detection from large-scale accessible Electronic Health Records (EHRs), including discharge summaries and radiology reports, is a crucial task for various healthcare applications, such as …


State-of-the-Art Natural Language Processing for Aviation: A Review

… Corpus like Common Crawl contains petabytes of web data collected from crawling two fifty billion web pages on which LLMs like Generative Pre-trained Transformer-3 (GPT-3), Large Language Model MetaAI (LLaMa), OpenLLaMa, and …


State–Fourier Diffusion Language Model (SFDLM): A Scalable, Novel Iterative Approach to Language Modeling

… WikiText-103 contains longer articles and a broader vocabulary, while C4 (a large Common Crawl–derived dataset) stresses the model’s capacity to handle real-world text complexity. Across all datasets, we used a Byte-Pair Encoding (BPE) vocabulary …


Statistically Optimized SGNS Model: Enhancing Word Vector Representation with Global Semantic Weight

Addressing the limitations of the Skip-gram with Negative Sampling (SGNS) model related to negative sampling, subsampling, and its fixed context window mechanism, this paper first presents an in-depth statistical analysis of the optimal solution for …


stEELlm: An LLM for Generating Semantic Annotations of Tabular Data

M Cremaschi, F D'Adda, A Maurino - ACM Transactions on Intelligent Systems and …, 2025

The capabilities of LLMs represent a pivotal step in transforming how we manage and interact with information and data. We witness an increasingly pervasive use of such models in various computational tasks. In some preliminary works, attempts to …


Stochastic Resonance Pathways for Latent Knowledge Reassembly in Large Language Models

D Crutchfield, S Wetherell, R Marchbanks, C Woodley…

… Model Selection The base LLM used in this study was OpenLLaMA-13B, a publicly released autoregressive transformer trained on a mixture of Common Crawl, Wikipedia, and curated open-access datasets. The model architecture followed the …


Stochastic Topological Memory Embedding in Large Language Models: An Empirical Analysis Using Open-Source Neural Architectures

T Connor, Z Molyneux, E Watson, A Scolto, J Wilson

Stochastic approaches to memory have long held promise for improving information retention in high-capacity sequence models, yet integration with topological constructs has rarely been explored in practice. Introducing stochastic topological …


Stories that (are) Move (d by) Markets: A Causal Exploration of Market Shocks and Semantic Shifts across Different Partisan Groups

F Drinkall, S Zohren, M McMahon, JB Pierrehumbert - arXiv preprint arXiv …, 2025

Macroeconomic fluctuations and the narratives that shape them form a mutually reinforcing cycle: public discourse can spur behavioural changes leading to economic shifts, which then result in changes in the stories that propagate. We show …


StoryGem: Voronoi treemap Approach for Semantics-Preserving Text Visualization

N Oda, Y Onoue - arXiv preprint arXiv:2506.18793, 2025

… We use this pretrained model because it has word vectors for 157 languages, learned from CommonCrawl and Wikipedia using FastText, making it scalable for many languages. When extracting word vectors, we remove words that are not …


Strategies for Utilizing Generative AI in Educational Environments

… That data was taken from databases like Common Crawl, internetbased book corpora, Wikipedia, and an internal corpus of data scraped specifically by OpenAI for its quality. This data is what powers the capabilities of generative AI, but the creators …


Strategies for Utilizing Generative

WP Jones, SB Logan - Institutes of Higher Education (IHE) and Workforce …, 2025

… That data was taken from databases like Common Crawl, internet-based book corpora, Wikipedia, and an internal corpus of data scraped specifically by OpenAI for its quality. This data is what powers the capabilities of generative AI, but the creators …


Structural Latency Perturbation in Large Language Models Through Recursive State Induction

M Mangrum, J Pemberton, B Wetherby, P Montague - arXiv preprint arXiv:2502.00758, 2025

Computational efficiency has remained a critical consideration in scaling high-capacity language models, with inference latency and resource consumption presenting significant constraints on real-time applications. The study has introduced a …


Structure and Destructure: Dual Forces in the Making of Knowledge Engines

The making of knowledge engines in natural language processing has been shaped by two seemingly distinct paradigms: one grounded in structure, the other driven by massively available unstructured data. The structured paradigm leverages …


StudyTypeTeller—Large language models to automatically classify research study types for systematic reviews

SE Doneva, S de Viragh, H Hubarava… - Research Synthesis …, 2025

screening, a labor-intensive aspect of systematic review, is increasingly challenging due to the rising volume of scientific publications. Recent advances suggest that generative large language models like generative pre-trained transformer (GPT) …


Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMs

Z Chen, S Wang, T Xiao, Y Wang, S Chen, X Cai, J He… - arXiv preprint arXiv …, 2025

… Figure 6: The relationship between number of samples and cluster ID in Common Crawl dataset, with cluster IDs sorted in descending order by the number of samples. Points are sampled for illustration. In the figure, "Raw data" refers to the original …


Suggested keywords

A major task for the 2024-2029 European Commission will be to reconcile and simplify the European Union’s range of data-market laws into a more coherent framework. At the heart of the approach should be the non-rival nature of data …


SUICIDAL POST DETECTION ON REDDIT USING DEEP LEARNING TECHNIQUES

Suicide is a leading cause of death worldwide, particularly among the young generation. The increase in suicidal posts on social media platforms such as Reddit has presented both challenges and opportunities for mental health intervention. Our …


SUMO: Subspace-Aware Moment-Orthogonalization for Accelerating Memory-Efficient LLM Training

Y Refael, G Smorodinsky, T Tirer, O Lindenbaum - arXiv preprint arXiv:2505.24749, 2025

… For this evaluation, we trained large LLaMA-based models on the C4 dataset, a curated and extensive version of the Common Crawl web corpus [46]. This dataset is widely used for pre-training language models and developing word representations …


SUN'IY INTELLEKT VA KOMPYUTER GRAFIKASI AVTOMATIK TASVIR GENERATSIYASI

XS Sharifxon o'g'li - Журнал научных исследований и их решений, 2025

… AI tomonidan ishlab chiqarilgan kontentning sifati va haqiqiyligi bilan bog‘liq xavotirlar paydo bo‘ldi, tadqiqotlar shuni ko‘rsatdiki, Common Crawldan 6 milliarddan ortiq jumlalar namunasidagi jumlalarning 57% dan ortig‘i mashina …


Supernova: Achieving More with Less in Transformer Architectures

AV Tanase, E Pelican - arXiv preprint arXiv:2507.15773, 2025

We present Supernova, a 650M-parameter decoder-only transformer that demonstrates how careful architectural design and tokenization innovation can achieve the performance of larger models while maintaining computational efficiency …


SupraTok: Cross-Boundary Tokenization for Enhanced Language Model Performance

AV Tănase, E Pelican - arXiv preprint arXiv:2508.11857, 2025

Tokenization remains a fundamental yet underexplored bottleneck in natural language processing, with strategies largely static despite remarkable progress in model architectures. We present SupraTok, a novel tokenization architecture that …


Survey of Filtered Approximate Nearest Neighbor Search over the Vector-Scalar Hybrid Data

Y Lin, K Zhang, Z He, Y Jing, XS Wang - arXiv preprint arXiv:2505.06501, 2025

… Each vector pair includes a 512-dimensional image embedding and a corresponding 512-dimensional text embedding, both generated from the Common Crawl corpus using the same CLIP model [81]. The scalar part includes the image …


SURVEY ON ENHANCING DIALOGUE AGENT ALIGNMENT THROUGH MINILLM WITH TARGETED HUMAN ASSESSMENTS.

SB MAHAJAN, CD VAIDYA, BL NARWARE… - i-Manager's Journal on …, 2025

This paper presents the development of a compact and effective language model inspired by the LLaMA architecture. The model's design is based on the fundamental principles of LLaMA, which influenced the architectural decisions and …


SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling

KC Puvvada, F Ladhak, SA Serrano, CP Hsieh… - arXiv preprint arXiv …, 2025

We present a decoder-only Transformer architecture that robustly generalizes to sequence lengths substantially longer than those seen during training. Our model, SWAN-GPT, interleaves layers without positional encodings (NoPE) and sliding-window …


Synchronic and Diachronic Predictors of Socialness Ratings of Words

V Bochkarev, A Shevlyakova, A Achkeev - Journal of Language and Education, 2024

… trained on the CommonCrawl corpus that … the CommonCrawl corpus, therefore, the predictors employing these vectors show lower accuracy. It should be noted that a slightly lower result was obtained using the Glove-840B pre-trained vectors …


Syntactic Choice Is Shaped by Fine-Grained, Item-Specific Knowledge

E Goodwin, B Levin, E Morgan - Proceedings of the Annual Meeting of the Cognitive …, 2025

There is a longstanding debate over how much idiosyncratic, item-specific knowledge is contained in our mental grammars, in addition to productive knowledge of item-general rules and constraints. A key source of evidence is …


Synthesize-on-Graph: Knowledgeable Synthetic Data Generation for Continue Pre-training of Large Language Models

X Jiang, S Ma, C Xu, C Yang, L Zhang, J Guo - arXiv preprint arXiv:2505.00979, 2025

Large Language Models (LLMs) have achieved remarkable success but remain data-inefficient, especially when learning from small, specialized corpora with limited and proprietary data. Existing synthetic data generation methods for continue pre-training …


Synthetic bootstrapped pretraining

Z Yang, A Zhang, H Liu, T Hashimoto, E Candès… - arXiv preprint arXiv …, 2025

We introduce Synthetic Bootstrapped Pretraining (SBP), a language model (LM) pretraining procedure that first learns a model of relations between documents from the pretraining dataset and then leverages it to synthesize a vast new corpus for joint …


Synthetic Browsing Histories for 50 Countries Worldwide: Datasets for Research, Development, and Education

D Komosny, SU Rehman, MS Ayub - Scientific Data, 2025

… We are using state-of-the-art tools such as Common Crawl to ensure the dataset includes real web data while safeguarding privacy. … The website endpages are compiled from Common Crawl 7 . Common Crawl provides extensive datasets of …


Synthetic CVs To Build and Test Fairness-Aware Hiring Tools

J Saldivar, A Gatzioura, C Castillo - arXiv preprint arXiv:2508.21179, 2025

… They further extended this dataset [43] by including short biographies to the CVs using the Common Crawl Bios dataset [11] which contains online biographies related to 28 different occupations. The Common Crawl Bios dataset [11] was …


Synthetic Data Enhances Mathematical Reasoning of Language Models Based on Artificial Intelligence

Z Han, W Jiang - Information Technology and Control, 2025

Current large language models (LLMs) training involves extensive training data and computing resources to handle multiple natural language processing (NLP) tasks. This paper endeavors to assist individuals to compose feasible mathematical …


Synthetic Dataset Generation for Customer Inquiry Classification

This thesis focuses on intent classification for customer inquiries, which will help improve the chatbot application by enhancing its ability to understand and respond accurately to user requests. It covers the entire data science pipeline, from data …


Synthetic Document Question Answering in Hungarian

J Li, Z Csaki, N Hiremath, E Guha, F Hong, E Ma… - arXiv preprint arXiv …, 2025

… HuDocVQAmanual is a small manually curated dataset based on Hungarian documents from Common Crawl [8], while HuDocVQA is a … We also present HuCCPDF, a dataset of 117k pages from Hungarian Common Crawl PDFs along …


Synthetic Social Engineering Scenario Generation using LLMs for Awareness-Based Attack Resilience

J Webb, F Abri, S Akther - IEEE Access, 2025

Social engineering is found in a strong majority of cyberattacks today, as it is a powerful manipulation tactic that does not require the technical skills of hacking. Calculated social engineers utilize simple communication to deceive and exploit …


Systematic Technical Survey on LLMOps: Lifecycle, Tools, Challenges, and Emerging Practices

The emergence of Large Language Models (LLMs) has transformed artificial intelligence applications across industries, yet their operational management presents challenges that exceed traditional Machine Learning Operations (MLOps) …


T\'yr-the-Pruner: Unlocking Accurate 50% Structural Pruning for LLMs via Global Sparsity Distribution Optimization

G Li, Y Xu, Z Li, J Liu, X Yin, D Li, E Barsoum - arXiv preprint arXiv:2503.09657, 2025

Structural pruning enhances hardware-agnostic inference efficiency for large language models (LLMs) but often struggles to maintain performance. Local pruning performs efficient layer-by-layer compression but ignores global topology. Global …


TABLET: A Large-Scale Dataset for Robust Visual Table Understanding

I Alonso, I Miranda, E Agirre, M Lapata - arXiv preprint arXiv:2509.21205, 2025

While table understanding increasingly relies on pixel-only settings where tables are processed as visual representations, current benchmarks predominantly use synthetic renderings that lack the complexity and visual diversity of real-world tables …


Tabular Deep Learning: A Survey from Small Neural Networks to Large Language Models

Tabular data are ubiquitous in several real-world domains, including finance, healthcare, cybersecurity, and ecommerce. In spite of the dominance of deep learning for homogeneous data (such as computer vision and natural language …


TAH-QUANT: Effective Activation Quantization in Pipeline Parallelism over Slow Network

G He, Y Cao, Y He, T Bai, K Yuan, B Yuan - arXiv preprint arXiv:2506.01352, 2025

… 3B on (v) C4 common crawl corpus for 6,000 iterations. These setups cover both general and specialized tasks, as well as supervised … 3B on the C4 common crawl corpus for 6000 iterations, with a batch size of 131072 tokens. The learning rate is …


Taming LLMs by Scaling Learning Rates with Gradient Grouping

S Li, J Tian, Z Wang, X Jin, Z Liu, W Zhang, D Xu - arXiv preprint arXiv:2506.01049, 2025

… The C4 dataset, a meticulously cleaned and processed version of Common Crawl’s web corpus, serves as a benchmark for pre-training language models and learning word representations. To closely replicate real-world pre-training conditions, we …


Taxi1500: A Dataset for Multilingual Text Classification in 1500 Languages

C Ma, A Imani, H Ye, R Pei, E Asgari, H Schütze - … of the 2025 Conference of the …, 2025

… Compared with data of other benchmarks that normally use Common Crawl or Wikipedia, the domain of the Bible is too specific to extract categories merely according to common sense. Instead, theological knowledge may assist in category …


taz2024full: Analysing German Newspapers for Gender Bias and Discrimination across Decades

S Urchs, V Thurner, M Aßenmacher, C Heumann… - arXiv preprint arXiv …, 2025

Open-access corpora are essential for advancing natural language processing (NLP) and computational social science (CSS). However, large-scale resources for German remain limited, restricting research on linguistic trends and societal issues …


TDM and AI Training in the European Union–From 'LAION'to Possible Ways Ahead?

M Leistner, L Antoine - GRUR International, 2025

… To create this data set, LAION, a nonprofit organisation, used an already existing data set originally provided by ‘Common Crawl’. LAION downloaded all the images referred to in the URL collection, used a software tool to verify the existing image …


Teachers First

CM Moran, DG Pyles - Reimagining Literacy in the Age of AI: Theory and …, 2025

… Trained on the Common Crawl dataset of billions of publicly available web pages, ChatGPT analyzes language and looks for common patterns to answer queries. Controversy has dogged ChatGPT since its debut with some publications calling it a “global …


Team OpenWebSearch at LongEval: Using Historical Data for Scientific Search

D Alexander, M Fröbe, G Hendriksen, M Hagen… - 2025

We describe the submissions of the OpenWebSearch team for the CLEF 2025 LongEval Sci-Retrieval track. Our approaches aim to explore how historical data from the past can be re-used to build effective rankings. The Sci-Retrieval track uses …


Technical and ethical debt in the AI fair use crisis

G Toscano, E Petrov, L Li, VS Bahl - 2025

The widespread use of copyrighted data to train AI systems without permission triggered a prolonged “AI fair use crisis.” This paper explores the legal, policy, technical, and ethical dimensions of this issue through the lens of technical and …


Technical Challenges of Rightsholders' Opt-out From Gen AI Training after Robert Kneschke v. LAION

S Havlikova - JIPITEC–Journal of Intellectual Property, Information …, 2025

… Common Crawl Foundation proclaims to comply with Robots.txt and no follow policies of the scraped websites (for these purposes the Common Crawl … ), at the same time Common Crawl’s publicly available Terms of use explicitly limit Common …


Technical, legal, and ethical challenges of generative artificial intelligence: an analysis of the governance of training data and copyrights

M Pasetti, JW Santos, NK Corrêa, N de Oliveira… - Discover Artificial …, 2025

This article examines the legal, technical, and ethical challenges of generative AI, focusing on the governance of training data and copyright compliance. It addresses the growing tension between AI development and the rights of content creators …


Technological Determination of AI-Relevant Press and Copyright Law and Generative Content's Relevance for EU Competition Law-The referral in Case C-250/25 …

… It is therefore not clear whether they also cover other players in the AI value chain – such as providers responsible for web scraping and crawling (eg Common Crawl), providers of training tools and datasets (eg LAION) and – perhaps most importantly …


TEDI: Trustworthy and Ethical Dataset Indicators to Analyze and Compare Dataset Documentation

W Hutiri, M Cimpoi, M Scheuerman, V Matthews… - arXiv preprint arXiv …, 2025

Dataset transparency is a key enabler of responsible AI, but insights into multimodal dataset attributes that impact trustworthy and ethical aspects of AI applications remain scarce and are difficult to compare across datasets. To address this …


Tehran's US Options

In our cover article this month, Matthew Levitt examines potential retaliation by Iran against the US homeland following its 12-day war with Israel and US airstrikes against three of its nuclear facilities.“Iran may seek to carry out reprisal attacks in the …


Temporally Extending Existing Web Archive Collections for Longitudinal Analysis

L Frew, ML Nelson, MC Weigle - arXiv preprint arXiv:2505.24091, 2025

… Existing data sets of webpage crawls commonly used for academic purposes, such as ClueWeb [28] and Common Crawl,2 aim to collect snapshots of a large amount of unique URLs. Other large crawls focused on specific domains include the …


TepiSense: A Social Computing based Real-Time Epidemic Surveillance System using Artificial Intelligence.

Artificial Intelligence (AI) technologies have enabled researchers to develop tools to monitor real-world events and user behavior using social media platforms. Twitter is particularly useful for gathering invaluable information related to diseases and …


Term-Driven Classification of Low-Resource Mathematical Documents in Uzbek Language

N Boltayev, S Urazmetova, S Yakubov, X Shonazarov… - 2025 IEEE 26th …, 2025

This article describes an algorithm for classifying mathematical documents in the Uzbek language that relies on the use of terms with different meanings. For each section of mathematics (discrete mathematics, probability theory, mathematical …


Territorial Control of Data and Compute in Generative AI: A New Paradigm of Competitive Advantage

* Ancien membre du Collège de l’Autorité de la concurrence française au titre de personnalité qualifiée pour les professions réglementées. CNRS–GREDEG–Université Côte d’Azur. CIRANO† HEC Montréal. CIRANO/OBVIA, CEIMIA/GPAI-OECD …


TeSent: A Benchmark Dataset for Fairness-aware Explainable Sentiment Classification in Telugu

VR Kumar, S Manna, N Sett, CVH Harshitha… - arXiv preprint arXiv …, 2025

In the Indian subcontinent, Telugu, one of India's six classical languages, is the most widely spoken Dravidian Language. Despite its 96 million speaker base worldwide, Telugu remains underrepresented in the global NLP and Machine Learning …


Test-Time Code-Switching for Cross-lingual Aspect Sentiment Triplet Extraction

D Sheng, K Han, H Li, Y Zhang, Y Huang, J Lang… - arXiv preprint arXiv …, 2025

Aspect Sentiment Triplet Extraction (ASTE) is a thriving research area with impressive outcomes being achieved on high-resource languages. However, the application of cross-lingual transfer to the ASTE task has been relatively unexplored …


Test-Time Learning for Large Language Models

J Hu, Z Zhang, G Chen, X Wen, C Shuai, W Luo, B Xiao… - arXiv preprint arXiv …, 2025

While Large Language Models (LLMs) have exhibited remarkable emergent capabilities through extensive pre-training, they still face critical limitations in generalizing to specialized domains and handling diverse linguistic variations …


Testimony by LLMs

Artificial testimony generated by large language models (LLMs) can be a source of knowledge. However, the requirement that artificial testifiers must satisfy for successful knowledge acquisition is different from the requirement that human …


Text sentiment analysis using machine learning and deep learning models

S Suruthi, R Sandhiya, P Usha - Advances in Electrical and Computer Technologies, 2025

… Using GloVe 300-dimensional word vectors, 840 billion characters and 2.2 million tokens from the lexicon were utilized to generate a Common Crawl library. Using an unsupervised learning method, the model depicts words as vectors. The semantic …


Text Summarization and Multilingual Text to Audio Translation using Deep Learning Models

B Soni, SK Bharti, A Choudhury - … Conference on Intelligent Computing and Emerging …, 2024

Different technologies has been combined in this work to read the research paper or review paper. This work introduces a system that can automatically summarize research or review papers into audio files in Hindi and English, making it easier for …


Text Summarization of Indo-Aryan Languages Using Self-attention

S Hadawle, P Kotkar, O Bhatia, S Dongre, AR Singh - … Proceedings of ICEEE 2024, Volume 2

Text summarization plays a crucial role in distilling relevant informa-tion from large textual documents. However, there are negligible language models available for working on texts available in local or indigenous languages. Regional languages …


Text Summarization of Medical Records

… To generalize the RoBERTa model for multilingual data, authors of [21] built a multilingual dataset consisting of CommonCrawl Corpus1 in … It is again trained on a dataset extracted from Common Crawl, consisting of 25 languages. The model …


Text-to-Image Generation Using GANs: A Comprehensive Tutorial

Text-to-image generation has become a central problem in cross-modal generative modeling, aiming to translate natural language descriptions into realistic and semantically consistent images. Recent advances, particularly driven by generative …


Textagon: Boosting Language Models with Theory-guided Parallel Representations

JP Lalor, R Qin, D Dobolyi, A Abbasi - Proceedings of the 63rd Annual Meeting of the …, 2025

Pretrained language models have significantly advanced the state of the art in generating distributed representations of text. However, they do not account for the wide variety of available expert-generated language resources and lexicons that …


TextAtlas5M: A Large-scale Dataset for Dense Text Image Generation

AJ Wang, D Mao, J Zhang, W Han, Z Dong, L Li, Y Lin… - arXiv preprint arXiv …, 2025

… After obtaining images generated by Stable Diffusion and images from CommonCrawl, which contain large fillable text areas (such as billboards, electronic screens, etc.), we use YOLO v11 and RT-DETR_r50vd to identify and label the …


The AI Fetish: When Wooden Brains Begin to Think

A Levant - Caderno Brasileiro de Ensino de Física, 2025

Este artigo examina como a inteligência artificial se tornou fetichizada no discurso contemporâneo, sendo imaginada como uma força autônoma ao invés de trabalho humano coletivo cristalizado. Baseando-se na teoria do fetichismo da mercadoria …


The AI tool that can interpret any spreadsheet instantly

… For example, LLMs such as OpenAI’s GPT-4 are pre-trained on hundreds of billions of documents (if not more), using sources such as Common Crawl (see commoncrawl.org). By contrast, there are very few tabular data sets: Kaggle, one of …


The Arabic AI Fingerprint: Stylometric Analysis and Detection of Large Language Models Text

MS Al-Shaibani, M Ahmed - arXiv preprint arXiv:2505.23276, 2025

… This model was pretrained on 2.5TB of filtered CommonCrawl data spanning 100 languages. We used Huggingface Transformers and PyTorch-Lightning to fine-tune the model with early stopping of 3 consecutive evaluation if not improvement …


The Artificial Intelligence and Machine Learning Blueprint: Foundations, Frameworks, and Real-World Applications

In the current era of data-centric transformation, Artificial Intelligence (AI) and Machine Learning (ML) are influencing organizational strategies and operations. The AI and Machine Learning Blueprint serves as a guide connecting academic …


The Breeze 2 Herd of Models: Traditional Chinese LLMs Based on Llama with Vision-Aware and Function-Calling Capabilities

CJ Hsu, CS Liu, MH Chen, M Chen, PC Hsu, YC Chen… - arXiv preprint arXiv …, 2025

Breeze 2 is a suite of advanced multi-modal language models, available in 3B and 8B parameter configurations, specifically designed to enhance Traditional Chinese language representation. Building upon the Llama 3, Breeze 2 continues pretraining …


The Cake that is Intelligence and Who Gets to Bake it: An AI Analogy and its Implications for Participation

M Mundt, A Ovalle, F Friedrich, P Agrawal, S Paul… - arXiv preprint arXiv …, 2025

In a widely popular analogy by Turing Award Laureate Yann LeCun, machine intelligence has been compared to cake - where unsupervised learning forms the base, supervised learning adds the icing, and reinforcement learning is the cherry …