The Collapse of GPT

… LLMs work by learning the statistical distribution of so-called tokens—words or parts of words—within a language by examining billions of sentences garnered from sources including book databases, Wikipedia, and the Common Crawl dataset, a …


The Common Pile v0. 1: An 8TB Dataset of Public Domain and Openly Licensed Text

N Kandpal, B Lester, C Raffel, S Majstorovic… - arXiv preprint arXiv …, 2025

Large language models (LLMs) are typically trained on enormous quantities of unlicensed text, a practice that has led to scrutiny due to possible intellectual property infringement and ethical concerns. Training LLMs on openly licensed text …


The contribution of LLMs to relation extraction in the economic field

M Ettaleb, M Kamel, N Aussenac-Gilles, V Moriceau - … of the Joint Workshop of the 9th …, 2025

Relation Extraction (RE) is a fundamental task in natural language processing, aimed at deducing semantic relationships between entities in a text. Traditional supervised extraction methods relation extraction methods involve training models …


The Cultural Devaluation of Feminized Work: The Evolution of US Occupational Prestige and Gender Typing in Linguistic Representations, 1900 to 2019

Previous research on occupational devaluation typically evaluates the potential wage declines associated with a significant inflow of women into an occupation; results have been mixed. Few studies, however, examine the cultural mechanism …


The Data Governance Challenges

… Mozilla recently published a study noting the dangers of relying on the Common Crawl for trustworthy AI. Author Stefan Baack noted that the crawl’s mission does not align with the needs of trustworthy AI developers. He also pointed out that because …


The data-quality illusion: Rethinking Classifier-based quality filtering for LLM Pretraining

TN Saada, L Bethune, M Klein, D Grangier, M Cuturi… - arXiv preprint arXiv …, 2025

Large-scale models are pretrained on massive web-crawled datasets containing documents of mixed quality, making data filtering essential. A popular method is Classifier-based Quality Filtering (CQF), which trains a binary classifier to distinguish …


The Disruption of Due Diligence: How Generative AI is Transforming M&A Due Diligence Processes

This thesis explores the emerging role of Generative Artificial Intelligence (GenAI) in transforming due diligence processes within mergers and acquisitions (M&A), with a particular focus on financial and operational due diligence conducted by large …


The Effectiveness of Uncased Tokeniziaion for Clinical Notes

C Paik, K von der Wense - Findings of the Association for Computational …, 2025

The impact of case-sensitive tokenization on clinical notes is not well understood. While clinical notes share similarities with biomedical text in terminology, they often lack the proper casing found in polished publications. Language models, unlike …


The Emergence of Abstract Thought in Large Language Models Beyond Any Language

Y Chen, Y Zhao, Y Zhang, A Zhang, K Kawaguchi… - arXiv preprint arXiv …, 2025

… Common Crawl is a publicly available web crawl spanning petabytes of data. OSCAR further processes this raw data to produce monolingual corpora across a wide range of languages, making it a valuable resource for training large language …


The evolving landscape of web and social media archiving: a comprehensive review of current practices in national libraries and archives: Vlassenroot et al.

E Vlassenroot, Y Tao, P Mechant - International Journal of Digital Humanities, 2025

The dynamic and ephemeral nature of online content underscores the critical role of web and social media archiving in preserving digital cultural heritage and supporting research. This study presents a comprehensive review of the current …


The Feasibility and Comparability of Using Artificial Intelligence for Qualitative Data Analysis in Equity-Focused Research

Y Jiang, L Ko-Wong, I Valdovinos Gutierrez - Educational Researcher, 2025

In this essay, we explored the feasibility of utilizing artificial intelligence (AI) for qualitative data analysis in equity-focused research. Specifically, we compare thematic analyses of interview transcripts conducted by human coders with those …


The Future of HPC and AI

MM Resch, J Gebert, D Hoppe - 2025 International Conference on Intelligent Control …, 2025

In recent years, the world of HPC has seen two challenges. One is the end of Moore's law, the other is the impact of AI. Both will change the world of HPC as we know it. In this paper, we explore the state of HCP and AI. Looking into AI, we …


The Gendered, Epistemic Injustices of Generative AI

I Barry, E Stephenson - Australian Feminist Studies, 2025

The rise of generative artificial intelligence (GenAI) brings optimism for productivity, economic, and social progress, but also raises concerns about algorithmic bias and discrimination. Regulators and theorists face the urgent task of identifying potential …


The general factor of personality (GFP) in natural language: A deep learning approach

D van der Linden, A Cutler, PA Van der Linden… - Journal of Research in …, 2025

Using Large Language Models (LLMs), we tested the presence of a general factor of personality (GFP) in trait words in natural language (eg, thousands of books and posts on internet). We included three set of trait words, extracted from well-known …


The generative revolution: AI foundation models in geospatial health—applications, challenges and future research

B Resch, P Kolokoussis, D Hanny, MA Brovelli… - International Journal of …, 2025

In an era of rapid technological advancements, generative artificial intelligence and foundation models are reshaping industries and offering new advanced solutions in a wide range of scientific areas, particularly in public and environmental health …


The geography of digital and green (twin) firms in Germany

L Kriesch, M Abbasiharofteh, S Losacker - Regional Studies, Regional Science, 2025

… Our analysis is based on a dataset containing textual data from websites of 678,381 firms in Germany, collected from the CommonCrawl web archive in 2023. In a first step, we processed the website texts into 44,221,656 meaningful paragraphs …


The Global Brain Argument: Nodes, Computroniums and the AI Megasystem (Target Paper for Special Issue)

… S draws from a significant amount of internet content (eg, Wikipedia, publically available books such as those at The Gutenberg Project, publically accessible news sites, GitHub, social media, blogs, the public access web scraping service contents …


The Governance of Generative AI: Three Conditions for Research and Policy

F Ferrari - Governing the Digital Society: Platforms, Artificial …, 2025

… GPT-3.5, for example, was trained on 45 terabytes of text data, which adds up to approximately 300 billion words extracted from public sources like Wikipedia, CommonCrawl, and GitHub, but also from undisclosed other sources. Open models …


The GPT revolution: Bridging the gap between artificial and human intelligence

AI Nezer, BM Nema - AIP Conference Proceedings, 2025

… 60% of the weighted pretraining dataset for GPT-3 is derived from a refined version of the Common Crawl dataset, which consists of 410 billion byte-pair-encoded tokens. Additional data is drawn from various other sources. Specifically, 19 billion …


The Human Labour of Data Work: Capturing Cultural Diversity through World Wide Dishes

SM Hall, S Dalal, R Sefala, F Yuehgoh, A Alaagib… - arXiv preprint arXiv …, 2025

We provide a window into the process of constructing a dataset for machine learning (ML) applications by reflecting on the process of building World Wide Dishes (WWD), an image and text dataset consisting of culinary dishes and their associated customs …


The Impact of Varying Knowledge on Question-Answering System

ANT Ha, TN Quoc, TN Van, HP Trung, VT Hoang… - 2024 Asian Conference on …, 2024

Scale up the large language models to store vast amounts of knowledge within their parameters incur higher costs and training times. Thus, in this study, we aim to examine the effects of language models enhancing external knowledge and …


The Influence of Audiovisual Semantics on Attention

The information captured by human senses is highly structured by meaningful relationships to each other and to the scene as a whole. In visual scenes, semantic modulation of attention is well established, but had not yet been studied extensively …


The Key Pillars of Responsible AI

T Lau - Banking on (Artificial) Intelligence: Navigating the …, 2025

While AI, especially generative AI, is often associated with chatbots and digital arts tools, the reality is, AI is much more than that. In this chapter, we will review the key pillars and importance of responsible innovation, including the humans behind the …


The KL3M Data Project: Copyright-Clean Training Resources for Large Language Models

MJ Bommarito II, J Bommarito, DM Katz - arXiv preprint arXiv:2504.07854, 2025

… CommonCOW: Massively huge web corpora from CommonCrawl data and a method to distribute them freely under restrictive EU copyright laws. In Proceedings of the tenth international conference on language resources and evaluation (LREC’16) …


The Landscape of Arabic Large Language Models (ALLMs): A New Era for Arabic Language Technology

S Al-Khalifa, N Durrani, H Al-Khalifa, F Alam - arXiv preprint arXiv:2506.01340, 2025

The emergence of ChatGPT marked a transformative milestone for Artificial Intelligence (AI), showcasing the remarkable potential of Large Language Models (LLMs) to generate human-like text. This wave of innovation has revolutionized how we …


The Landscape of Arabic Large Language Models

S Al-Khalifa, N Durrani, H Al-Khalifa, F Alam - Communications of the ACM

… For pre-training, the datasets include Web content (for example, Common Crawl), Wikipedia, books, news, and code, covering a wide range of disciplines. Every ALLM development initiative curates, filters, and processes these datasets within …


The Law and Ethics of AI Creativity

The rise of generative artificial intelligence (“AI”) systems has triggered a backlash among creatives across the globe. In December 2022, artists initiated the No to AI Art movement on social media, 1 primarily as a response to AI companies exploiting …


The LiveRAG Challenge at SIGIR 2025

D Carmel, S Filice, G Horowitz, Y Maarek, O Somekh… - Proceedings of the 48th …, 2025

… The Fineweb dataset [7] consists of cleaned and de-duplicated Web data from CommonCrawl6. This dataset is relatively clean, compared to other Web datasets, however, it still contains some toxic/offensive data, as well as non-English pages …


The Lucie-7B LLM and the Lucie Training Dataset: Open resources for multilingual language generation

O Gouvert, J Hunter, J Louradour, C Cerisara… - arXiv preprint arXiv …, 2025

… The Common Crawl foundation15 regularly crawls the web to pick up new material, paying attention to respect opt-out choices from url holders. The OSCAR … We note that RedPajama also uses CCNet models to filter CommonCrawl content as …


The MultiGEC-2025 shared task on multilingual grammatical error correction at NLP4CALL

A Masciolini, A Caines, O De Clercq, J Kruijsbergen… - Proceedings of the 14th …, 2025

… Although prompting is only officially supported for a subset of the MultiGEC languages (English, German and Italian), this model has likely been exposed to most if not all of them during training on the continuously updated web-scraped Common Crawl …


The Narcissus Hypothesis: Descending to the Rung of Illusion

R Cadei, C Internò - arXiv preprint arXiv:2509.17999, 2025

Modern foundational models increasingly reflect not just world knowledge, but patterns of human preference embedded in their training data. We hypothesize that recursive alignment-via human feedback and model-generated corpora-induces a …


The persistent shadow of the supermassive black hole of M87-II. Model comparisons and theoretical interpretations

K Akiyama, E Albentosa-Ruíz, A Alberdi, W Alef… - Astronomy & Astrophysics, 2025

The Event Horizon Telescope (EHT) observation of M87 ∗ in 2018 has revealed a ring with a diameter that is consistent with the 2017 observation. The brightest part of the ring is shifted to the southwest from the southeast. In this paper, we provide …


The Pinocchio Effect: Why AI Lies Without Knowing

This essay explores the so-called Pinocchio Effect in generative artificial intelligence: the production of false but plausible content without the intention to deceive. It outlines what “hallucination” means in the context of LLMs, the architectural and …


The politics of artificial intelligence supply chains

J Muldoon, A Valdivia, A Badger - AI & SOCIETY, 2025

The rising demand for generative artificial intelligence (AI) is fueling the growth of extractive supply chains to build and power the infrastructures this technology demands. However, there is ambiguity within the scholarly literature about what …


The PrivaSeer Project: Large-Scale Resources for Analysis of Privacy Policy Text

S Wilson, F Schaub, L Matheson, S Shayesteh, L Xian

Privacy policies provide insight into organizations’ data processing practices, but the wealth of privacy policies available on the web contrasts with the challenges of understanding the state of digital privacy at scale. We report on progress made by …


The promise and perils of smart (city) bots as educational tools

… GPT-3 was trained extensively on several data sets such as Common Crawl’s web archive, WebText2 (a private OpenAI dataset created by crawling links from social news website and forum Reddit that had three upvotes) and Wikipedia. …


The Representational Alignment between Humans and Language Models is implicitly driven by a Concreteness Effect

C Iaia, B Choksi, E Wiebers, G Roig, CJ Fiebach - arXiv preprint arXiv:2505.15682, 2025

The nouns of our language refer to either concrete entities (like a table) or abstract concepts (like justice or love), and cognitive psychology has established that concreteness influences how words are processed. Accordingly, understanding how …


The Rise of AfricaNLP: Contributions, Contributors, and Community Impact (2005-2025)

TD Belay, KY Hussen, SH Imam, I Ameer, IS Ahmad… - arXiv preprint arXiv …, 2025

Natural Language Processing (NLP) is undergoing constant transformation, as Large Language Models (LLMs) are driving daily breakthroughs in research and practice. In this regard, tracking the progress of NLP research and automatically …


The Risks of Generative AI Non-Verifiable Interpretation: Exploring Japanese youkai in English

M Moriguchi, O Kennedy - Lexicography, 2025

Since Open AI's release of ChatGPT 3.5 in November 2022, generative AI has greatly impacted lexicography. During this time, Japanese popular culture including manga and anime has become increasingly popular throughout the world, in which …


The role of compute thresholds for AI governance

M Pistillo, S Van Arsdale, L Heim, C Winter - George Washington Journal of Law & …

… [https://perma.cc/7JK3-JQJ7] (noting that OpenAI filtered the Common Crawl dataset down from 45TB to 570GB, and that the curated dataset was used for 60% of the examples during training). Data can also be filtered in other ways, such as to …


The role of GPT in promoting inclusive higher education for people with various learning disabilities: a review

TR Gadekallu, G Yenduri, R Kaluri, DS Rajput… - PeerJ Computer Science, 2025

The generative pre-trained transformer (GPT) is a notable breakthrough in the field of artificial intelligence, as it empowers machines to effectively comprehend and engage in interactions with humans. The GPT exhibits the capacity to enhance …


The Role of Grammatical Gender and Religiosity in Shaping Implicit Gender Attitudes: An Investigation Into Pashto and Dari Languages

This thesis contributes to the ongoing debate on the relationship between language and cognition by evaluating the impact of language-specific features, such as grammatical gender, on non-linguistic cognitive processes like implicit gender …


The Role of Similar-Sounding Words in Representing Meanings

… of the phonological similarity of associates Study 4 Table 4.1: Tokenizations of “dog” in four languages Table 4.2: Semantic information over the random baseline and value added over length-matched segments Table 4.3: GPT-4 MMLU score as a …


The Second International Workshop on Open Web Search (WOWS)

SM Farzana, M Fröbe, M Granitzer, G Hendriksen…

We organize the second International Workshop on Open Web Search (WOWS) at ECIR 2025 with two calls for contributions: The first call targets scientific contributions on collaborative search engine building, including crawling …


The State of Large Language Models for African Languages: Progress and Challenges

KY Hussen, WT Sewunetie, AA Ayele, SH Imam… - arXiv preprint arXiv …, 2025

Large Language Models (LLMs) are transforming Natural Language Processing (NLP), but their benefits are largely absent for Africa's 2,000 low-resource languages. This paper comparatively analyzes African language coverage across six LLMs, eight …


The Structural Safety Generalization Problem

J Broomfield, T Gibbs, E Kosak-Hine, G Ingebretsen… - arXiv preprint arXiv …, 2025

LLM jailbreaks are a widespread safety challenge. Given this problem has not yet been tractable, we suggest targeting a key failure mechanism: the failure of safety to generalize across semantically equivalent inputs. We further focus the target by …


The structure and correlates of beliefs about mental state intensity dynamics

… This stands in contrast to the language-level embeddings, which were trained on long-form content including Wikipedia and a portion of the Common Crawl. Other confounding factors may account for the variation in the effects both within and …


The WebAI Paradigm of Innovation Research: Extracting Insight From Organizational Web Data Through AI

J Dahlke, S Schmidt, D Lenz, J Kinne, R Dehghan… - 2025

This paper introduces the WebAI paradigm as a promising approach for innovation studies, business analytics, and informed policymaking. By leveraging artificial intelligence to systematically analyze organizational web data, WebAI techniques …


TheBlueScrubs-v1, a comprehensive curated medical dataset derived from the internet

L Felipe, C Garcia, IE Naqa, M Shotande, A Tripathi… - arXiv preprint arXiv …, 2025

… For instance, a study by DeepSeek2 curated and filtered mathematical content within Common Crawl to create state-of-the-art mathematical and coding reasoning models—underscoring the value of large, heterogeneous datasets for specialized …


Thinking Augmented Pre-training

L Wang, N Yang, S Huang, L Dong, F Wei - arXiv preprint arXiv:2509.20186, 2025

This paper introduces a simple and scalable approach to improve the data efficiency of large language model (LLM) training by augmenting existing text data with thinking trajectories. The compute for pre-training LLMs has been growing at an …


Three Dimensions of Speech Coherence in People with Early Psychosis and Their Family Members

D Cokal, A Aloraini, C Palominos, C Demirlek, B Verim…

Alterations in speech coherence have long been noted in schizophrenia spectrum disorders (SSD). Three distinctive dimensions of such alterations are:(i) Use and distribution of noun phrases (NPs, eg a man; that large cat) in discourse;(ii) …


TigerCoder: A Novel Suite of LLMs for Code Generation in Bangla

N Raihan, A Anastasopoulos, M Zampieri - arXiv preprint arXiv:2509.09101, 2025

Despite being the 5th most spoken language, Bangla remains underrepresented in Large Language Models (LLMs), particularly for code generation. This primarily stems from the scarcity of high-quality data to pre-train and/or finetune such models …


TigerLLM--A Family of Bangla Large Language Models

N Raihan, M Zampieri - arXiv preprint arXiv:2503.10995, 2025

The development of Large Language Models (LLMs) remains heavily skewed towards English and a few other high-resource languages. This linguistic disparity is particularly evident for Bangla - the 5th most spoken language. A few initiatives …


Time Series Foundation Models and Their Applications to Scientific Discoveries

The advent of large foundation models like ChatGPT and the concept of artificial general intelligence have shifted the machine learning paradigm from “one model per task” to “one large model, many tasks.” On one hand, the prevalent LLM-based …


Time-IMM: A Dataset and Benchmark for Irregular Multimodal Multivariate Time Series

C Chang, J Hwang, Y Shi, H Wang, WC Peng, TF Chen… - arXiv preprint arXiv …, 2025

… • Textual Data: To provide contextual background for environmental conditions, we collect news articles from Common Crawl spanning January to October 2024. Articles are filtered to include only those that mention both the county name and the …


Tiny Language Models for Automation and Control: Overview, Potential Applications, and Future Research Directions

I Lamaakal, Y Maleh, K El Makkaoui, I Ouahbi… - Sensors, 2025

… RedPajama [93]: A dataset comprising over 100 billion text documents, extracted from 84 CommonCrawl snapshots and processed via the … RoBERTa [95] CCNewsV2: A dataset that includes an updated version of the English text from the …


TISEA: A Scalable Deep Learning Framework for Multi-Faceted Text Analytics with Topic Modeling, Summarization, and Emotion Classification

R Bera, abdul quadir, CJ Joshua, A Shahina - Engineering Research Express, 2025

… The Common Crawl German dataset was used to create synthetic data that was used to build the abstractive summary of the provided text. Using the ROUGE (Recall-Oriented Understudy for Gisting Evaluation) and BLUE scores as a gauge, the model's …


TituLLMs: A Family of Bangla LLMs with Comprehensive Benchmarking

SK Nahin, RN Nandi, S Sarker, QS Muhtaseem… - arXiv preprint arXiv …, 2025

In this paper, we present TituLLMs, the first large pretrained Bangla LLMs, available in 1B and 3B parameter sizes. Due to computational constraints during both training and inference, we focused on smaller models. To train TituLLMs, we collected a …


To a Fandom Far, Far Away: Exploring the Influence of Science Fiction and Fantasy on Modal Judgment, Moral Permissibility, and Creativity

This study explores the impact of genre preference—science fiction and fantasy—on modal judgment, moral permissibility, creativity, and the role of fan identity. As these genres grow in popularity and influence culture, they spark discussions on futuristic …


Token and Span Classification for Entity Recognition in French Historical Encyclopedias

L Moncla, H Zeghidi - arXiv preprint arXiv:2506.02872, 2025

… For our experiments, we used pretrained French Flair embeddings trained on OSCAR and Common Crawl corpora. Overall, Flair achieved competitive performance, especially considering its smaller size and lighter computational …


Token Memory Transformer with Infinite Context

T Sun, K Fujita, K Markov, S Chang - International Conference on Intelligent …, 2025

This study proposes an infinite context Transformer model based on Token Memory, which aims to solve the problem of contextual limitation in long text processing. The core of this model is Token Memory, which stores the context memory for each token …


Token-level Ensembling of Models with Different Vocabularies

R Wicks, K Ravisankar, X Yang, P Koehn, M Post - arXiv preprint arXiv:2502.21265, 2025

Model ensembling is a technique to combine the predicted distributions of two or more models, often leading to improved robustness and performance. For ensembling in text generation, the next token's probability distribution is derived from …


Tokenization is Sensitive to Language Variation

A Wegmann, D Nguyen, D Jurgens - arXiv preprint arXiv:2502.15343, 2025

Variation in language is ubiquitous and often systematically linked to regional, social, and contextual factors. Tokenizers split texts into smaller units and might behave differently for less common linguistic forms. This might affect downstream LLM …


TorchTitan: A PyTorch Native Platform for Training Generative AI Models

T Liu, W Liang - Championing Open-source DEvelopment in ML …

TorchTitan is a PyTorch native open-source platform (GitHub: https://github.com/pytorch/torchtitan) designed for scalable and flexible training of generative AI models. Integrated tightly with PyTorch's distributed stack while offering efficient optimizations and modular …


Toward a Representative DNS Data Corpus: A Longitudinal Comparison of Collection Methods

C Kranig, E Pauley, WS Wung, P Barford, M Crovella… - 2025 9th Network Traffic …, 2025

… c) Common Crawl: Since the collection of each common crawl dataset occurs over the course of a month we consider every domain that … 3) Common Crawl: Common Crawl is the smallest dataset that we considered and focuses on the Web …


Towards a Better Understanding of IoT Domain Names: A Study of IoT Backend

I Ayoub, MS Lenders, B Ampeau, S Balakrichenan… - IEEE Access, 2025

In this paper, we study IoT domain names, the domain names of backend servers on the Internet that are accessed by IoT devices. We investigate how they compare to non-IoT domain names based on their statistical and DNS properties, and the …


Towards an AI narratology: the possibilities of LLM classification for the quantification of abstract narrative concepts in literary studies

C Carroll - The Routledge Handbook of AI and Literature, 2024

While narratology and computational literary criticism share a value for categorisation of formal textual features, the application of computational approaches to narratology has been limited. In this chapter, I argue that the reason …


Towards Best Practices for Open Datasets for LLM Training

S Baack, S Biderman, K Odrozek, A Skowron, A Bdeir… - arXiv preprint arXiv …, 2025

… behind the Common pile sent transcripts of all the videos they used to the YouTube channels and the work they did on identifying creative commons licensed subsets of Common Crawl is being gifted to Common Crawl for them to use to …


Towards Cognitive Cyber-Physical Production Systems in the Age of Generative Artificial Intelligence

I Makovec, E Hozdic - International Conference “New Technologies …, 2025

… The largest repositories include: Common Crawl, Wikipedia and GitHub. The primary corpus used for training LLMs is Common Crawl, which contains over 250 billion web pages (21). Wikipedia serves as a comprehensive encyclopaedia, while …


Towards Efficient and Effective Alignment of Large Language Models

Y Jiang - arXiv preprint arXiv:2506.09329, 2025

Large language models (LLMs) exhibit remarkable capabilities across diverse tasks, yet aligning them efficiently and effectively with human expectations remains a critical challenge. This thesis advances LLM alignment by introducing novel …


Towards Empirically Validated Models of Cogno-Social Belief Dynamics

Our beliefs, shaped by our traits, upbringing, and social environment, and coupled with cognitive and social biases, steer how we process information and interact with the world. Beliefs rarely exist in isolation but interact with other beliefs, evidence …


Towards EnergyGPT: A Large Language Model Specialized for the Energy Sector

A Chebbi, B Kolade - arXiv preprint arXiv:2509.07177, 2025

… For instance, [41] reports that 12% of Common Crawl snapshots are composed of duplicate documents, with a significant portion, up to 40%, being exact duplicates. Training large language models on such highly redundant datasets across multiple …


TOWARDS EQUITABLE

IN AI-ENHANCED - The Routledge Handbook of the Sociopolitical Context …, 2025

… The Common Crawl dataset of automatically collected pages from the Web is an even larger freely available source of training data for LLMs. Its archives go back to 2008: it now totals petabytes of data, and it continues to grow at a rate of billions of …


Towards Equitable, Diverse and Inclusive Language Models in AI-enhanced Education Technology for Language Learners

A Caines, P Buttery, G Seed - The Routledge Handbook of the Sociopolitical Context …, 2025


Towards Experiment Execution in Support of Community Benchmark Workflows for HPC

G von Laszewski, W Brewer, SR Wilkinson, A Shao… - arXiv preprint arXiv …, 2025

A key hurdle is demonstrating compute resource capability with limited benchmarks. We propose workflow templates as a solution, offering adaptable designs for specific scientific applications. Our paper identifies common usage patterns for these …


Towards Improving Visual Chatbot Builders with Intent-based Copilots

E Lacic, P Fribert, A Lukac, I Lovric - 2025

Modern visual-based chatbot building solutions have the aim to simplify the development and deployment of sophisticated and responsive conversational agents. But such an environment sometimes presents a steep learning curve when …


Towards Internet-Scale Training For Agents

B Trabucco, G Sigurdsson, R Piramuthu… - arXiv preprint arXiv …, 2025

… In this experiment, we run the task proposer with the same prompts used to scale annotation the top 1M sites in the CommonCrawl PageRank, and we consider a site to be marked positive for unsafe content if the task proposer generates “N/A” rather …


Towards Open Foundation Language Model and Corpus for Macedonian: A Low-Resource Language

S Krsteski, M Tashkovska, B Sazdov, H Gjoreski… - arXiv preprint arXiv …, 2025

… Sourced from 99 CommonCrawl snapshots that span from 2013 to 2024, the data underwent deduplication and quality filtering. For our … Given the significant overlap between web-based sources (particularly those derived from CommonCrawl) …


Towards Representative Pre-Training Corpora for Arabic Natural Language Processing

… For example, the pre-training dataset for OpenAI’s GPT-3 model was a filtered version of the Common Crawl corpus, developed by using a classifier trained to select documents similar to those in the OpenAI’s GPT-2 pre-training dataset, which …


Towards Safer Pretraining: Analyzing and Filtering Harmful Content in Webscale datasets for Responsible LLMs

S Krishna Mendu, H Yenala, A Gulati, S Kumar… - arXiv e-prints, 2025

SK Mendu, H Yenala, A Gulati, S Kumar, P Agrawal - arXiv preprint arXiv:2505.02009, 2025

… We sampled 1 million random web pages from the Common Crawl dataset to ensure a … Our findings reveal widespread harmful content in webcrawled datasets like Common Crawl… We give analyze and quantify issues in open text datasets like …


Towards Secure and Privacy-Preserving Machine Learning Systems

In recent years, machine learning (ML) has advanced at an unprecedented pace, driving the widespread adoption of increasingly sophisticated models across a broad range of real-world applications, including healthcare, finance, autonomous …


Towards Teams being Led by a Conversational Agent

J Simpson, H Stening, P Nalepka, M Dras, RW Kallen… - Proceedings of the 7th ACM …, 2025

Rapid advances in conversational agents are making it possible for them to lead human teams. However, considerably less is known about how human teams perform under their guidance. To investigate this, we devised a Wizard-of-Oz study …


Towards Text Simplification

J Mandravickaitė, E Rimkienė, DK Kapkan - Data Science in Applications: Towards AI …, 2025

… To train mT5, the authors introduced a multilingual variant of the C4 dataset called mC4, which comprises textual data in 101 languages drawn from the public Common Crawl web scrape. This makes mT5 model particularly suitable for …


Towards the Development of Balanced Synthetic Data for Correcting Grammatical Errors in Arabic: An Approach Based on Error Tagging Model and Synthetic Data …

A Alrehili, A Alhothali - arXiv preprint arXiv:2502.05312, 2025

Synthetic data generation is widely recognized as a way to enhance the quality of neural grammatical error correction (GEC) systems. However, current approaches often lack diversity or are too simplistic to generate the wide range of grammatical …


Towards Trustworthy Explanations in Machine Learning With Applications in Health

As machine learning models become integral to fields like healthcare, finance, autonomous systems, and criminal justice, their complexity has also increased to capture more nuanced data patterns with fewer assumptions. While the increasing …


Toxic Discourse in the Digital Battlefield: Analysing Telegram Channels During the Russia–Ukraine 'Conflict'

A Tretiakov, S D'Antonio‐Maceiras, ÁAS Hernández… - Expert Systems, 2025

Instant messenger Telegram has emerged as a favoured platform for far‐right activism, conspiracy theories, political propaganda, and misinformation, which has its own target audience. This study explores the application of multilingual pre‐trained …


TRACING SENSORY EVENTS: A FRAME-BASED APPROACH TO AUTOMATICALLY CAPTURE AND DIACHRONICALLY ANALYSE OLFACTORY AND …

The study of sensory language has long been influenced by Europe’s visual-centric perspective, which relegated smell and taste to the status of “minor” human senses. Investigating sensory language generally requires identifying the words associated …


Tracking and Identifying International Propaganda and Influence Networks Online

HWA Hanley - Proceedings of the AAAI Conference on Artificial …, 2025

Misinformation and propaganda undermine trust in institutions, spread falsehoods, and sometimes incite violence. However, recent advancements in transformer-based AI models can help combat the proliferation of disinformation globally and in real …


Tracking Civic Space in Developing Countries with a High-Quality Corpus of Domestic Media and Transformer Models

D Moratz, J Springman, E Wibbels, FS Adiguzel… - 2025

… Alternatively, “big data” media repositories like GDELT, Internet Archive, and Common Crawl use automated crawlers to collect news … In Section , we show that the large-scale, indiscriminate crawling by GDELT, Internet Archive, and Common …


Tracking the Takes and Trajectories of English-Language News Narratives across Trustworthy and Worrisome Websites

Understanding how misleading and outright false information enters news ecosystems remains a difficult challenge that requires tracking how narratives spread across thousands of fringe and mainstream news websites. To do this, we …


Training Data Attribution (TDA): Examining Its Adoption & Use Cases

D Cheng, J Bae, J Bullock, D Kristofferson - arXiv preprint arXiv:2501.12642, 2025

This report investigates Training Data Attribution (TDA) and its potential importance to and tractability for reducing extreme risks from AI. First, we discuss the plausibility and amount of effort it would take to bring existing TDA research efforts from their …


Training Matryoshka Mixture-of-Experts for Elastic Inference-Time Expert Utilization

Y Wang, Q Hu, Y Ding, R Wang, Y Gong, J Jiao… - arXiv preprint arXiv …, 2025

Mixture-of-Experts (MoE) has emerged as a promising paradigm for efficiently scaling large language models without a proportional increase in computational cost. However, the standard training strategy of Top-K router prevents MoE models from …


Training Sparse Mixture Of Experts Text Embedding Models

Z Nussbaum, B Duderstadt - arXiv preprint arXiv:2502.07972, 2025

Transformer-based text embedding models have improved their performance on benchmarks like MIRACL and BEIR by increasing their parameter counts. However, this scaling approach introduces significant deployment challenges, including …


Training-Free Layer Selection for Parameter-Efficient Fine-Tuning of Language Models

Adapting large language models (LLMs) to downstream tasks via full fine-tuning is computationally prohibitive. While parameter-efficient fine-tuning (PEFT) methods exist, they often rely on predefined heuristics, incur training overhead for parameter …


Transcript Engineering

M Morzy - Spoken Language Processing: Conversational AI for …, 2025

… In the described scenario, the punctuation used in conversational transcripts noticeably differs from the punctuation utilized in the Common Crawl corpus. The transcripts are prone to ASR errors, where inaccurate transcriptions may contain …


Transductive zero-shot learning via knowledge graph and graph convolutional networks: Q. Li et al.

Q Li, X Sun, J Dong - Scientific Reports, 2025

Zero-shot learning methods are used to recognize objects of unseen categories. By transferring knowledge from the seen classes to describe the unseen classes, deep learning models can recognize unseen categories. However, relying solely on a …


Transfer Learning of Slavic Syllabification for Hyphenation Patterns

The development of quality hyphenation patterns, essential for refined digital typography, has historically been constrained by the scarcity of comprehensive hyphenated word lists. This scarcity particularly affects medium-and low-resource …


TRANSFORMANDO O CENÁRIO JURÍDICO: UMA ESTRUTURA ORIENTADA POR IA PARA PROCESSAMENTO DE TEXTOS JUDICIAIS

L Zanuz, SJ Rigo - … -Revista Científica Multidisciplinar-ISSN 2675-6218, 2025

… Another important resource is the OSCAR corpus, a massive multilingual dataset extracted from the Common Crawl, containing 10.7 billion words. … Judicial texts, which are often underrepresented in general corpora like Wikipedia or Common …


Transformation of ChatGPT into Threat: The Effects of Generative AI on Data Protection and Security

NJ Manjula, K Randhi, SR Bandarapu - American Journal of Computing and …, 2024

Purpose: For 2022, GenAI models were the main digital transformation advancement. Cybersecurity is crucial when GenAI models like ChatGPT and Google Bard get more complex. Cybersecurity incidents have highlighted GenAI's offensive and …


Transformer Architectures for Vocabulary Test Item Difficulty Prediction

… 2020) is pre-trained on text from 100 languages using a large-scale CommonCrawlbased corpus. It employs SentencePiece tokenisation and is trained with a masked language modelling (MLM) objective. XLM-RoBERTa has been …


Transformer Architectures

P Singh, B Raman - Deep Learning Through the Prism of Tensors, 2024

This chapter delves into transformer architectures, which have revolutionized natural language processing (NLP) and beyond. It covers the historical context, self-attention mechanisms, and the encoder–decoder structure of transformers. Popular …


Transformer-Based Re-Ranking Model for Enhancing Contextual and Syntactic Translation in Low-Resource Neural Machine Translation

A Javed, H Zan, O Mamyrbayev, M Abdullah, K Ahmed… - Electronics, 2025

Neural machine translation (NMT) plays a vital role in modern communication by bridging language barriers and enabling effective information exchange across diverse linguistic communities. Due to the limited availability of data in low-resource …


Transformers and Beyond: A Review of Key NLP Developments

… its massive parameter size and training on a broad dataset comprising Common Crawl, WebText, books, and Wikipedia. The training … XLM-R is pretrained on a dataset derived from CommonCrawl, called CC100, which includes cleaned and …


Transformers in Natural Language Processing

P Singh, B Raman - The Geometry of Intelligence: Foundations of …, 2025

Language modeling is one of the foundational tasks in NLP and is crucial for various downstream tasks such as machine translation, summarization, and dialog systems. Transformers, with their self-attention mechanisms and ability to model complex …


Transformers-based feedback analysis of e-commerce: a focused study on quality assessment of agriculture products

W Xu - … Journal of Information and Communication Technology, 2025

Ensuring the quality of agricultural products in e-commerce is a significant challenge due to product variability and the absence of direct inspection before purchase. Customer reviews serve as a critical source of information, offering insights into …


Transforming Translation: The Evolution and Impact of AI on Language Transfer and Communication

DA DePalma, A Lommel - Translation Studies in the Age of Artificial Intelligence

This chapter examines the evolution of translation technologies from ancient writing systems to the current era dominated by artificial intelligence (AI). It highlights how each technological advancement has not only enhanced the efficiency and reach of …


Translate, then Detect: Leveraging Machine Translation for Cross-Lingual Toxicity Classification

SJ Bell, E Sánchez, D Dale, P Stenetorp, M Artetxe… - arXiv preprint arXiv …, 2025

Multilingual toxicity detection remains a significant challenge due to the scarcity of training data and resources for many languages. While prior work has leveraged the translate-test paradigm to support cross-lingual transfer across a range of …


Translation in the Wild

Y Balashov - arXiv preprint arXiv:2505.23548, 2025

Large Language Models (LLMs) excel in translation among other things, demonstrating competitive performance for many language pairs in zeroand few-shot settings. But unlike dedicated neural machine translation models, LLMs are not …


Translation Studies in the Age of Artificial Intelligence

Sun, Liu, Moratto, and the team of contributors provide an in-depth exploration of the implications of artificial intelligence (AI) in the ever-evolving field of translation studies. With key insights to inform future research on this rapidly evolving field in …


Trends and Challenges of Multimodal Solutions for Text and Image Context Extraction

T Kvietkauskas, P Stefanovič - 2025 IEEE Open Conference of Electrical, Electronic …, 2025

The number of published images, texts, and other information rapidly increases in today’s digital space. The ability to simultaneously process textual and visual information helps to interpret content more accurately. It enables the application of …


Trends, Challenges and New Frontiers of Artificial Intelligence

P Pal Chaudhuri, A Dutta, S Pal Choudhury… - New Kind of Machine …, 2025

… Many AI models have been trained on contemporary web texts such as in the CommonCrawl Dataset [115], a free, open repository of web crawl data that can be used by anyone, which likely might have a liberal undertone. This is a departure …


Triple-Entry Accounting and Other Secure Methods to Preserve User Privacy and Mitigate Financial Risks in AI-Empowered Lifelong Education

K Sgantzos, P Tzavaras, M Al Hemairy, ER Porras - Journal of Risk and Financial …, 2025

Within the past five years, and as Artificial Intelligence (AI) increasingly per‑vades the academic and educational landscape, a delicate balance has emerged between leveraging AI’s transformative potential and safeguarding individual privacy, which …


Trust at Your Own Peril: A Mixed Methods Exploration of the Ability of Large Language Models to Generate Expert-Like Systems Engineering Artifacts and a …

TG Topcu, M Husain, M Ofsa, P Wach - arXiv preprint arXiv:2502.09690, 2025

Multi-purpose Large Language Models (LLMs), a subset of generative Artificial Intelligence (AI), have recently made significant progress. While expectations for LLMs to assist systems engineering (SE) tasks are paramount; the interdisciplinary …


Turing Test 2.0: The General Intelligence Threshold

G Mappouras - arXiv preprint arXiv:2505.19550, 2025

… Popular LLMs, like ChatGPT are mainly trained using data that is publicly available on the internet [3] like the massive Common Crawl data set [1]. From that we can infer some information about the training data used to train the LLMs. For …


Two Spelling Normalization Approaches Based on Large Language Models

M Domingo, F Casacuberta - arXiv preprint arXiv:2506.23288, 2025

The absence of standardized spelling conventions and the organic evolution of human language present an inherent linguistic challenge within historical documents, a longstanding concern for scholars in the humanities. Addressing this issue …


U'Dedup: Updatable Block-Level Deduplication Scheme over Similar Data in Fog-Assisted Cloud Storage

M He, X Zhang, L Wang - IEEE Internet of Things Journal, 2025

Fog-assisted cloud storage enables efficient collection and management of Internet of Things data, while large-scale data raise severe requirements for storage space. Deduplication schemes over similar data have been investigated to relieve the …


UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis

X Liu, X Zhang, Z Zhang, Y Lu - arXiv preprint arXiv:2504.11257, 2025

… For Web platform, we use the dumped webpage metadata in Common Crawl (Common Crawl… The Web is our main data source due to the variety of layouts and design styles across websites, as well as the extensive quantity of webpages available in …


UI-Evol: Automatic Knowledge Evolving for Computer Use Agents

Z Zhang, X Liu, X Zhang, J Wang, G Chen, Y Lu - arXiv preprint arXiv:2505.21964, 2025

External knowledge has played a crucial role in the recent development of computer use agents. We identify a critical knowledge-execution gap: retrieved knowledge often fails to translate into effective real-world task execution. Our analysis shows …


UIPro: Unleashing Superior Interaction Capability For GUI Agents

H Li, J Su, J Chen, Z Ju, Y Chen, Q Li, Z Zhang - arXiv preprint arXiv:2509.17328, 2025

… GUI understanding data: Common Crawl: Following [22], we select the pages from the top-200 domains in Common Crawl and design an inhouse web crawler that interacts with elements rendered on the web page and collects interaction …


Ultra-FineWeb: Efficient Data Filtering and Verification for High-Quality LLM Training Data

Y Wang, Z Fu, J Cai, P Tang, H Lyu, Y Fang, Z Zheng… - arXiv preprint arXiv …, 2025

Data quality has become a key factor in enhancing model performance with the rapid development of large language models (LLMs). Model-driven data filtering has increasingly become a primary approach for acquiring high-quality data. However, it …


UmuTeam at CheckThat! 2025: Language-Specific versus Multilingual Models for Fact-Checking

T Bernal-Beltrán, R Pan, JA García-Díaz… - 2025

… For multilingual and zero-shot scenarios, we employed XLM-RoBERTa-Large [30], a Transformerbased multilingual encoder model trained on 2.5 TB of filtered CommonCrawl data covering 100 languages. This model was pre-trained using the …


Uncovering inequalities in new knowledge learning by large language models across different languages

C Wang, H Tang, X Yang, Y Xie, J Suh, S Sitaram… - arXiv preprint arXiv …, 2025

… Following existing research approaches to multilingual natural language processing (NLP) (26,27), we classify languages into different resource levels based on their proportions in the CommonCrawl corpus, which was used to pre-train GPT-3 …


Understand, Solve and Translate: Bridging the Multilingual Mathematical Reasoning Gap

H Ko, G Son, D Choi - arXiv preprint arXiv:2501.02448, 2025

Large language models (LLMs) demonstrate exceptional performance on complex reasoning tasks. However, despite their strong reasoning capabilities in high-resource languages (eg, English and Chinese), a significant performance gap persists in …


Understanding and Mitigating the Bias Inheritance in LLM-based Data Augmentation on Downstream Tasks

M Li, H Chen, Y Wang, T Zhu, W Zhang, K Zhu… - arXiv preprint arXiv …, 2025

… 2019] contains real-world English biographies sourced from Common Crawl for several occupations. To ensure gender balance, we sample 600 examples for each profession, with an equal split between male and female data. An example from …


Understanding Reinforcement Learning for Model Training, and future directions with GRAPE

R Patel - arXiv preprint arXiv:2509.04501, 2025

… The key difference is during pre-training you are using a much larger corpus such as the common crawl whereas for SFT you have a much smaller and higher quality dataset. Let’s actually write down the loss function for the next token prediction …


Understanding the Impact of Domain Term Explanation on Duplicate Bug Report Detection

U Mukherjee, MM Rahman - arXiv preprint arXiv:2503.18832, 2025

… The T5 model is trained on the Colossal Clean Crawled Corpus (C4), a large collection of ≈750GB of English texts sourced from Common Crawl for text generation tasks. Since pre-training is extremely costly and T5 is already trained on …


Understanding the Price Impact of Coordinated Retail Investors: An NLP Study on r/WallStreetBets

This study explores the effectiveness of hybrid models in predicting meme stock prices and returns by incorporating coordinated retail investor signals from social media platforms, specifically r/WallStreetBets. Motivated by extreme market …


Understanding Verbatim Memorization in LLMs Through Circuit Discovery

I Lasy, P Knees, S Woltran - arXiv preprint arXiv:2506.21588, 2025

Underlying mechanisms of memorization in LLMs -- the verbatim reproduction of training data -- remain poorly understood. What exact part of the network decides to retrieve a token that we would consider as start of memorization sequence? How …


UniBERTs: Adversarial Training for Language-Universal Representations

AM Avram, M Lupaşcu, DC Cercel, I Mironică… - arXiv preprint arXiv …, 2025

This paper presents UniBERT, a compact multilingual language model that leverages an innovative training framework integrating three components: masked language modeling, adversarial training, and knowledge distillation. Pre-trained on …


Unified Framework for Efficient Cross-Lingual Transfer Learning Across Low-Resource Languages Using Knowledge-Augmented Multilingual Models

… Optionally, an Unstructured RAG Corpus can be used, where retrievalaugmented generation (RAG) methods find contextually aligned passages from high-resource sources like Wikipedia or CommonCrawl for queries in low-resource languages …


Unified Multimodal Understanding and Generation Models: Advances, Challenges, and Opportunities

X Zhang, J Guo, S Zhao, M Fu, L Duan, GH Wang… - arXiv preprint arXiv …, 2025

… • DataComp [195]: DataComp, contains 1.4 billion samples derived from Common Crawl using carefully designed filtering strategies (CLIP score and Image-based filtering), intended to provide higher quality image-text pairs than raw crawled data. • …


Unifying Retrieval and Generation: A Survey on Retrieval-Augmented Generation in NLP

… These models are typically trained on a massive corpus, such as the Common Crawl or other large datasets, using unsupervised or self-supervised learning techniques. They learn to predict the next word or token in a sequence (in the case …


Universal Dependencies for Sindhi

J Bauer, S Shah, M Shaheer, MAA Talpur, Z Sanjrani… - Eighth Workshop on …, 2025

Sindhi is an Indo-Aryan language spoken primarily in Pakistan and India by about 40 million people. Despite this extensive use, it is a low-resource language for NLP tasks, with few datasets or pretrained embeddings available. In this work, we explore …


Unlocking Scaling Law in Industrial Recommendation Systems with a Three-step Paradigm based Large User Model

B Yan, S Liu, Z Zeng, Z Wang, Y Zhang, Y Yuan, L Liu… - arXiv preprint arXiv …, 2025

Recent advancements in autoregressive Large Language Models (LLMs) have achieved significant milestones, largely attributed to their scalability, often referred to as the "scaling law". Inspired by these achievements, there has been a growing …


Unlocking the Mysteries of OpenAI o1: A Survey of the Reasoning Abilities of Large Language Models

G Wang, S Zhang, T Zhan, Z Shen, J Li, X Hu, X Sun…

… Extracting Mathematical Content: The classifier extracts additional mathematical content from Common Crawl, which is then refined through human annotation. To maintain quality and avoid data contamination, web pages that include questions or …


Unlocking the Potential of Transformers with mT5 and Attention Mechanisms in Multilingual Plagiarism Detection

C Bouaine, F Benabbou, C Zaoui - SN Computer Science, 2025

… Pretrained on a vast Common Crawl-based multilingual dataset spanning 101 languages, mT5 employs transfer learning with exceptional versatility and robustness across various linguistic contexts. One of mT5’s notable features is its …


Unsupervised Machine Translation: How Machines Learn to Understand Across Languages

… The majority of monolingual corpora used in MT papers is derived by automatic filtering of the CommonCrawl corpus. For example, the open source OSCAR2 project compiled a large multilingual corpus by language classification and filtering …


UNSUPERVISED TEXT-TO-SOUND MAPPING VIA EMBEDDING SPACE ALIGNMENT

… For fastText we use the cc.en.300 pretrained model, which is trained on Common Crawl and Wikipedia [30]. Both models embed words in a 300-dimensional space. As previously mentioned, word2vec and fastText are static embedders in which the …


Unsupervised Topic Models are Data Mixers for Pre-training Language Models

J Peng, X Zhuang, Q Jiantao, R Ma, J Yu, T Bai, C He - arXiv preprint arXiv …, 2025

… Additionally, SlimPajama categorizes the data into seven distinct domains: arXiv, Books, C4, CommonCrawl, GitHub, StackExchange, and … In contrast, data derived from CommonCrawl and C4 reveals a high correlation with a majority of topics. This …


Unveiling GPT-4V's hidden challenges behind high accuracy on USMLE questions: Observational Study

Z Yang, Z Yao, M Tasmin, P Vashisht, WS Jang… - Journal of Medical Internet …, 2025

… Since AMBOSS is not publicly available and its licensing terms restrict the automatic website scraping of its proprietary content, they are not in the CommonCrawl data set used to train GPTs [31]. We randomly selected and …


UrBLiMP: A Benchmark for Evaluating the Linguistic Competence of Large Language Models in Urdu

F Adeeba, B Dillon, H Sajjad, R Bhatt - arXiv preprint arXiv:2508.01006, 2025

… This corpus comprises approximately 735 million tokens and includes diverse web-based content from sources such as Common Crawl, Twitter, and news articles. For each paradigm, specific regular expressions were crafted to extract potentially …


Urdu paraphrased text reuse and plagiarism detection using pre-trained large language models and deep hybrid neural networks

H Rizwan Iqbal, M Sharjeel, J Shafi, U Mehmood… - Multimedia Tools and …, 2025

The growing prevalence of text reuse and plagiarism in various fields has led to an urgent need for reliable computational methods for detection. However, current commercial plagiarism detection systems are ineffective in identifying paraphrased …


Urdu Toxic Comment Classification with PURUTT Corpus Development

HH Saeed, T Khalil, F Kamiran - IEEE Access, 2025

… 10) m-T5 A massively multilingual pre-trained text-to-text transformer (m-T5) proposed by [68] is trained on the Common Crawl dataset for 101 … It is a large multilingual language model that was trained on 2.5 terabytes of Common Crawl …


Urdu-Punjabi Code Switched Sentiment Analysis Empowered by a Deep Learning Framework Integrating XLM-R, and GPT

M Hussain, S Ali, H Sattar, A Raza, MH Akbar… - VAWKUM Transactions on …, 2025

… The self-supervised FastText embedding model is trained on large Wikipedia and Common Crawl datasets. An optimized FastText model understands over 150 different dialects and languages, including Urdu. In our proposed models, we …


UrduLLaMA 1.0: Dataset Curation, Preprocessing, and Evaluation in Low-Resource Settings

L Fiaz, MH Tahir, S Shams, S Hussain - arXiv preprint arXiv:2502.16961, 2025

Multilingual Large Language Models (LLMs) often provide suboptimal performance on low-resource languages like Urdu. This paper introduces UrduLLaMA 1.0, a model derived from the open-source Llama-3.1-8B-Instruct architecture and …


Use of Generative Artificial Intelligence in Teaching and Learning: Engineering Instructors' Perspectives

PM Simelane, J Kittur - Computer Applications in Engineering Education, 2025

… The first iteration of this technology, GPT-1, was released in June 2018 with 118 million parameters, trained on datasets like Common Crawl and BookCorpus, aimed at improving downstream task performance such as next-word prediction and …


Using AI Tools to Investigate the Health Effects of Traffic Related PM2. 5 Air Pollution on Chronic Disease Prevalence in the Georgetown Community of Salisbury …

Research has established that PM 2.5 is a major public and environmental health concern and a key driver of cardiovascular disease, diabetes, and hypertension. PM 2.5 contains a variety of pollutants such as carbon dioxide (CO 2), lead (Pb), arsenic …


Using Different Modes of Correction to Improve Fairness

As the prevalence of artificial intelligence continues to grow, we are seeing the increasing impact of algorithmic decision making on people’s daily lives, eg, credit approval, hiring, criminal justice, and student assessment. Given the potential impact …


Using Knowledge Graphs to harvest datasets for efficient CLIP model training

S Ging, S Walter, J Bratulić, J Dienert, H Bast, T Brox - arXiv preprint arXiv …, 2025

… They collected image-text pairs from CommonCrawl and filtered them using Wikipedia and WordNet, then balanced the results. Gadre et al. [10] proposed DataComp, a filtering challenge with a candidate pool of up to 13B image-text pairs …


Using Scaling Laws for Data Source Utility Estimation in Domain-Specific Pre-Training

O Ostapenko, C Guille-Escuret, L Kumar, M Tian… - arXiv preprint arXiv …, 2025

We introduce a framework for optimizing domain-specific dataset construction in foundation model training. Specifically, we seek a cost-efficient way to estimate the quality of data sources (eg synthetically generated or filtered web data, etc.) in order …


Utilizing ChatGPT to integrate world English and diverse knowledge: A transnational perspective in critical artificial intelligence (AI) literacy

A Ghimire - Computers and Composition, 2025

This article proposes the implementation of a transnational post-digital pedagogy and Critical AI literacy incorporating ChatGPT in the classroom. It draws upon Scott Graham's suggestion for a multidimensional recursive writing process, emphasizing …


Utilizing Hybrid CNN-SVM and FastText Word Embedding for Twitter Cyberbullying Classification

A Qoiriah, DO Putri, Y Yamasari, IM Suartana, RE Putra… - 2024 Seventh International …, 2024

People’s new habits in using social media, including spreading comments or hate speech that leads to cyberbullying behavior, is a concern because it has a serious impact on victims. Twitter has 400 million users with the majority being young users …


Utilizing large language models for gastroenterology research: a conceptual framework

P Berry, RR Dhanakshirur, S Khanna - Therapeutic Advances in Gastroenterology, 2025

Large language models (LLMs) transform healthcare by assisting clinicians with decision-making, research, and patient management. In gastroenterology, LLMs have shown potential in clinical decision support, data extraction, and patient …


Utilizing Parameter-Efficient Fine-Tuning Methods to Improve Controllable Text Generation/submitted by Hector Auvinen

Abstract Language models have become a crucial component in the field of natural language processing (NLP). Despite demonstrating remarkable performance in various tasks, the text generated by these models often lacks accurate control. This …


Validating Malicious URLs in Phishing Campaigns Using CNNs

Fraud rates are growing [18], property crime has shifted towards fraud [17], and the world is increasingly online: in 2023, around 89% of households had access to internet [42]. As we continue to move our daily interactions to the internet …


Variational Autoencoding and Segmentation

Mobile Robotics rely on Simultaneous Localization and Mapping (SLAM) to navigate in unknown and dynamic environments. SLAM works by detecting landmarks in the environment and relating the position of the robot to these landmarks. Semantic …


VECTOR REPRESENTATION OF CZECH TEXT

This thesis presents Czechtriever, a retrieval model designed for the Czech language and trained without annotated data. The model is based on the architecture and methodology of Contriever and employs contrastive learning to …


VIBE: Vector Index Benchmark for Embeddings

E Jääsaari, V Hyvönen, M Ceccarello, T Roos… - arXiv preprint arXiv …, 2025

Approximate nearest neighbor (ANN) search is a performance-critical component of many machine learning pipelines. Rigorous benchmarking is essential for evaluating the performance of vector indexes for ANN search. However, the datasets …


ViBidirectionMT-Eval: Machine Translation for Vietnamese-Chinese and Vietnamese-Lao language pair

HV Tran, MQ Nguyen, VV Nguyen - arXiv preprint arXiv:2501.08621, 2025

TH Viet, NM Quy, N Van Vinh - Journal of Computer Science and Cybernetics, 2025

This paper presents an results of the VLSP 2022-2023 Machine Translation Shared Tasks, focusing on Vietnamese-Chinese and Vietnamese-Lao machine translation. The tasks were organized as part of the 9th, 10th annual workshop on Vietnamese …


ViQA-COVID: COVID-19 Machine Reading Comprehension Dataset for Vietnamese

HC Nguyen-Phung, NC Lê, VC Nguyen, HT Nguyen… - arXiv preprint arXiv …, 2025

… • XLM-R [2]: based on RoBERTa [8] - an optimal BERT-based approach, XLM-R was trained on over two terabytes of cleaned CommonCrawl [27] data in 100 languages. XLM-R outperformed mBERT in many crosslingual benchmarks and …


Virtual Doctor: The role, responsibilities, and counterparts of the general medical AI

EL Carbonell, J Huang, Y Shen, J Ke - 2024 IEEE International Conference on …, 2024

… By using unlabeled data from the Common Crawl webarchive, the research team created an enlarged corpus calledCORTEX+pCC, significantly improving the system’s abilityto recognize emotions. Terabot has the potential to bettersupport psychiatric …


Vishing detection over call transcripts using Zero-shot Learning and machine learning

RESEARCH QUESTIONS• How can Zero-Shot Learning be utilized to detect fraud data without relying on task-specific labeled data?• How does ZSL tackle the issue of limited vishing data for training models?• How does ZSL perform compared to …


Vision Language Models in Medicine

BC Kalpelbe, AG Adaambiik, W Peng - arXiv preprint arXiv:2503.01863, 2025

With the advent of Vision-Language Models (VLMs), medical artificial intelligence (AI) has experienced significant technological progress and paradigm shifts. This survey provides an extensive review of recent advancements in Medical Vision-Language …


Vision LLMs: Bridging Language and Visual Understanding-Case study of IndoAI

Vision LLMs are trained on vast datasets containing paired image-text samples, allowing them to perform tasks such as image captioning, visual question answering (VQA) and multimodal reasoning. These Models (Vision LLMs) mark a …


Vision-centric Token Compression in Large Language Model

L Xing, AJ Wang, R Yan, J Tang - arXiv preprint arXiv:2502.00791, 2025

Large Language Models (LLMs) have revolutionized natural language processing, excelling in handling longer sequences. However, the inefficiency and redundancy in processing extended in-context tokens remain a challenge. Many attempts to …


VisR-Bench: An Empirical Study on Visual Retrieval-Augmented Generation for Multilingual Long Document Understanding

J Chen, M Li, J Kil, C Wang, T Yu, R Rossi, T Zhou… - arXiv preprint arXiv …, 2025

… We start from a large-scale, diverse, multilingual corpus of PDF files from all over the Internet using Common Crawl [37]. All documents are extracted using a document parser2. We excluded documents with PDF or quality issues and obtained …


Visual Language Models show widespread visual deficits on neuropsychological tests

G Tangtartharakul, KR Storrs - arXiv preprint arXiv:2504.10786, 2025

Visual Language Models (VLMs) show remarkable performance in visual reasoning tasks, successfully tackling college-level challenges that require high-level understanding of images. However, some recent reports of VLMs struggling to …


Visual Pleasure & Neural Cinema: How can cinematic myths in AI video systems be identified, measured, and navigated?

This research investigates methods to identify, quantify, and creatively navigate the cinematic myths embedded in generative AI video systems. It is premised on the hypothesis that cinematic iconography, mythologies, and visual tropes—particularly …


VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large Vision-Language Models in Fact-Seeking Question Answering

Y Wang, Y Zhao, X Chen, S Guo, L Liu, H Li, Y Xiao… - arXiv preprint arXiv …, 2025

Large vision-language models (LVLMs) have demonstrated remarkable achievements, yet the generation of non-factual responses remains prevalent in fact-seeking question answering (QA). Current multimodal fact-seeking benchmarks primarily …


VisualTrap: A Stealthy Backdoor Attack on GUI Agents via Visual Grounding Manipulation

Z Ye, Y Zhang, W Shi, X You, F Feng, TS Chua - arXiv preprint arXiv:2507.06899, 2025

Graphical User Interface (GUI) agents powered by Large Vision-Language Models (LVLMs) have emerged as a revolutionary approach to automating human-machine interactions, capable of autonomously operating personal devices (eg, mobile …


VisualWebInstruct: Scaling up Multimodal Instruction Data through Web Search

Y Jia, J Li, X Yue, B Li, P Nie, K Zou, W Chen - arXiv preprint arXiv:2503.10582, 2025

Vision-Language Models have made significant progress on many perception-focused tasks, however, their progress on reasoning-focused tasks seem to be limited due to the lack of high-quality and diverse training data. In this work, we aim to address the …


Volume I: Aromatics

Volume I covers aromatic oils, the plants that provide them, and ointments made from them. They include what are probably cardamom, nard, valerian, cassia or senna, cinnamon, balm of Gilead, hops, mastic, turpentine, pine resin, bitumen …


Voting Ensemble Learning for Multilingual Sentiment Analysis: A Comparative Study

A Salah, EA Elhadidi, MM Abdallah, SM Darwish - Journal of Computing and …, 2025

… XLM-R is selected for its robust multilingual performance, having learned from 2.5 TB of Common Crawl data in 100 languages via the masked language modeling task, which allows it to generalize across languages well for sentiment tasks [5]. MPNet is …


Vulnerabilities of Large Language Models

… In particular, we select samples from a subset of Common Crawl3 to feed as context to the model.… 3http://commoncrawl.org/ 4It is possible there is some intersection between these two datasets, effectively allowing this strategy to “cheat” …


WanJuanSiLu: A High-Quality Open-Source Webtext Dataset for Low-Resource Languages

J Yu, F Yuan, R Min, J Yu, P Chu, J Li, W Li, R Zhang… - arXiv preprint arXiv …, 2025

This paper introduces the open-source dataset WanJuanSiLu, designed to provide high-quality training corpora for low-resource languages, thereby advancing the research and development of multilingual models. To achieve this, we have …


WARC Annotation

… • Thisfirstprototypefocusesonorganizingandcompilingsomeof the rapid developments of data pipelines in the NLP/ML space and bringing them to the web archiving community • Our prototype focuses on textual data, as Common Crawl is a text-only archive …


WaterPool: A Language Model Watermark Mitigating Trade-Offs among Imperceptibility, Efficacy and Robustness

B Huang, X Wan - Proceedings of the 2025 Conference of the Nations of …, 2025

Watermarking is a prominent technique to trace the usage of specific large language models (LLMs) by injecting patterns into modelgenerated content. An ideal watermark should be imperceptible, easily detectable, and robust to text alterations …


Weakly Supervised Deep Learning for Arabic Tweet Sentiment Analysis on Education Reforms: Leveraging Pre-trained Models and LLMs with Snorkel

A Alotaibi, F Nadeem, M Hamdy - IEEE Access, 2025

This study introduces a novel approach to sentiment classification of Arabic tweets regarding educational reforms in Saudi Arabia. The complexity of the Arabic language, with its numerous dialects, poses challenges for natural language …


Weakly-Supervised Multimodal Video Pre-Training via Image-Caption Pseudo-Labeling

… In the domain of natural language processing, mining from large web corpora—such as Wikipedia and Common Crawl—has been pivotal to the success of models like BERT and GPT-3 [6,17]. This paradigm extends naturally to multimodal learning …


Web-Scale Deduplication

… The Common Crawl project tracks the majority of the surface web by regularly publishing the results of web crawls [32]. Their individual crawl datasets each contain hundreds of billions of URIs stored at an average length of 72 bytes. This implies a …


Web-scale Retrieval Experimentation with chatnoir-pyterrier

JH Merker, J Bevendorff, M Fröbe, T Hagen, H Scells…

The IR community has always aimed to improve the realism of retrieval experiments by increasing the size of the document collections. As collection sizes grow from megabytes to giga-, tera-, and maybe soon petabytes, IR labs are challenged to …


Web2Wiki: Characterizing Wikipedia Linking Across the Web

V Veselovsky, T Piccardi, A Anderson, R West, A Arora - arXiv preprint arXiv …, 2025

… Using a dataset from Common Crawl, we identify over 90 million Wikipedia links spanning 1.68% of Web domains and examine their distribution, context, and function. Our analysis of English Wikipedia reveals three key findings: (1) Wikipedia …


WebFAQ: A Multilingual Collection of Natural Q&A Datasets for Dense Retrieval

M Dinzinger, L Caspari, KG Dastidar, J Mitrović… - arXiv preprint arXiv …, 2025

… Common Crawl. Our work builds upon the efforts of the Web Data Commons3 (WDC) project, whose focus is the large-scale extraction of structured data from the Common Crawl4 … also utilizes QA pairs extracted from Common Crawl. CCQA …


Webis at TREC 2024: Biomedical Generative Retrieval, Retrieval-Augmented Generation, and Tip-of-the-Tongue Tracks

M Fröbe, L Gienapp, JH Merker, H Scells, EO Schmidt…

In this paper, we describe the Webis Group’s participation in the 2024 edition of TREC. We participated in the Biomedical Generative Retrieval track, the Retrieval-Augmented Generation track, and the Tip-of-the-Tongue track. For the biomedical track, we …


WebMall--A Multi-Shop Benchmark for Evaluating Web Agents

R Peeters, A Steiner, L Schwarz, JY Caspary, C Bizer - arXiv preprint arXiv …, 2025

… To populate the shops with real-world product offers, we used the WDC Extraction of the October 2024 Common Crawl4. The extraction files contain product offers from thousands of real-world e-shops that mark up product offers on their websites using …


WeLT: Weighted Loss Trainer for Biomedical Joint Entity and Relation Extraction

The exponential growth of unstructured textual data has emphasised the need for Information Extraction (IE) to transform raw text into actionable knowledge. IE involves automatically identifying and categorising relevant entities, relationships …


What are Foundation Models Cooking in the Post-Soviet World?

A Lavrouk, T Naous, A Ritter, W Xu - arXiv preprint arXiv:2502.18583, 2025

The culture of the Post-Soviet states is complex, shaped by a turbulent history that continues to influence current events. In this study, we investigate the Post-Soviet cultural food knowledge of foundation models by constructing BORSch, a …


What Are They Filtering Out? A Survey of Filtering Strategies for Harm Reduction in Pretraining Datasets

MA Stranisci, C Hardmeier - arXiv preprint arXiv:2503.05721, 2025

… Sampling from Common Crawl. We gathered 5 samples of documents from the Common Crawl (CC) snapshot released in August 20246. Each sample is composed of 20 Web ARChive (WARC) files that have been processed according to …


What do librarians look like? Stereotyping of a profession by generative Ai

DHR Spennemann, K Oddone - Journal of Librarianship and Information Science, 2025

This study aims to investigate the presence of bias in the visual representation of librarians generated by ChatGPT across three different library settings: school, public, and academic. It focuses on analysing biases related to gender, ethnicity, age, attire …


What Gets Measured Gets Managed: Mitigating Supply Chain Attacks with a Link Integrity Management System

J So, M Ferdman, N Nikiforakis - arXiv preprint arXiv:2509.14583, 2025

… To quantify this discussion, we leverage the data archived by the Common Crawl (CC) project [8] to emulate longitudinal analyses. The CC project periodically archives the web with their crawlers and provides its data for public use in anindex defined by the …


What happens when generative AI models train recursively on each others' generated outputs?

HA Vu, G Reeves, E Wenger - arXiv preprint arXiv:2505.21677, 2025

… Mapping these to realistic scenarios, D∗ could be a public dataset like Common Crawl;Dk could be a private dataset of math problems curated by entity k; and Dt could be an internet scrape from after initial model training. We weight the relative …


What Is The Political Content in LLMs' Pre-and Post-Training Data?

T Ceron, D Nikolaev, D Stammbach, D Nozza - arXiv preprint arXiv:2509.22367, 2025

Large language models (LLMs) are known to generate politically biased text, yet how such biases arise remains unclear. A crucial step toward answering this question is the analysis of training data, whose political content remains largely …


When Bad Data Leads to Good Models

K Li, Y Chen, F Viégas, M Wattenberg - arXiv preprint arXiv:2505.04741, 2025

In large language model (LLM) pretraining, data quality is believed to determine model quality. In this paper, we re-examine the notion of "quality" from the perspective of preand post-training co-design. Specifically, we explore the …


When Dimensionality Hurts: The Role of LLM Embedding Compression for Noisy Regression Tasks

F Drinkall, JB Pierrehumbert, S Zohren - arXiv preprint arXiv:2502.02199, 2025

Large language models (LLMs) have shown remarkable success in language modelling due to scaling laws found in model size and the hidden dimension of the model's text representation. Yet, we demonstrate that compressed representations of …


When Does Language Transfer Help? Sequential Fine-Tuning for Cross-Lingual Euphemism Detection

J Sammartino, L Barak, J Peng, A Feldman - arXiv preprint arXiv:2508.11831, 2025

Euphemisms are culturally variable and often ambiguous, posing challenges for language models, especially in low-resource settings. This paper investigates how cross-lingual transfer via sequential fine-tuning affects euphemism detection across …


When Hallucination Costs Millions: Benchmarking AI Agents in High-Stakes Adversarial Financial Markets

Z Dai, Z Peng, Z Cheng, RY Li - arXiv preprint arXiv:2510.00332, 2025

We present CAIA, a benchmark exposing a critical blind spot in AI evaluation: the inability of state-of-the-art models to operate in adversarial, high-stakes environments where misinformation is weaponized and errors are irreversible. While …


When natural language is not enough: The limits of in-context learning demonstrations in multilingual reasoning

L Ranaldi, B Haddow, A Birch - Findings of the Association for Computational …, 2025

Previous studies have demonstrated the effectiveness of reasoning methods in eliciting multi-step reasoned answers from Large Language Models (LLMs) by leveraging in-context demonstrations. These methods, exemplified by Chain-of-Thought …


WHEN PLATFORMS FALL

A Salter, B Blodgett - The Routledge Companion to Media Fandom, 2025

Our collective understanding of fandom frequently relies upon conversations across platforms: looking back, scholars have continually trusted the accessibility of fandom discourse on platforms ranging from early web spaces such as FanFiction. net …


When Platforms Fall: Scraping and Interpreting Fandom Data in Tumultuous Times

Our collective understanding of fandom frequently relies upon conversations across platforms: looking back, scholars have continually relied upon scraping data and coding themes, content, and trends from FanFiction.net and LiveJournal through to …


Which Chatbot Is the Most Empathic

T Kühbacher, T Schlippe, K Schaaff - Artificial Intelligence in Education Technologies: New …

There is an increasing amount of approaches to use chat-bots to support teaching [1-5]. However, studies show that teachers and learning partners should cover empathic skills to achieve optimal learn-ing progress [6]. Therefore, we investigated the extent …


Which Model Mimics Human Mental Lexicon Better? A Comparative Study of Word Embedding and Generative Models

Word associations are commonly applied in psycholinguistics to investigate the nature and structure of the human mental lexicon, and at the same time an important data source for measuring the alignment of language models with human semantic …


WiFiQnA: a WiFi Dataset for Large Language Models

A Zouhri, L Zitoune, I Lahsen-Cherif - 2025

The telecommunications field is a vast domain that includes several subfields, each offering numerous functionalities to its users. In our case, our research field is Wi-Fi, a technology that has the ability to connect multiple devices. This domain has a lot of …


Will LLMs Scaling Hit the Wall? Breaking Barriers via Distributed Resources on Massive Edge Devices

T Shen, D Zhu, Z Zhao, C Wu, F Wu - arXiv preprint arXiv:2503.08223, 2025

The remarkable success of foundation models has been driven by scaling laws, demonstrating that model performance improves predictably with increased training data and model size. However, this scaling trajectory faces two critical challenges …


WinoWhat: A Parallel Corpus of Paraphrased WinoGrande Sentences with Common Sense Categorization

I Gevers, V De Marez, L De Bruyne, W Daelemans - arXiv preprint arXiv:2503.23779, 2025

In this study, we take a closer look at how Winograd schema challenges can be used to evaluate common sense reasoning in LLMs. Specifically, we evaluate generative models of different sizes on the popular WinoGrande benchmark. We …


Without Burning the Servers Down

… - We use Exponential Backoff and Jitter to develop our own download client for Common Crawl, cc-downloader [8]. - We develop it in the rust programming language [9] and release pre-compiled binaries for Linux (ARM and x86-64), Mac (ARM …


Word Embeddings in NLP

T UÇKAN, E KURT - PIONEER AND INNOVATIVE STUDIES IN COMPUTER …

Word embeddings is a natural language processing (NLP) technology that represents words as multidimensional numerical arrays. This technology mathematically expresses the semantic and syntactic properties of words. For …


X-Transfer Attacks: Towards Super Transferable Adversarial Attacks on CLIP

H Huang, S Erfani, Y Li, X Ma, J Bailey - arXiv preprint arXiv:2505.05528, 2025

As Contrastive Language-Image Pre-training (CLIP) models are increasingly adopted for diverse downstream tasks and integrated into large vision-language models (VLMs), their susceptibility to adversarial perturbations has emerged as a …


xGen-small Technical Report

E Nijkamp, B Pang, E Pakhomov, A Gokul, J Qu… - arXiv preprint arXiv …, 2025

We introduce xGen-small, a family of 4B and 9B Transformer decoder models optimized for long-context applications. Our vertically integrated pipeline unites domain-balanced, frequency-aware data curation; multi-stage pre-training with …


XiHeFusion: Harnessing Large Language Models for Science Communication in Nuclear Fusion

X Wang, Q Yang, F Wang, Q Chen, W Wu, Y Jin… - arXiv preprint arXiv …, 2025

… We have collected multi-source knowledge about nuclear fusion tasks to support the training of this model, including the common crawl, eBooks, arXiv, dissertation, etc. After the model has mastered the knowledge of the nuclear fusion field, we …


xVLM2Vec: Adapting LVLM-based embedding models to multilinguality using Self-Knowledge Distillation

E Musacchio, L Siciliani, P Basile, G Semeraro - arXiv preprint arXiv:2503.09313, 2025

… For example, XLM-ROBERTA was trained on 2.5TB of CommonCrawl data in 100 languages. However, embeddings for the same concept in different languages may not have the same dense representation since there is no constraint during training …


Z-Pruner: Post-Training Pruning of Large Language Models for Efficiency without Retraining

SB Bhuiyan, MSH Adib, MA Bhuiyan, MR Kabir… - arXiv preprint arXiv …, 2025

… • C4: A large-scale, cleaned dataset derived from the Common Crawl web archive, totaling hundreds of gigabytes of English text. It filters out low-quality content, duplicates, and boilerplate, providing a diverse and highvolume source of real-world …