Lost and Found: Computational Quality Assurance of Crowdsourced Knowledge on Morphological Defectivity in Wiktionary

J Sakunkoo, A Sakunkoo - ACL 2025 Student Research Workshop

Morphological defectivity is an intriguing and understudied phenomenon in linguistics. Addressing defectivity, where expected inflectional forms are absent, is essential for improving the accuracy of NLP tools in morphologically rich languages …


Lost, but Preserved–A Web Archiving Perspective on the Ephemeral Web

… It is worth noting that the last three years in Figure 2 seem to be rescued almost completely, but it is due to some data contamination as we have started ingesting CommonCrawl data from the recent years into the Wayback Machine, which …


Low-Resource Neural Machine Translation Using Recurrent Neural Networks and Transfer Learning: A Case Study on English-to-Igbo

OA Ekle, B Das - arXiv preprint arXiv:2504.17252, 2025

In this study, we develop Neural Machine Translation (NMT) and Transformer-based transfer learning models for English-to-Igbo translation - a low-resource African language spoken by over 40 million people across Nigeria and West Africa. Our …


LumiViz: Automating Business Data Visualization with Generative AI

S Górtowski, E Lewańska - International Conference on Business Information …, 2025

The paper presents LumiViz, a system leveraging Generative AI to automate business data visualization processes. The tool processes user queries in natural language, retrieves relevant data, generates visualizations, and provides business-oriented …


Lyric-Based Passwords: Enhancing Security and Recall with AI

J Wise, MT Hoque - Cyber Security and Applications, 2025

In the digital age, text-based passwords remain the cornerstone of user authentication. However, the balance between security and memorability remains a significant challenge. Users often face a dilemma between creating complex …


M+: Extending MemoryLLM with Scalable Long-Term Memory

Y Wang, D Krotov, Y Hu, Y Gao, W Zhou, J McAuley… - arXiv preprint arXiv …, 2025

Equipping large language models (LLMs) with latent-space memory has attracted increasing attention as they can extend the context window of existing language models. However, retaining information from the distant past remains a challenge …


Machine Learners Should Acknowledge the Legal Implications of Large Language Models as Personal Data

H Nolte, M Finck, K Meding - arXiv preprint arXiv:2503.01630, 2025

Does GPT know you? The answer depends on your level of public recognition; however, if your information was available on a website, the answer is probably yes. All Large Language Models (LLMs) memorize training data to some extent. If an …


Machine learning for Early Detection of Phishing URLs in Parked Domains: An Approach applied to a financial institution

JD Duarte, P Chagas, EJ Costa, LP De Melo… - IEEE Access, 2025

Phishing attacks remain a critical threat in the digital era, exploiting social engineering tactics to compromise user trust and sensitive information, often resulting in financial loss and identity theft. These attacks typically exploit multiple …


Machine Learning for Historical Web Application Vulnerability Data

The thesis investigates different machine learning techniques that can be used to extract information from web application vulnerability data. The vulnerability data contains information on vulnerabilities over several years and a tree-like information …


Machine Learning for Malicious URL Classification with Expanded Feature Selection and Natural Language Processing: A Temporal Analysis

This praxis further investigates the research performed by Evan Wehr (2024), who argued that URLs change over time, and that when Machine Learning (ML) is applied to malicious URL classification, performance should decay over time. This …


Machine Learning has the Capability to Monitor the Advancement of Climate Technology Innovation Using Climate-Related Texts

… such as materials from WIKIPEDIA published in 2015, a segment of the CCNEWS dataset discussed in work by Nagel in 2016, the OPEN-WEBTEXT corpus derived from web material linked via Reddit according to Gokaslan and Cohen in 2019, and …


Machine Learning Method Employed for the Objective of Identifying Text on Tweet Dataset

S Pandey - Demystifying Emerging Trends in Machine Learning, 2025

When it comes to training ML systems, internet-based data is invaluable. Despite the difficulty in collecting this information, teams of experts from academic institutions and research labs have created publicly accessible databases. Twitter and other …


Machine Learning-Based Phishing Websites Classification Using Diverse Datasets: An Empirical Analysis

S Haider, B Khan, W Khan, S Ullah, Z Ali - … of Blockchain, Internet of Everything, and …, 2025

Recent technological developments make users vulnerable to several cyber-attacks, where phishing attacks compromise users’ sensitive information. To identify these attacks, there are different social techniques, which bring user awareness. However …


Machine Translation Model Optimization Based on Deep Learning

R Li - 2025 3rd International Conference on Integrated …, 2025

Machine translation (MT) is widely used in cross-language communication as globalization accelerates. However, existing MT models, such as rule-based methods, still have issues, such as inadequate accuracy of translation. To address …


MAGA: MAssive Genre-Audience Reformulation to Pretraining Corpus Expansion

X Hao, K Shen, C Li - arXiv preprint arXiv:2502.04235, 2025

Despite the remarkable capabilities of large language models across various tasks, their continued scaling faces a critical challenge: the scarcity of high-quality pretraining data. While model architectures continue to evolve, the natural language …


MAGNET: Augmenting Generative Decoders with Representation Learning and Infilling Capabilities

S Khosla, K Kafle, S Jenni, H Zhao, J Collomosse, J Shi - arXiv preprint arXiv …, 2025

While originally designed for unidirectional generative modeling, decoder-only large language models (LLMs) are increasingly being adapted for bidirectional modeling. However, unidirectional and bidirectional models are typically trained separately …


MAiDE-up: Multilingual Deception Detection of AI-generated Hotel Reviews

O Ignat, X Xu, R Mihalcea - Findings of the Association for Computational …, 2025

Deceptive reviews are becoming increasingly common, especially given the increase in performance and the prevalence of LLMs. While work to date has addressed the development of models to differentiate between truthful and …


MAIN DIRECTIONS OF COMPUTATIONAL LINGUISTICS

D Alisherova - Journal of Multidisciplinary Sciences and Innovations, 2025

Computational linguistics, an interdisciplinary field at the intersection of linguistics and computer science, focuses on developing algorithms and models to process and understand human language. This article explores the main directions of …


Maintaining Academic Integrity in the Era of

AD Latief, R Fajri - Advanced AI and Prompt Engineering Techniques and …, 2025

This chapter provides comprehensive guidance for academic researchers on effectively integrating Large Language Models (LLMs) in research workflows. Beginning with technical foundations and capabilities, it examines LLMs’ architecture …


Managing Output Risks From Imperfect LLMS

M Sanmugam, J Boldiston - Enhancing Learning Experiences With Digital Tools: AI …, 2025

… URLs crawled are specifically selected by the Common Crawl society, and the Common Crawl data comprises 60% of ChatGPT 3’s entire … the Common Crawl dataset is not easily found or comprehensible. The pages and their content are …


Mangosteen: An Open Thai Corpus for Language Model Pretraining

W Phatthiyaphaibun, C Udomcharoenchaikit… - arXiv preprint arXiv …, 2025

… We describe the setup of the data ablation studies on Common Crawl and FineWeb2 as follows: Common Crawl. We train a GPT-2 model on each of five dataset variations, with each containing 10 billion tokens. Each dataset variation …


Mapping of the Nepali Dependency Treebank to Universal Dependencies

A Das, P Rai, S Chatterji - ACM Transactions on Asian and Low-Resource …, 2025

Universal Dependencies (UD) have garnered notable focus for the systematic assessment of cross-lingual methods in the task of dependency parsing. In this paper, we present our initiative towards the development of a dependency treebank for the …


Markup Language Modeling for Web Document Understanding

S Liu, B Bi, J Bakus, PK Velalam, V Yella, V Hegde - arXiv preprint arXiv:2509.20940, 2025

Web information extraction (WIE) is an important part of many e-commerce systems, supporting tasks like customer analysis and product recommendation. In this work, we look at the problem of building up-to-date product databases by extracting …


Masks Can be Learned as an Alternative to Experts

P Liu, T Wei, B Zhu, WX Zhao, S Yan - Proceedings of the 63rd Annual Meeting of the …, 2025

In this work, we investigate how to sparsify a pre-trained dense large language model into a mixture-of-experts (MoE) architecture for faster inference. Our approach applies mask matrix to the activations for each expert, constrained by L 0 …


MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Models

J Li, L Yu, Q Cui, Z Zhang, J Zhou, Y Ye, C Zhang - arXiv preprint arXiv:2503.14917, 2025

… It has been filtered and extracted from over 200 billion HTML files on Common Crawl, resulting in a refined set of 6.3 million documents containing a total of 14.7 billion tokens. OpenWebMath-pro is an enhanced version of OpenWebMath, further …


Matina: A Large-Scale 73B Token Persian Text Corpus

SB Hosseinbeigi, F Taherinezhad, H Faili, H Baghbani… - arXiv preprint arXiv …, 2025

… 2018) when preprocessing Common Crawl for better data quality. A linear classifier was used to … 2020) was constructed from Common Crawl data to train the T5 model. The langdetect2 tool … by our team and data taken from two public …


Matrix factorization techniques for Large Language Models

In the last years the development of Large Language Models (LLMs) has revolutionized the field of natural language processing (NLP), enabling significant advancements in several contexts, such as text translation, code generation and …


Matryoshka Model Learning for Improved Elastic Student Models

C Verma, AS Timmaraju, C Jui-Hsieh, S Damle, N Bui… - arXiv preprint arXiv …, 2025

Industry-grade ML models are carefully designed to meet rapidly evolving serving constraints, which requires significant resources for model development. In this paper, we propose MatTA, a framework for training multiple accurate Student …


MaXIFE: Multilingual and Cross-lingual Instruction Following Evaluation

Y Liu, Z Ma, X Jiang, J Hu, J Chang, L Li - arXiv preprint arXiv:2506.01776, 2025

With the rapid adoption of large language models (LLMs) in natural language processing, the ability to follow instructions has emerged as a key metric for evaluating their practical utility. However, existing evaluation methods often focus on …


Mdsbots@ nlu of devanagari script languages 2025: Detection of language, hate speech, and targets using murtweet

P Ale, A Thapaliya, S Paudel - Proceedings of the First Workshop on Challenges in …, 2025

In multilingual contexts, an automated system for accurate language identification, followed by hate speech detection and target identification, plays a critical role in processing low-resource hate speech data and mitigating its negative impact. This …


Measuring Controversy in online discussions

… 18], that are trained on Common Crawl and Wikipedia for German, which is highly beneficial for our analysis of posts on derStandard.at. These pretrained models provide a solid foundation for embedding the textual data, ensuring that both …


Measuring Evolution of Cookie Dialogues

This thesis investigates how the cookie dialogues evolved in response to data protection regulations, introducing a scalable methodology that combines web archiving with machine learning to track these changes over time. We explore the …


Measuring memorization in language models via probabilistic extraction

J Hayes, M Swanberg, H Chaudhari, I Yona…

… Since we know that Llama relied on Common Crawl data for training, we use 10,000 examples drawn from Common Crawl. It is, of course, possible that the examples we use were not contained in the OPT or Llama training datasets. We …


Measuring Risks to Users' Health Privacy Posed by Third-Party Web Tracking and Targeted Advertising

E Zeng, X Wu, EN Ertmann, L Huang, DF Johnson… - 2025

Online advertising platforms may be able to infer privacy-sensitive information about people, such as their health conditions. This could lead to harms like exposure to predatory targeted advertising or unwanted disclosure of health conditions to …


Measuring the Prevalence and Variety of Online Age Gates

The legal landscape regarding age-based restrictions (age gates) for online services is rapidly changing. In order to comply with existing and proposed regulations, online services must determine whether users are older or younger than …


Mechanistic Interpretability in the Presence of Architectural Obfuscation

M Florencio, T Barton - arXiv preprint arXiv:2506.18053, 2025

… By contrast, GPT-3 was trained using a mixture of data from Common Crawl, WebText2, books, and Wikipedia [3]. For our case, the custom model was trained on a large corpus of text data called Fineweb-Edu, available on HuggingFace, which …


Medical foundation large language models for comprehensive text analysis and beyond

Q Xie, Q Chen, A Chen, C Peng, Y Hu, F Lin, X Peng… - npj Digital Medicine, 2025

Recent advancements in large language models (LLMs) show significant potential in medical applications but are hindered by limited specialized medical knowledge. We present Me-LLaMA, a family of open-source medical LLMs integrating extensive …


Medical large language models are vulnerable to data-poisoning attacks

DA Alber, Z Yang, A Alyakin, E Yang, S Rai, AA Valliani… - Nature Medicine, 2025

… The lack of oversight leaves vulnerable subsets susceptible to data poisoning; for instance, malicious users can create unverified web pages that end up in the Common Crawl, upload code to GitHub at will, or add comments to Stack Exchange …


MegaMath: Pushing the Limits of Open Math Corpora

F Zhou, Z Wang, N Ranjan, Z Cheng, L Tang, G He… - arXiv preprint arXiv …, 2025

… We present MegaMath, an open dataset curated from diverse, mathfocused sources through following practices: (1) Revisiting web data: We re-extracted mathematical documents from Common Crawl with mathoriented HTML …


Megrez-Omni Technical Report

B Li, Y Li, Z Li, C Liu, W Liu, G Niu, Z Tan, H Xu, Z Yao… - arXiv preprint arXiv …, 2025

In this work, we present the Megrez models, comprising a language model (Megrez-3B-Instruct) and a multimodal model (Megrez-3B-Omni). These models are designed to deliver fast inference, compactness, and robust edge-side intelligence through a software-hardware …


MEL: Legal Spanish Language Model

DB Sánchez, NA García, ÁB Jiménez, MG Nieto… - arXiv preprint arXiv …, 2025

Legal texts, characterized by complex and specialized terminology, present a significant challenge for Language Models. Adding an underrepresented language, such as Spanish, to the mix makes it even more challenging. While pre-trained …


MELLA: Bridging Linguistic Capability and Cultural Groundedness for Low-Resource Language MLLMs

Y Gao, J Fei, N Chen, R Chen, G Yan, Y Lan, B Shi - arXiv preprint arXiv:2508.05502, 2025

Multimodal Large Language Models (MLLMs) have shown remarkable performance in high-resource languages. However, their effectiveness diminishes significantly in the contexts of low-resource languages. Current multilingual enhancement methods …


Membership Inference Attack Against Fine-tuned Language Models

In recent years language models—mathematical models that describe text written by humans—have been rapidly developing and their use is getting more widespread. They evolved from the first statistical language models [1],[2] into the powerful large …


Memorization Inheritance in Sequence-Level Knowledge Distillation for Neural Machine Translation

V Dankers, V Raunak - arXiv preprint arXiv:2502.01491, 2025

In this work, we explore how instance-level memorization in the teacher Neural Machine Translation (NMT) model gets inherited by the student model in sequence-level knowledge distillation (SeqKD). We find that despite not directly seeing the original …


Memory Transmission Based Referring Video Object Segmentation

Z Liu, L Wang, Y Hu, B Yin - Neural Networks, 2025

Referring Video Object Segmentation (RVOS) addresses the task of segmenting target objects described by textual descriptions from videos. In order to ensure the consistency of objects segmented from video frames, inter-frame modeling is …


Mental Multi-class Classification on Social Media: Benchmarking Transformer Architectures against LSTM Models

K Hasan, J Saquer, Y Zhang - arXiv preprint arXiv:2509.16542, 2025

Millions of people openly share mental health struggles on social media, providing rich data for early detection of conditions such as depression, bipolar disorder, etc. However, most prior Natural Language Processing (NLP) research has focused on …


MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs

Y Zhou, G Karamanolakis, V Soto, A Rumshisky… - arXiv preprint arXiv …, 2025

The recent success of specialized Large Language Models (LLMs) in domains such as mathematical reasoning and coding has led to growing interest in methods for merging these expert LLMs into a unified Mixture-of-Experts (MoE) model, with the …


Meta-Learning Transformers to Improve In-Context Generalization

L Braccaioli, A Vettoruzzo, P Singh, J Vanschoren… - arXiv preprint arXiv …, 2025

In-context learning enables transformer models to generalize to new tasks based solely on input prompts, without any need for weight updates. However, existing training paradigms typically rely on large, unstructured datasets that are costly to …


Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models

X Zhuang, J Peng, R Ma, Y Wang, T Bai, X Wei, J Qiu… - arXiv preprint arXiv …, 2025

… To isolate the impact of domain diversity, we constrain data selection and pre-training to a single domain—Common Crawl—while avoiding explicit control over domain sampling ratios. As shown in Figure 3, restricting pre-training to Common Crawl …


MetaCLIP 2: A Worldwide Scaling Recipe

YS Chuang, Y Li, D Wang, CF Yeh, K Lyu… - arXiv preprint arXiv …, 2025

… 2024) introduces a scalable data curation algorithm to meticulously extract a billion-scale English dataset that exhausts long-tailed concepts in Common Crawl The algorithm transforms the distribution of the raw Internet into controllable and …


Metadata Conditioning Accelerates Language Model Pre-training

T Gao, A Wettig, L He, Y Dong, S Malladi, D Chen - arXiv preprint arXiv:2501.01956, 2025

The vast diversity of styles, domains, and quality levels present in language model pre-training corpora is essential in developing general model capabilities, but efficiently learning and deploying the correct behaviors exemplified in each of these …


Metadata-less Dataset Recommendation Leveraging Dataset Embeddings by Pre-trained Tabular Language Models

K Manabe, Y Fujita, M Kuwahara, T Hayashi - 2024 IEEE International Conference on …, 2024

… the ELECTRA objective on approximately 27 million web tables extracted from Common Crawl for five epochs [21]. The pretraining data were … GitTables data were constructed from CSV files collected from GitHub repositories and reported that …


MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generation

H Riaz, S Bhabesh, V Arannil, M Ballesteros… - arXiv preprint arXiv …, 2025

… combinations of mixing Common Crawl texts with synthetic documents and instructions in 1:1 and 1:2 token mixing ratios (refer to appendix B for prompt settings). In Finance, we observe that 25M MetaSynthgenerated tokens—without real …


Methods and Resources in Germanic Variationist Linguistics

J Nerbonne, V Blaschke, H Schütze, B Plank - Oxford Research Encyclopedia of …, 2025

Variationist linguistics, encompassing dialectology and sociolinguistics, studies how linguistic variation is distributed and the dynamics behind the distribution. This article aims to present the most important current resources—methods and data and …


Methods for Analyzing Similarity of Attitudes and Their Application to Science and Technology Across Residents of Countries and Large Language Models

K DokicD, B Radisic, B Pisker - Advances in Intelligent Systems and Digital …, 2025

This paper analyses distance measurement methods in multidimensional vector space to quantify the similarity of responses between LLMs and citizens of individual countries. Four LLMs created in different countries and cul-tural environments (ChatGPT …


Microdosing Psychedelics for Cognitive Enhancement: A Naturalistic Exploration of User Experiences

Psychedelics have recently grown in scientific interest, among this, there have been promising findings regarding its potential to treat a range of health problems (such as depression, anxiety, PTSD). Despite the growing research interest in psychedelics …


Microsoft Corporation: OpenAI and ChatGPT in the Workplace

V Chauhan, J Lomeli, P Hough, JS O'Rourke - 2025

… The datasets used to pre-train GPT-1 were Common Crawl, a dataset of web pages with billions of words, and the BookCorpus, a collection of over 11,000 books covering numerous genres.7 Some of its limitations included producing repetitive text …


MIGRATE: Cross-Lingual Adaptation of Domain-Specific LLMs through Code-Switching and Embedding Transfer

S Hong, S Lee, H Moon, HS Lim - Proceedings of the 31st International Conference …, 2025

… 2023), which contains highquality mathematical texts sourced from Common Crawl. … Pre-trained FastText word vectors for 157 languages are trained on Common Crawl and Wikipedia using CBOW with position-weights. The embeddings …


Mind the Gap: Assessing Wiktionary's Crowd-Sourced Linguistic Knowledge on Morphological Gaps in Two Related Languages

J Sakunkoo, A Sakunkoo - arXiv preprint arXiv:2506.17603, 2025

… Common Crawl (CC-100) (Wenzek et al., 2020): From CC-100, we use an 8.3GB dataset … Common Crawl (Raw Text Corpus) Tokenize with UDPipe Morphologically Tagged Corpus … Tube model is used to annotate text from the …


Mind the Gap: Computational Quality Assurance of Crowd-Sourced Linguistic Knowledge on Latin and Italian Morphological Gaps

J Sakunkoo, A Sakunkoo - Society for Computation in Linguistics, 2025

… This study uses Universal Dependencies (UD), Common Crawl, and Wiktionary in the computational validation of morphological gaps. Universal Dependencies is a collection of multilingual treebanks for syntactic and morphological analysis across …


MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipe

T Yu, Z Wang, C Wang, F Huang, W Ma, Z He, T Cai… - arXiv preprint arXiv …, 2025

Multimodal Large Language Models (MLLMs) are undergoing rapid progress and represent the frontier of AI development. However, their training and inference efficiency have emerged as a core bottleneck in making MLLMs more accessible …


MiniCPM4: Ultra-Efficient LLMs on End Devices

M Team, C Xiao, Y Li, X Han, Y Bai, J Cai, H Chen… - arXiv preprint arXiv …, 2025

This paper introduces MiniCPM4, a highly efficient large language model (LLM) designed explicitly for end-side devices. We achieve this efficiency through systematic innovation in four key dimensions: model architecture, training data …


MiniLingua: Training a Multilingual Small Language Model

This thesis presents the development of a multilingual large language model (LLM) MiniLingua trained on European languages, with a focus on efficiency, linguistic diversity, and open access. The model contains approximately 1 billion, placing it in …


MiniMax-01: Scaling Foundation Models with Lightning Attention

A Li, B Gong, B Yang, B Shan, C Liu, C Zhu, C Zhang… - arXiv preprint arXiv …, 2025

We introduce MiniMax-01 series, including MiniMax-Text-01 and MiniMax-VL-01, which are comparable to top-tier models while offering superior capabilities in processing longer contexts. The core lies in lightning attention and its efficient …


Mining Hidden Thoughts from Texts: Evaluating Continual Pretraining with Synthetic Data for LLM Reasoning

Y Ishibashi, T Yano, M Oyamada - arXiv preprint arXiv:2505.10182, 2025

Large Language Models (LLMs) have demonstrated significant improvements in reasoning capabilities through supervised fine-tuning and reinforcement learning. However, when training reasoning models, these approaches are primarily …


Minnesota Journal of Law, Science & Technolog y

The European Union's Artificial Intelligence Act (AI Act) represents a pioneering attempt to regulate AI technologies. However, this Paper argues that the Act's framework is inadequate for addressing the challenges posed by generative and …


MiST: Understanding the Role of Mid-Stage Scientific Training in Developing Chemical Reasoning Models

AM Bran, T Xie, S Pranesh, J Goumaz, XV Nguyen…

Large Language Models (LLMs) can acquire emergent reasoning via online fine-tuning with simple rule-based rewards when tasks are already latent-solvable by the base model. We study chemical reasoning and identify two pre-requisites for RL-based …


Mitigating Bias LLM-Powered Employee Engagement Models: AI Ethics in Enterprise HR Systems

… Large Language Models in review were developed on the backbone of publicly available datasets, including Common Crawl, Wikipedia, and OpenWebText. Proprietary HR datasets provided by industry collaborators added depth to the …


Mitigating Distribution Bias in Multimodal Datasets via Clustering-Based Curation

M El Aichouni, L Gomez, L Kang - Iberian Conference on Pattern Recognition and …, 2025

… Large-scale multimodal datasets are commonly sourced from web crawls, such as CommonCrawl [15, 16], and are filtered using heuristic rules—eg, constraints on image resolution, limiting captions to English with a predefined vocabulary or clip …


Mixed-Initiative Conversational Intelligence in the Era of Large Pre-Trained Models

With the rise of large pre-trained models, the idea of intelligent conversational agents has quickly gained attention in the public eye. Such conversational agents promise impressive capabilities in a multi-turn interaction setting, whether it be …


Mixtera: A Data Plane for Foundation Model Training

M Böther, X Yao, T Kerimoglu, A Klimovic - arXiv preprint arXiv:2502.19790, 2025

… A mixture describes how the data is mixed based on their characteristics, ie, we might train on 50 % data from Common Crawl and 50 % from movie subtitles. The data can be combined based on multiple characteristics simultaneously. For instance …


Mixture of Hidden-Dimensions: Not All Hidden-States' Dimensions are Needed in Transformer

Y Chen, J Shang, Z Zhang, J Sheng, T Liu, S Wang… - Forty-second International …

Transformer models encounter inefficiency when scaling hidden dimensions due to the uniform expansion of parameters. When delving into the sparsity of hidden dimensions, we observe that only a small subset of dimensions are highly activated …


MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality Instruction and Reasoning Data Built from Permissive-First Text Sources

H Nguyen, V May, H Raj, M Nezhurina, Y Wang, Y Luo… - arXiv preprint arXiv …, 2025

We present MixtureVitae, an open-access pretraining corpus built to minimize legal risk while providing strong model performance. MixtureVitae follows a risk-mitigated sourcing strategy that combines public-domain and permissively licensed text (eg …


Model and qualification

In developing Alexis de Tocqueville's observations, Marx identified civil society as the economic base and political society as the political superstructure. 2 Marx postulated the essentials of the base–superstructure concept in his preface to A …


Model-Agnostic Gender Bias Control for Text-to-Image Generation via Sparse Autoencoder

C Wu, Z Wang, K Xie, NK Devulapally, VS Lokhande… - arXiv preprint arXiv …, 2025

Text-to-image (T2I) diffusion models often exhibit gender bias, particularly by generating stereotypical associations between professions and gendered subjects. This paper presents SAE Debias, a lightweight and model-agnostic framework for …


Modeling False Memories with Conceptual Spaces

The most well researched associative memory illusion (AMI) is the Deese–Roediger–McDermott (DRM) paradigm: a list learning task found to induce false memories though study lists with associated words. Typically, the study lists are composed of the 12—15 …


Modeling Language as Social and Cultural Data

Abstract Language shows up everywhere. It's in the digital content we circulate online, and it's in our conversations with each other. It's also in the training data and generations of language models, which are increasingly integrated into our …


Modeling Multimodal Emotion with Dynamic Interaction-Focused Representation Network

A Brooks, M Rivera, L Hsu, Z Carter - 2025

… Textual Embedding We convert transcriptions into sequences of 300-dimensional vectors using GloVe embeddings pretrained on the 840B Common Crawl corpus. These embeddings provide rich semantic features and maintain high performance …


Modeling Scaled Offensiveness in Greek Texts through Regression with Best–Worst Scaling and Pretrained Language Models

In recent years, offensive language has emerged as a widespread phenomenon across social media platforms, fueled by their increasing use and accessibility. As more users engage in posting harmful or derogatory content, often directed at …


Modelling Intertextuality with N-gram Embeddings

Y Xing - arXiv preprint arXiv:2509.06637, 2025

Intertextuality is a central tenet in literary studies. It refers to the intricate links between literary texts that are created by various types of references. This paper proposes a new quantitative model of intertextuality to enable scalable analysis and …


Modelling Misinformation in Swahili-English Code-switched Texts

… GloVe contains pre-trained embeddings trained on large sets of Common Crawl, Wikipedia and Twitter data. This study used the 100-dimensional GloVe vectors trained on 2 billion tweets. FastText contains word vectors pre-trained on Webcrawl …


ModernGBERT: German-only 1B Encoder Model Trained from Scratch

A Ehrmanntraut, J Wunderle, J Pfister, F Jannidis… - arXiv preprint arXiv …, 2025

Despite the prominence of decoder-only language models, encoders remain crucial for resource-constrained applications. We introduce ModernGBERT (134M, 1B), a fully transparent family of German encoder models trained from scratch …


Mono-InternVL-1.5: Towards Cheaper and Faster Monolithic Multimodal Large Language Models

G Luo, W Dou, W Li, Z Wang, X Yang, C Tian, H Li… - arXiv preprint arXiv …, 2025

This paper focuses on monolithic Multimodal Large Language Models (MLLMs), which integrate visual encoding and language decoding into a single model. Existing structures and pre-training strategies for monolithic MLLMs often suffer from …


MoQE: Improve Quantization Model performance via Mixture of Quantization Experts

J Zhang, Y Zhang, B Zhang, Z Liu, D Cheng - arXiv preprint arXiv:2508.09204, 2025

… C4, a cleaned version of Common Crawl filtered for quality, assesses model robustness on noisy, real-world web text. WikiText-2, composed of structured Wikipedia articles, provides a standard benchmark for evaluating perplexity and …


Moving beyond word error rate to evaluate automatic speech recognition in clinical samples: Lessons from research into schizophrenia-spectrum disorders

SA Just, B Elvevåg, S Pandey, I Nenchev, AL Bröcker… - Psychiatry Research, 2025

Natural language processing applications to mental health research depend on automatic speech recognition (ASR) to study large samples and develop scalable clinical tools. To ensure safe and effective implementation, it is crucial to understand …


mSCAN-A Multilingual Dataset for Compositional Generalization Evaluation

Abstract Language models achieve remarkable results on a variety of tasks, yet still struggle on compositional generalization benchmarks. The majority of these benchmarks evaluate performance in English only, leaving open the question of …


MuBench: Assessment of Multilingual Capabilities of Large Language Models Across 61 Languages

W Han, Y Zhang, Z Chen, B Liu, H Lin, B Zhang… - arXiv preprint arXiv …, 2025

… To estimate the distribution of each language in web-scale data, we also report the number of tokens per language in the Common Crawl corpus. For this, we randomly selected one snapshot from each year between 2022 and 2024 and …


Multi Domain Specific Sentiment Analysis for A Strategic Customer Queries and Feedback, from both Direct and Latent Sentiment by Semantic Associations: Survey.

MK Shruthishree, JV Gorabal - Journal of Computational Analysis & Applications, 2025

… On the other hand, FEEL-IT is built upon UmBERTo2, which is based on the RoBERTa architecture and pre-trained on the Common Crawl Italian dataset. FEEL-IT focuses on classifying four emotional categories (excluding neutral), …


Multi-Agent Multimodal Models for Multicultural Text to Image Generation

P Bhalerao, M Yalamarty, B Trinh, O Ignat - arXiv preprint arXiv:2502.15972, 2025

Large Language Models (LLMs) demonstrate impressive performance across various multimodal tasks. However, their effectiveness in cross-cultural contexts remains limited due to the predominantly Western-centric nature of existing data and …


Multi-Label Classification of Indonesian Voice Phishing Conversations: A Comparative Study of XLM-RoBERTa and ELECTRA

A Hidayat, S Madenda, H Hustinawaty - Journal of Applied Data Sciences, 2025

Mobile phones have become a primary means of communication, yet their advancement has also been exploited by cybercriminals, particularly through voice phishing schemes. Voice phishing is a form of social engineering fraud carried out …


Multi-lingual Functional Evaluation for Large Language Models

V Ojewale, ID Raji, S Venkatasubramanian - arXiv preprint arXiv:2506.20793, 2025

… CommonCrawl (see 3) – these languages were thus selected to cover a spectrum of high-resource and lower-resource contexts. As with prior work (Montariol et al., … Table 3 classifies the languages used in our evaluation based on their relative …


Multi-Modal Framing Analysis of News

A Arora, S Yadav, M Antoniak, S Belongie, I Augenstein - arXiv preprint arXiv …, 2025

… We query the publicly available Common Crawl archives4 for the corresponding publishers and extracted each article’s text, headline, publication date, image_urls and other metadata in JSON format. Post scraping, we filter extremely short and long …


Multi-Modal Twitter Data Analysis for Identifying Offensive Posts Using a Deep Cross Attention based Transformer Framework

J Paul, S Mallick, A Mitra, A Roy, J Sil - ACM Transactions on Knowledge Discovery …, 2025

In today’s society dissemination of information among the individuals occur very rapidly due to the widespread usage of social media platforms like Twitter (now-a-days acclaimed as X). However, information may pose challenges to maintaining a …


Multi-Task Learning approach to identify sentences with impact and affected location in a disaster news report

S Banerjee, S Mukherjee, S Bandyopadhyay - … of the Fourth Workshop on NLP for …, 2025

The first priority of action in the Sendai Framework for Disaster Risk Reduction 2015-2030 advocates the understanding of disaster risk by collecting and processing practical information related to disasters. A smart collection may be the compilation of …


Multi-view multi-label canonical correlation analysis for cross-modal multimedia retrieval

A Rani, R Sanghavi, Y Verma - Multimedia Tools and Applications, 2025

We address the problem of cross-modal retrieval in presence of multi-view and multi-label data. For this, we present Multi-view Multi-label Canonical Correlation Analysis (or MVMLCCA), which is a generalization of Canonical Correlation Analysis (CCA) for …


MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs

J Jumelet, L Weissweiler, A Bisazza - arXiv preprint arXiv:2504.02768, 2025

… Figure 1: Gemma3-27B accuracy per language on MultiBLiMP 1.0, plotted against language frequency in Common Crawl. MultiBLiMP … (2024), which were computed on a 3.9T token split of the Common Crawl corpus. Common Crawl …


MultiCoPIE: A Multilingual Corpus of Potentially Idiomatic Expressions for Cross-lingual PIE Disambiguation

Abstract Language models are able to handle compositionality and, to some extent, noncompositional phenomena such as semantic idiosyncrasy, a feature most prominent in the case of idioms. This work introduces the MultiCoPIE corpus that …


MultiJustice: A Chinese Dataset for Multi-Party, Multi-Charge Legal Prediction

X Wang, J Pei, D Shui, Z Han, X Sun, D Zhu, X Shen - arXiv preprint arXiv …, 2025

Legal judgment prediction offers a compelling method to aid legal practitioners and researchers. However, the research question remains relatively under-explored: Should multiple defendants and charges be treated separately in LJP? To address …


Multilingual and Cross-Linguistic Challenges in NLP

D Jain - Transformative Natural Language Processing, 2025

While NLP advancements have predominantly focused on high-resource languages, linguistic diversity presents significant challenges. This chapter examines issues related to low-resource languages, cross-lingual transfer learning, and strategies for …


Multilingual Attribute Extraction from News Web Pages

P Bedrin, M Varlamov, A Yatskov - arXiv preprint arXiv:2502.02167, 2025

… MarkupLM model was pre-trained on 24M English web pages from CommonCrawl5 … CommonCrawl dataset. DOM-LM pre-training was performed on our multilingual news dataset and on a one-day sample of 37,473 news pages of different …


Multilingual Blending: Large Language Model Safety Alignment Evaluation with Language Mixture

J Song, Y Huang, Z Zhou, L Ma - Findings of the Association for Computational …, 2025

… The CommonCrawl corpus (Crawl… We consider most state-of-the-art LLMs to have trained on these 55 source languages, as these languages are enclosed in the CommonCrawl corpus. All multilingual translations are conducted using Google …


Multilingual capabilities of GPT: A study of structural ambiguity

This study examines the multilingual capabilities of GPT, focusing on its handling of syntactic ambiguity across English, Korean, and Japanese. We investigate whether GPT can capture language-specific attachment preferences or if it relies primarily on …


Multilingual Definition Modeling

E Marrese-Taylor, EK Shimomoto, A Solano, E Reid - arXiv preprint arXiv:2506.01489, 2025

In this paper, we propose the first multilingual study on definition modeling. We use monolingual dictionary data for four new languages (Spanish, French, Portuguese, and German) and perform an in-depth empirical study to test the performance of pre-trained …


Multilingual Entity Linking

Identifying entities and concepts, disambiguating them, and grounding them in encyclopedic resources, is a crucial step toward understanding natural language text. In this monograph, we consider the problem of grounding concepts and entities …


Multilingual Financial Text Summarisation

With the increasing growth in the number of public firms worldwide, the volume of financial disclosures and financial texts in different languages and forms is increasing sharply; therefore, the study of Natural Language Processing (NLP) …


Multilingual Hope Speech Detection: A Comparative Study of Logistic Regression, mBERT, and XLM-RoBERTa with Active Learning

TO Abiola, KD Abiodun, OE Olumide, OO Adebanji… - arXiv preprint arXiv …, 2025

Hope speech language that fosters encouragement and optimism plays a vital role in promoting positive discourse online. However, its detection remains challenging, especially in multilingual and low-resource settings. This paper presents a …


Multilingual language classification model for offensive comments categorisation in social media using HAMMC tree search with enhanced optimisation technique

B Aarthi, BJ Chelliah - International Journal of Computational Science and …, 2025

… An LSTM network and common crawl-trained fast text embeds provide the best performance (0.823 F1) in English, Tamil, French, Hindi, and … If not, pre-trained fast text embeddings trained on common crawl or something similar were advised …


Multilingual Performance Biases of Large Language Models in Education

V Gupta, SP Chowdhury, V Zouhar, D Rooein… - arXiv preprint arXiv …, 2025

Large language models (LLMs) are increasingly being adopted in educational settings. These applications expand beyond English, though current LLMs remain primarily English-centric. In this work, we ascertain if their use in education settings …


Multilingual Retrieval-Augmented Generation for Knowledge-Intensive Task

L Ranaldi, B Haddow, A Birch - arXiv preprint arXiv:2504.03616, 2025

… Common crawl 2021. Web. Accessed: 2023-12-12. … Table 6 reports the language distribution of CommonCrawl, and Table 7 the number of documents in the Wikipedia dump used in our work (§3). … Table 6: Language distribution of …


Multilingual Table-to-Text Generation with Question-Answer Plans

… is based on a single month’s worth of scraped web data in the Common Crawl (CC) dataset, but applies several filtering heuristics to create a … mC4 is based on 71 months of Common Crawl data instead of one, to gather a greater diversity of …


Multimodal approaches to automatic lyric generation

Music plays a fundamental role in human culture, serving as a universal language that transcends barriers and resonates deeply with people’s emotions and experiences. As artificial intelligence continues to advance, there is growing interest …


Multimodal deep learning model for bitcoin price prediction with news and market prices

GV Vardhan, B Subburaj - Neural Computing and Applications, 2025

… In this research, we predict bitcoin prices by leveraging news data extracted from Common Crawl News dataset, in conjunction with bitcoin prices obtained from Coinbase Application Programming Interface. We propose a multimodal deep …


Multimodal Generative AI and The Metamedium Condition

M Suryajaya - Proceeding of Internasional Seminar on Arts, Artificial …, 2024

Since the decline of medium-specific approaches characteristic of early 20th-century modernism, various art theorists have introduced critical frameworks to address the fluidity of contemporary artistic practices. This article explores the implications of …


Multimodal large language models for zero-shot real-world classification tasks: benchmark, taxonomy of prompting methods, and application to human-object …

Multimodal Large Language Models (MLLMs) excel as zero-shot reasoners across diverse domains. However, their application to real-world classification tasks, particularly in direct comparison with specialized models, remains underexplored …


Multimodal Large Language Models: A Survey

L Han, A Mubarak, A Baimagambetov, N Polatidis… - arXiv preprint arXiv …, 2025

… Later LLMs leveraged massive public datasets such as The Common Crawl project1, which rapidly expanded to 460 TiB of uncompressed content as of January 2025. This abundance of data played a crucial role in unlocking the full potential of …


Multimodal prior knowledge determines false memory formation

MA Petilli, FM Rodio, D Gatti, M Marelli, L Rinaldi

Memory formation is a complex phenomenon shaped by various experiential traces, yet their exact contributions remain unclear. This study investigates the generation of false memories leveraging different datadriven computational models to …


Multimodal Prosody Modeling: A Use Case for Multilingual Sentence Mode Prediction

… For word-level embedding representation, we used the XLM-ROBERTA [30] model, pre-trained on 2.5 TB of filtered CommonCrawl data containing 100 languages. Furthermore, the selection of phoneme-level FRS was justified by the …


Multimodal Transformer Training in Personalized Federated Learning

X Cao, G Sun, Z Li, H Yu - Proceedings of the Second International Conference …

… These are generated from Common Crawl, extracting image sources with corresponding alt-text. Caption generation is tested using the MS COCO Caption [31] and Flickr30k [32] datasets, employing the COCO Karpathy split test set and …


Multiple large language models versus clinical guidelines for postmenopausal osteoporosis: a comparative study of ChatGPT-3.5, ChatGPT-4.0, ChatGPT-4o, Google …

CR Lin, YJ Chen, PA Tsai, WY Hsieh, SHL Tsai, TS Fu… - Archives of Osteoporosis, 2025

The study assesses the performance of AI models in evaluating postmenopausal osteoporosis. We found that ChatGPT-4o produced the most appropriate responses, highlighting the potential of AI to enhance clinical decision-making and improve …


MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining

Z Chen, P Guo, W Han, Y Zhang, B Liu, H Lin, F Liu… - arXiv preprint arXiv …, 2025

Data quality is a critical driver of large language model performance, yet existing model-based selection methods focus almost exclusively on English. We introduce MuRating, a scalable framework that transfers high-quality English data-quality …


Native OS-Integrated AI in a European OSS/Open-Core Operating System

This document, produced with the assistance of ChatGPT o3 and GPT-5 Thinking, is released under the Apache 2.0 license. It is a voluntary defensive publication (prior art) and therefore enters the prior art upon release under the applicable patent …


Natural Fingerprints of Large Language Models

T Suzuki, R Ri, S Takase - arXiv preprint arXiv:2504.14871, 2025

Large language models (LLMs) often exhibit biases -- systematic deviations from expected norms -- in their outputs. These range from overt issues, such as unfair responses, to subtler patterns that can reveal which model produced them. We …


Natural Intelligence: the information processing power of life

S Lloyd, M Reilly - arXiv preprint arXiv:2506.16478, 2025

Merely by existing, all physical systems contain information, and physical dynamics transforms and processes that information. This note investigates the information processing power of living systems. Living systems harvest free energy from the sun …


Natural Language Processing and Large Language Models

P Wulff, M Kubsch, C Krist - Applying Machine Learning in Science Education …, 2025

In this chapter we introduce the basics of natural language processing techniques that are important to systematically analyze language data. In particular, we will utilize simple large language models and showcase examples of how to apply them …


NATURAL LANGUAGE PROCESSING IN THE AGE OF ARTIFICIAL INTELLIGENCE: TECHNICAL ADVANCES, OPPORTUNITIES AND CHALLENGES

A ŞEKER - Artificial Intelligence: Foundations, Applications and …, 2025

N atural Language Processing (NLP) is a subfield of artificial intelligence that enables interaction between human languages and computers by aiming to understand and interpret human language. The primary goal of NLP is to transform …


Natural Language Processing Techniques for Information Retrieval Enhancing Search Engines with Semantic Understanding

S Subi, B Shanthini, M SilpaRaj, K Shekar… - ITM Web of Conferences, 2025

This paper investigates new Natural Language Processing (NLP) methods which seek to improve information retrieval systems via semantic knowledge and focuses on enhancing search engines. The proposed ideas focus on reducing the size of the …


NaturalReasoning: Reasoning in the Wild with 2.8 M Challenging Questions

W Yuan, J Yu, S Jiang, K Padthe, Y Li, D Wang… - arXiv preprint arXiv …, 2025

Scaling reasoning capabilities beyond traditional domains such as math and coding is hindered by the lack of diverse and high-quality questions. To overcome this limitation, we introduce a scalable approach for generating diverse and challenging …


Navigating CLIPedia: Architectonic instruments for querying and questing a latent encyclopedia

A Nickl - Frontiers of Architectural Research, 2025

This paper introduces CLIPedia, an entirely local, highly scalable, multimodal search engine that integrates the structured knowledge of Wikipedia into the latent embedding space of OpenCLIP. Alongside its technical development, the paper …


Navigating Data Contamination in Natural Language Processing: A Comprehensive Survey of Detection and Mitigation Techniques

AI Muhammad, A Mustapha, JT Herrera, JG Sierra…

As machine learning models get more advanced, they face a problem called data contamination. This happens when some of the data used to train the model is also used to test it. Because of this overlap, models might seem to perform well during …


Navigating the linguistic landscape

J Snehi, I Kansal, A Kumar - … Methods for Computational Intelligent Systems: Design …, 2025

The digital realm has experienced a significant shift characterized by a rapid rise in the production of textual data. This section will examine the ways in which the vast amount of unstructured text data is influenced by a vari-ety of contemporary activities …


Navigating the linguistic landscape: Unveiling the future trajectory of natural language processing in an evolving digital era

J Snehi, I Kansal, A Kumar - High-Performance Automation Methods for …

Comprehending unstructured text data from a variety of sources, including emails, social media, and digital periodicals, is becoming increasingly difficult in today’s digital environment due to its exponential expansion. This study investigates how …


neDIOM: Dataset and Analysis of Nepali Idioms

R Pokharel, A Agrawal - Proceedings of the First Workshop on Challenges in …, 2025

Idioms, integral to any language, convey nuanced meanings and cultural references. However, beyond English, few resources exist to support any meaningful exploration of this unique linguistic phenomenon. To facilitate such an inquiry in a …


Negative news posts are less prevalent and generate lower user engagement than non-negative news posts across six countries

S Talaga, D Batorski, M Wojcieszak - arXiv preprint arXiv:2507.19300, 2025

Although news negativity is often studied, missing is comparative evidence on the prevalence of and engagement with negative political and non-political news posts on social media. We use 6,081,134 Facebook posts published between January 1 …


Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset

R Karimi Mahabadi, S Satheesh, S Prabhumoye… - arXiv e-prints, 2025

RK Mahabadi, S Satheesh, S Prabhumoye, M Patwary… - arXiv preprint arXiv …, 2025

… However, existing math-focused datasets built from Common Crawl suffer from degraded quality due to brittle extraction heuristics, lossy … In this work, we introduce Nemotron-CC-Math, a large-scale, high-quality mathematical corpus …


NEMOTRON-CROSSTHINK: Scaling Self-Learning beyond Math Reasoning

SN Akter, S Prabhumoye, M Novikov, S Han, Y Lin… - arXiv preprint arXiv …, 2025

… We (a) curate QA pairs from from synthetic (Common Crawl) and open-source datasets, categorized into general-purpose reasoning (Dgpr) and mathematical reasoning (Dmr); (b) apply structured templates to convert data into multiple-choice (MCQ) …


Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models

A Blakeman, A Basant, A Khattar, A Renduchintala… - arXiv preprint arXiv …, 2025

… To achieve this, we start with technical pre-training documents from Common Crawl and leverage Nemotron-4-340B to generate dialogues, where a knowledgeable persona guides a less-experienced one (eg, an interaction between …


NeoBERT: A Next-Generation BERT

LL Breton, Q Fournier, ME Mezouar, S Chandar - arXiv preprint arXiv:2502.19587, 2025

… become small in comparison to modern web-scraped datasets built by filtering and deduplicating Common Crawl dumps. Following the same trend, we pre-trained NeoBERT on RefinedWeb (Penedo et al., 2023), a massive dataset containing 600B …


NepaliGPT: A Generative Language Model for the Nepali Language

S Pudasaini, A Shakya, S Shrestha, S Bhatta, S Thapa… - arXiv preprint arXiv …, 2025

After the release of ChatGPT, Large Language Models (LLMs) have gained huge popularity in recent days and thousands of variants of LLMs have been released. However, there is no generative language model for the Nepali language, due to …


Network information security protection method based on additive Gaussian noise and mutual information neural network in cloud computing background

Y Zhong, X Li - Egyptian Informatics Journal, 2025

In the cloud computing environment, data security and privacy have received unprecedented attention, but current information security protection methods cannot simultaneously balance data utility and privacy protection effects. Therefore, a …


NEU-ESC: A Comprehensive Vietnamese dataset for Educational Sentiment analysis and topic Classification toward multitask learning

PQH Mai, QH Nguyen, PG Duong, HH Nguyen… - arXiv preprint arXiv …, 2025

In the field of education, understanding students' opinions through their comments is crucial, especially in the Vietnamese language, where resources remain limited. Existing educational datasets often lack domain relevance and student slang. To …


Neural AQG, Part 2: Transformers

M Flor - Automatic Question Generation, 2025

… Unlike UniLM, which was trained on data from English Wikipedia and a corpus of 11K e-books, T5 was trained on a much larger collection of text data—the Common Crawl web archive. Specifically, T5 was trained in 5 different sizes, from T5-small …


Neural dynamics of semantic control underlying generative storytelling

… The first was a CBOW model trained using fastText on a concatenation of the Common Crawl and Wikipedia105, while the second model was trained using the Skip-gram approach on Wikipedia106. The third model was also a Skip-gram model …


Neural Text Embeddings in Psychological Research: A Guide With Examples in R

In this guide, we review neural embedding models and compare three methods of quantifying psychological constructs for use with embeddings: Distributed Dictionary Representation (DDR), Contextualized Construct Representation (CCR), and a …


NeurIPS 2025 E2LM Competition: Early Training Evaluation of Language Models

M Yagoubi, Y Dahou, B Mokeddem, Y Belkada… - arXiv preprint arXiv …, 2025

Existing benchmarks have proven effective for assessing the performance of fully trained large language models. However, we find striking differences in the early training stages of small models, where benchmarks often fail to provide meaningful …


Neurobiber: Fast and Interpretable Stylistic Feature Extraction

K Alkiek, A Wegmann, J Zhu, D Jurgens - arXiv preprint arXiv:2502.18590, 2025

Linguistic style is pivotal for understanding how texts convey meaning and fulfill communicative purposes, yet extracting detailed stylistic features at scale remains challenging. We present Neurobiber, a transformer-based system for fast …


NeuronMerge: Merging Models via Functional Neuron Groups

W Gu, Q Gao, Z Li-Xin, X Shen, J Ye - Findings of the Association for Computational …, 2025

Abstract Model merging techniques like task arithmetic, which combines model parameters through weighted averaging, have proven effective. However, the success of task arithmetic relies on the linearity between model weight differences …


NeuroTrails: Training with Dynamic Sparse Heads as the Key to Effective Ensembling

B Grooten, F Hasanov, C Zhang, Q Xiao, B Wu… - arXiv preprint arXiv …, 2025

Model ensembles have long been a cornerstone for improving generalization and robustness in deep learning. However, their effectiveness often comes at the cost of substantial computational overhead. To address this issue, state-of-the-art methods …


New Kind of Machine Learning—Cellular Automata Model

All the authors of the book join me to acknowledge the contributions of the students who worked under the guidance of authors noted below. The students helped us to set up the experiments and derive the experimental results presented in different …


Next Token Prediction Towards Multimodal Intelligence: A Comprehensive Survey

L Chen, Z Wang, S Ren, L Li, H Zhao, Y Li, Z Cai… - arXiv preprint arXiv …, 2024

Building on the foundations of language modeling in natural language processing, Next Token Prediction (NTP) has evolved into a versatile training objective for machine learning tasks across various modalities, achieving considerable success …


Ngalawan Ujaran Sengit: hate speech detection in indonesian code-mixed social media data

EW Pamungkas, P Chiril - Language Resources and Evaluation, 2025

… We used pre-trained FastText Indonesian word vectors, which have an embedding dimension of 300 and were trained on Wikipedia and Common Crawl Additionally, to leverage the benefits of multilingual word embeddings, we have also …


NGU_Research at CheckThat! 2025: an LLM based hybrid fact-checking pipeline for numerical claims

MA Abdallah, RM Fekry, SR El-Beltagy - Faggioli et al, 2025

In this work, we present a four-stage, retrieval-augmented LLM pipeline for fact-checking numerical claims. The pipeline rewrites each numerical claim into a focused question, fuses OpenAI dense vectors with BM25 to fetch evidence, answers in context with …


NLP modeling recommendations for restricted data availability in clinical settings

F Villena, F Bravo-Marquez, J Dunstan - BMC Medical Informatics and Decision …, 2025

… A multilingual version of XLM-RoBERTa masked language model, pre-trained using a self-supervised technique on a corpus of 2.5 TB of filtered CommonCrawl raw text data containing one hundred languages [43]. This model is the broadest of …


NormLens: Massively Multicultural MLLM Reasoning with Fine-Grained Social Awareness

YR Fung, H Ji - First Workshop on Social Simulation with LLMs

Multimodal large language models (MLLMs) have revolutionized many applications but still face challenges related to cultural bias and a lack of cultural commonsense knowledge crucial for guiding cross-culture communication and interactions. In …


Not All Documents Are What You Need for Extracting Instruction Tuning Data

C Zhang, H Zhong, H Li, C Chai, J Hong, Y Deng… - arXiv preprint arXiv …, 2025

… In reality, there is plenty of high-quality web corpus (eg, Common Crawl) which contains rich knowledge and can be leveraged as highquality instruction data. However, this wealth of knowledge is widely spread within the corpus. Recently, Yue et al. …


Not All Models Suit Expert Offloading: On Local Routing Consistency of Mixture-of-Expert Models

J Liang, S Wang, M Tian, Y Li, D Tang, Z Wei - arXiv preprint arXiv:2505.16056, 2025

Mixture-of-Experts (MoE) enables efficient scaling of large language models (LLMs) with sparsely activated experts during inference. To effectively deploy large MoE models on memory-constrained devices, many systems introduce *expert offloading …


Notifications 0 new

… They trained on 2 trillion tokens of English and Chinese text acquired by deduplicating the Common Crawl. [26] … Further pretrain with 500B tokens (6% DeepSeekMath Corpus, 4% AlgebraicStack, 10% arXiv, 20% GitHub code, 10 …


NusaAksara: A Multimodal and Multilingual Benchmark for Preserving Indonesian Indigenous Scripts

MF Adilazuarda, MI Wijanarko, L Susanto, K Nur'aini… - arXiv preprint arXiv …, 2025

Indonesia is rich in languages and scripts. However, most NLP progress has been made using romanized text. In this paper, we present NusaAksara, a novel public benchmark for Indonesian languages that includes their original scripts. Our …


NusaDialogue: Dialogue Summarization and Generation for Underrepresented and Extremely Low-Resource Languages

A Purwarianti, D Adhista, A Baptiso, M Mahfuzh… - Proceedings of the Second …, 2025

Developing dialogue summarization for extremely low-resource languages is a challenging task. We introduce NusaDialogue, a dialogue summarization dataset for three underrepresented languages in the Malayo-Polynesian language family …


NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Model

A Basant, A Khairnar, A Paithankar, A Khattar… - arXiv preprint arXiv …, 2025

… Our curated Common Crawl-based multilingual data performed slightly better than the Fineweb2-based multilingual data, while the … The diverse pairs translated from English Common Crawl achieved the highest average score over the 8 …


OASIS Uncovers: High-Quality T2I Models, Same Old Stereotypes

S Dehdashtian, G Sreekumar, VN Boddeti - arXiv preprint arXiv:2501.00962, 2025

Images generated by text-to-image (T2I) models often exhibit visual biases and stereotypes of concepts such as culture and profession. Existing quantitative measures of stereotypes are based on statistical parity that does not align with the …


of thesis The Disruption of Due Diligence: How Generative AI is Trans

This thesis explores the emerging role of generative artificial intelligence (GenAI) in transforming due diligence processes within mergers and acquisitions (M&A), with a particular focus on financial and operational due diligence conducted by large …


OLMoASR: Open Models and Data for Training Robust Speech Recognition Models

H Ngo, M Deitke, M Bartelds, S Pratt, J Gardner… - arXiv preprint arXiv …, 2025

Improvements in training data scale and quality have led to significant advances, yet its influence in speech recognition remains underexplored. In this paper, we present a large-scale dataset, OLMoASR-Pool, and series of models, OLMoASR, to study …


On Multilingual Encoder Language Model Compression for Low-Resource Languages

D Gurgurov, M Gregor, J van Genabith, S Ostermann - arXiv preprint arXiv …, 2025

In this paper, we combine two-step knowledge distillation, structured pruning, truncation, and vocabulary trimming for extremely compressing multilingual encoder-only language models for low-resource languages. Our novel approach systematically …


On Regulating Downstream AI Developers

S Williams, J Schuett, M Anderljung - arXiv preprint arXiv:2503.11922, 2025

Foundation models - models trained on broad data that can be adapted to a wide range of downstream tasks - can pose significant risks, ranging from intimate image abuse, cyberattacks, to bioterrorism. To reduce these risks, policymakers are starting …


On resolving the out of vocabulary problem in DisCoCat-based quantum natural language processing

A Bhatuse, A Khandelwal, SS Udmale, MG Chandra… - Evolving Systems, 2025

… In this work, we employ pre-trained embeddings for words from a FastText model trained on the CommonCrawl and Wikipedia corpora (Grave et al. 2018). These word embeddings are later passed to DCA to create the quantum state for each …


On the Application of Fundamental Clustering Methods to Large Scale Cyber Security Log Classification

J Le, M Lazarescu, ST Soh, R Ryan, P Cai, Q Li - 2025 13th International Symposium …, 2025

This paper presents the results from an investigation of using traditional clustering approaches to address the problem of large-scale cyber security log entry classification. We applied two approaches to a large-scale dataset., and we …


On the caveats of AI autophagy

X Xing, F Shi, J Huang, Y Wu, Y Nan, S Zhang, Y Fang… - Nature Machine Intelligence, 2025

Generative artificial intelligence (AI) technologies and large models are producing realistic outputs across various domains, such as images, text, speech and music. Creating these advanced generative models requires significant resources …


On the Effectiveness of Large Language Models in Automating Categorization of Scientific Texts

GK Shahi, O Hummel - arXiv preprint arXiv:2502.15745, 2025

The rapid advancement of Large Language Models (LLMs) has led to a multitude of application opportunities. One traditional task for Information Retrieval systems is the summarization and classification of texts, both of which are important for supporting …


On the effects of machine translation on offensive language detection

A Dmonte, S Satapara, R Alsudais, T Ranasinghe… - Social Network Analysis and …, 2024

Abstract Machine translation (MT) is widely used to translate content on social media platforms aiming to improve accessibility. A great part of the content circulated on social media is user-generated and often contains non-standard spelling, hashtags …


On the Expressiveness of Softmax Attention: A Recurrent Neural Network Perspective

G Mongaras, EC Larson - arXiv preprint arXiv:2507.23632, 2025

… FineWeb is a cleaned and de-duplicated 5-trillion token dataset compiled from 96 different common crawl snapshots. Each tested model is about 300 million parameters and trained on a sequence length of 1024. Adding a gate or norm …


On the Impact of Noise in Differentially Private Text Rewriting

S Meisenbacher, M Chevli, F Matthes - arXiv preprint arXiv:2501.19022, 2025

… To create training samples for sentence infilling, we employ two datasets: English Wikipedia and Common Crawl. The exact dataset … Common Crawl (C4). We employ the Colossal Clean Crawled Corpus (C4) made available by Raffel et al. (2020) …


On The Origin of Cultural Biases in Language Models: From Pre-training Data to Linguistic Phenomena

T Naous, W Xu - arXiv preprint arXiv:2501.04662, 2025

Language Models (LMs) have been shown to exhibit a strong preference towards entities associated with Western culture when operating in non-Western languages. In this paper, we aim to uncover the origins of entity-related cultural biases in LMs by …


On the Path to Make Ukrainian a High-Resource Language

M Haltiuk, A Smywiński-Pohl - Proceedings of the Fourth Ukrainian Natural Language …, 2025

… Unlike some largescale efforts that process raw Common Crawl data directly, we focus on … reuse similar web sources, such as Common Crawl. Duplicates may occur both as exact … from processing the same documents from Common Crawl …


On the Varieties of Fractal Geometry of Word Embeddings

Prior research showed the instability of word embeddings. That is, the neighborhoods of word vectors differ depending on corpora and training methods. In this article we compute, using the correlation dimension algorithm, as well as a …


On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention

Y Ro, Z Zhang, S Kundu, Z Wang, A Akella - arXiv preprint arXiv:2506.09316, 2025

Large language models (LLMs) excel at capturing global token dependencies via self-attention but face prohibitive compute and memory costs on lengthy inputs. While sub-quadratic methods (eg, linear attention) can reduce these costs, they …


Ontology-based Information Extraction from Cultural Heritage Digital Representations: A Case Study in Portuguese Archives

Linked Data (LD) enables cultural heritage institutions to refine archival descriptions and improve findability, but manually creating LD descriptions remains labor-intensive. This paper presents an ontology-guided information extraction system that assists …


Open Problems and a Hypothetical Path Forward in LLM Knowledge Paradigms

X Ye, M Zhang, S Wu - arXiv preprint arXiv:2504.06823, 2025

… Many current LLM pretraining corpora are derived from web scraping, such as Common Crawl 1. Such corpora contain a significant amount of inherently conflicting information: some information is simply incorrect due to the variable quality of …


Open-sci-ref-0.01: open and reproducible reference baselines for language model and dataset comparison

M Nezhurina, T Nakamura, T Carstensen, N Ajroldi… - arXiv preprint arXiv …, 2025

We introduce open-sci-ref, a family of dense transformer models trained as research baselines across multiple model (0.13B to 1.7B parameters) and token scales (up to 1T) on 8 recent open reference datasets. Evaluating the models on various …


Open-Source Large Language Models as Multilingual Crowdworkers: Synthesizing Open-Domain Dialogues in Several Languages With No Examples in Targets and …

A Njifenjou, V Sucal, B Jabaian, F Lefèvre - arXiv preprint arXiv:2503.03462, 2025

The prevailing paradigm in the domain of Open-Domain Dialogue agents predominantly focuses on the English language, encompassing both models and datasets. Furthermore, the financial and temporal investments required for …


OpenCSG Chinese Corpus: A Series of High-quality Chinese Datasets for LLM Training

Y Yu, Z Dai, Z Wang, W Wang, R Chen, J Pei - arXiv preprint arXiv:2501.08197, 2025

… Pretraining large language models (LLMs) requires vast amounts of text data, yet raw datasets such as CommonCrawl (Team, 2024a) are often noisy and unstructured, making direct training inefficient and less effective. To address this, refined corpora …


Openness in AI and downstream governance: A global value chain approach

C Foster - arXiv preprint arXiv:2509.10220, 2025

The rise of AI has been rapid, becoming a leading sector for investment and promising disruptive impacts across the economy. Within the critical analysis of the economic impacts, AI has been aligned to the critical literature on data power and …


OpenThoughts: Data Recipes for Reasoning Models

E Guha, R Marten, S Keh, N Raoof, G Smyrnis… - arXiv preprint arXiv …, 2025

Reasoning models have made rapid progress on many benchmarks involving math, code, and science. Yet, there are still many open questions about the best training recipes for reasoning since state-of-the-art models often rely on proprietary datasets …


Operationalizing Common Crawl News: AI-Enabled Data Pipeline for Large-Scale News Analysis

A El Ouadi, W Knowlton, A Pimentel, D Beskow - 2025 IEEE International systems …, 2025

… This paper proposes an intelligent data system that operationalizes the Common Crawl News dataset for diverse applications, focusing on … Common Crawl News data for various academic, commercial, and government purposes. Additionally, it introduces …


Opinion Mining of Erowid's Experience Reports on LSD and Psilocybin-Containing Mushrooms

A Al-Imam, R Lora, MA Motyka, E Marletta, M Vezzaro… - Drug Safety, 2025

… RoBERTa, created by Facebook AI and released in 2019, is a “Robustly Optimized BERT Approach” trained on a more extensive dataset, including Wikipedia, BookCorpus, and Common Crawl. RoBERTa’s training involves advanced …


OpinioRAG: Towards Generating User-Centric Opinion Highlights from Large-scale Online Reviews

MT Nayeem, D Rafiei - Second Conference on Language Modeling

We study the problem of opinion highlights generation from large volumes of user reviews, often exceeding thousands per entity, where existing methods either fail to scale or produce generic, one-size-fits-all summaries that overlook personalized …


Opportunities and Challenges of Artificial Intelligence in Public Media Journalism

… ChatGPT- 3 was trained on a dataset called Common Crawl that included 100 million tokens (basic units of text) from the NYT and Wikipedia, the latter being one of the largest sources of data available everywhere. Qatar’s Al Jazeera and America’s …


Optical components for binary digital computer

The fundamental building block of modern electronic computers is the transistor. To replace electronic components with optical ones, an equivalent optical transistor is required. This is achieved by crystal optics (using materials with a non-linear …


Optimal Corpus Aware Training for Neural Machine Translation

YH Liao, C Shen - arXiv preprint arXiv:2508.05364, 2025

Corpus Aware Training (CAT) leverages valuable corpus metadata during training by injecting corpus information into each training example, and has been found effective in the literature, commonly known as the "tagging" approach. Models …


Optimising Contextual Embeddings for Meaning Conflation Deficiency Resolution in Low-Resourced Languages

MA Masethe, SO Ojo, HD Masethe - Computers, 2025

Meaning conflation deficiency (MCD) presents a continual obstacle in natural language processing (NLP), especially for low-resourced and morphologically complex languages, where polysemy and contextual ambiguity diminish model …


Optimising Controllable Sentence Simplification for Dutch

F Soete, V Vandeghinste - Computational Linguistics in the Netherlands Journal, 2025

The concept of Easy Language (Vandeghinste et al. 2021) involves the use of simple text, avoiding complex grammatical constructions and difficult vocabulary. Recent approaches (Seidl and Vandeghinste 2024) have shown promising results …


Optimising web accessibility evaluation: Population sourcing methods for web accessibility evaluation

A Hambley, Y Yesilada, M Vigo, S Harper - International Journal of Human-Computer …, 2025

Traditional methods for selecting web pages for evaluation lack a systematic approach. Web accessibility is crucial to improve equal access and usability for individuals with disabilities. However, current approaches to accessibility evaluation …


Optimizing Large Language Models for ESG Activity Detection in Financial Texts

M Birti, F Osborne, A Maurino - arXiv preprint arXiv:2502.21112, 2025

… The models were trained on diverse publicly available data, including CommonCrawl, C4 [17], GitHub repositories, ArXiv papers, Wikipedia, and Books3 [18], ensuring broad linguistic coverage and domain-specific expertise. The 2B model10 …


Optimizing LLM Architectures for Real-Time Applications in Full-Stack Development

T Kannadasan - 2024 3rd International Conference on Automation …, 2024

… To validate the proposed approach, we employ the Common Crawl Corpus dataset as a comprehensive case study, leveraging its extensive and diverse textual data to simulate real-world application scenarios. Our experiments demonstrate …


Optimizing Pause Context in Fine-Tuning Pre-trained Large Language Models for Dementia Detection

X Ke, MW Mak, H Meng - Proc. Interspeech 2025, 2025

Speech pauses serve as a valuable and non-invasive biomarker for the early detection of dementia. Our study aims to examine abnormal pauses, specifically their durations, for improving the detection performance. Inspired by the proven …


Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models

L Belenki, A Agarwal, T Shi, K Toutanova - arXiv preprint arXiv:2502.15950, 2025

We propose a method to optimize language model pre-training data mixtures through efficient approximation of the cross-entropy loss corresponding to each candidate mixture via a Mixture of Data Experts (MDE). We use this approximation …


Optimizing Pretraining Data Mixtures with LLM-Estimated Utility

W Held, B Paranjape, PS Koura, M Lewis, F Zhang… - arXiv preprint arXiv …, 2025

… OLMo V1.7 utilizes near proportional weights, with Wikipedia up-sampled and CommonCrawl data down-sampled – both by a factor of two… We estimate utility for each Dolma dataset, treating each perplexity bucket of CommonCrawl data …


Optimizing Sentiment Analysis in Multilingual Balanced Datasets: A New Comparative Approach to Enhancing Feature Extraction Performance with ML and DL …

H Jakha, S El Houssaini, MA El Houssaini, S Ajjaj… - Applied System Innovation, 2025

Social network platforms have a big impact on the development of companies by influencing clients’ behaviors and sentiments, which directly affect corporate reputations. Analyzing this feedback has become an essential component of …


OrchestraAI: A Multi-Agent Generative AI for Software Development

JLB da Silva Holanda, TS de Souza - Workshop sobre Bots na Engenharia de …, 2025

This paper presents OrchestraAI, a multi-agent AI assistant designed to support professional software developers by combining conversational interaction with autonomous execution of development tasks, including code generation and Git …


Organize the Web: Constructing Domains Enhances Pre-Training Data Curation

A Wettig, K Lo, S Min, H Hajishirzi, D Chen, L Soldaini - arXiv preprint arXiv …, 2025

… in a cleaned pre-training corpus based on CommonCrawl. See Appendix A for detailed … in a cleaned pre-training corpus derived from CommonCrawl. We also compare our domains to k-… Figure 5: Frequency statistics of URL domain names in …


organized by John McCarthy took place in Hanover. Al was defined as the science and engineering of making intelligent machines, especially intelligent computer …

F De Luzi - Engineering Information Systems with Large Language …, 2025

In 1966, Joseph Weizenbaum published ELIZA, 1 which is considered a milestone in the evolution of AI. However, during its development, natural language processing was a challenging task, requiring substantial effort and innovation. In the same year …


Origin of the ring ellipticity in the black hole images of M87

R Dahale, I Cho, K Moriyama, K Wiik, P Tiede… - arXiv preprint arXiv …, 2025

We investigate the origin of the elliptical ring structure observed in the images of the supermassive black hole M87*, aiming to disentangle contributions from gravitational, astrophysical, and imaging effects. Leveraging the enhanced capabilities of the …


OS Agents: A Survey on MLLM-based Agents for Computer, Phone and Browser Use

X Hu, T Xiong, B Yi, Z Wei, R Xiao, Y Chen, J Ye, M Tao…

… 2024a], datasets like CommonCrawl and RICO contain plenty of web page data and mobile screen data. In order to fully utilize these data to further enhance GUI grounding and planning abilities, several methods have been proposed. (1) Rule-Based …


Out of Sight Out of Mind, Out of Sight Out of Mind: Measuring Bias in Language Models Against Overlooked Marginalized Groups in Regional Contexts

F Elsafoury, D Hartmann - arXiv preprint arXiv:2504.12767, 2025

… And XLM-Roberta is trained on Wikipedia and Common Crawl data [70]. Since the majority of the data comes from English Wikipedia and common crawl, we find that on Wikipedia, some of the occurrences of these religious identities are found …


Overcoming Catastrophic Forgetting: Geometric Techniques in Incremental Machine Learning

… This model is a language model of significant scale, trained on a corpus of 300 billion tokens sourced from various text data repositories such as the Common Crawl corpus [27] (which amounts to 570 gigabytes of data after undergoing filtering …


Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling

H Shin, L Ji, X Liu, Y Gong - arXiv preprint arXiv:2503.19123, 2025

Using large teacher models to guide the training of smaller student models has become the prevailing paradigm for efficient and effective learning. However, vocabulary mismatches between teacher and student language models pose …


OVERLORD: Ultimate Scaling of DataLoader for Multi-Source Large Foundation Model Training

J Zhao, Q Lu, W Jia, B Wan, L Zuo, J Feng, J Jiang… - arXiv preprint arXiv …, 2025

Modern frameworks for training large foundation models (LFMs) employ data loaders in a data parallel paradigm. While this design offers implementation simplicity, it introduces two fundamental challenges. First, due to the quadratic …


Overview of Large Language Models for Social and Behavioral Scientists

T Holtdirk, L Saju, L Fröhling, C Wagner - 2025

In this guide, we give an overview of different large language models (LLMs) and their uses for research in the social and behavioral sciences. This guide does not only introduce essential concepts necessary to understand and think about this …


Parallel Corpora for Machine Translation in Low-resource Indic Languages: A Comprehensive Review

R Raja, A Vats - arXiv preprint arXiv:2503.04797, 2025

… Several other corpora derived from Wikipedia and Common Crawl have also contributed to large-scale MT training. The WikiMatrix … In contrast, CCMatrix [35], developed by Meta, is a much larger dataset mined from the CommonCrawl web …


PARAM-1 BharatGen 2.9 B Model

K Pundalik, P Sawarkar, N Sahoo, A Shinde, P Chanda… - arXiv preprint arXiv …, 2025

… While 3.48 trillion tokens come from high-quality English corpora such as FineWeb-Edu, DCLM, Nemotron-CC, and filtered Common Crawl, the remaining 1.52 trillion tokens are composed of rich Hindi data sourced from Books OCR …


Parameter-efficient fine-tuning in large language models: a survey of methodologies

L Wang, S Chen, L Jiang, S Pan, R Cai, S Yang… - Artificial Intelligence Review, 2025

The large language models, as predicted by scaling law forecasts, have made groundbreaking progress in many fields, particularly in natural language generation tasks, where they have approached or even surpassed human levels. However, the …


Parameters vs FLOPs: Scaling Laws for Optimal Sparsity for Mixture-of-Experts Language Models

S Abnar, H Shah, D Busbridge, AME Ali, J Susskind… - arXiv preprint arXiv …, 2025

Scaling the capacity of language models has consistently proven to be a reliable approach for improving performance and unlocking new capabilities. Capacity can be primarily defined by two dimensions: the number of model parameters and the …


Paraphrase detection for Urdu language text using fine-tune BiLSTM framework

MA Aslam, K Khan, W Khan, SU Khan, A Albanyan… - Scientific Reports, 2025

… In this study, we used Common Crawl pre-trained vectors trained on a large amount of web-based text (42 billion tokens, 1.9 million words, 50 d vectors). We downloaded the Common Crawl GloVe embeddings from the official GloVe website …


ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Data

T Chen, F Brahman, J Liu, N Mireshghallah, W Shi… - arXiv preprint arXiv …, 2025

… 2020) dataset, a filtered subset of the Common Crawl1 dataset, specifically designed for inclusion in The Pile (Gao et al.… 2024) often use similar sources such as Common Crawl. … 1https://commoncrawl.org/ 2https://huggingface.co/meta-llama/Llama-3.1-70B-Instruct …


Partial Parameter Updates for Efficient Distributed Training

A Filippova, A Katharopoulos, D Grangier, R Collobert - arXiv preprint arXiv …, 2025

We introduce a memoryand compute-efficient method for low-communication distributed training. Existing methods reduce communication by performing multiple local updates between infrequent global synchronizations. We demonstrate that …


Patent, Still a Leading Indicator in Al Technology Innovation?

S Tang - Tech Transformation and AI Readiness: Pioneering …

… Open repositories like Wikipedia or Common Crawl are a small portion. However, accessing, cleaning, standardizing, and processing relevant data can encounter numerous technical, legal, and financial barriers. A growing number of technology …


Peasant movements by country or region

Several peasant movement in India arose during the colonial era, when economic policies by various British colonial administrations led to the decline of traditional handicraft industries. These policies lead to change of ownership in lands, land …


PEFT A2Z: Parameter-Efficient Fine-Tuning Survey for Large Language and Vision Models

NJ Prottasha, UR Chowdhury, S Mohanto, T Nuzhat… - arXiv preprint arXiv …, 2025

Large models such as Large Language Models (LLMs) and Vision Language Models (VLMs) have transformed artificial intelligence, powering applications in natural language processing, computer vision, and multimodal learning. However …


Performance Evaluation of Deep Learning Models: A Review Based on The F2 Score

MB Tamsamani - Journal of Information Sciences, 2025

Emotion detection is a key area in Natural Language Processing (NLP), with applications ranging from recommendation systems to conversational agents based on the capabilities of large language models. This paper evaluates the effectiveness …


Performance evaluation of GPT-4o on South Korean national exams for building mechanical equipment maintenance

H Choi, J Lee, J Kim - Scientific Reports, 2025

This study evaluates the applicability of large language models (LLMs) in mechanical equipment maintenance in buildings by assessing GPT-4o’s performance on two national certification exams in South Korea: Engineer Energy …


Performance Evaluation of Text Summarization Models on SAMSUM Chat Data

J Bhatia, D Patel, J Patel, M Kumhar, U Chauhan… - Advances in Data-Driven …, 2025

There is widespread dependence on messaging apps and automated chat-bots in various situations. After a long debates, individuals may need to review the discussion's main points. Different approaches for extractive and abstractive text …


Performant Multilingual Modulated and Multiplexed Memory Distilled Model with Adaptive Activation Ensembles

S Dikshit, R Dixit, R Tiwari, P Jain - SN Computer Science, 2025

… [23] introduces multilingual form of RoBERTa trained on a 2.5 TB clean Common Crawl large data corpus from 100 diverse linguistics. A … mT5 is trained on a Common Crawl data corpus containing 101 dialects. Code and network are freely …


Personalised video summarisation using video-text multi-modal fusion

R Akhare, SK Shinde - International Journal of Computational Vision and …, 2025

Video summarisation techniques have evolved in recent years, mostly focusing on visual material and ignoring user preferences. In this work, the topic of query-focused video summarisation is addressed. Long videos are given as input, and the goal is …


Pfefferkorn and EmilIe K. Sunde

… LAION-5b, currently the largest image database, is a dataset of over 5 billion image-text pairs taken from an archive of scraped website data called Common Crawl (LAION 2022). These image-text pairs are extracted from website data; then, a …


EM Tosca, L Aiello, A De Carlo, P Magni - Pharmaceutics, 2025

Background: Open Access Perspective Pharmacometrics in the Age of Large Language Models: A Vision of the Future by Elena Maria Tosca , Ludovica Aiello , Alessandro De Carlo and Paolo Magni * Dipartimento di Ingegneria Industriale e dell’Informazione …


Phi-Ground Tech Report: Advancing Perception in GUI Grounding

M Zhang, Z Xu, J Zhu, Q Dai, K Qiu, Y Yang, C Luo… - arXiv preprint arXiv …, 2025

… To acquire larger-scale data for better scaling up of training, we also obtained web pages from CommonCrawl [50] and rendered … Index and domain deduplication We utilized the CC-MAIN-2024-46 crawl from CommonCrawl. After a …


PhishHunter-XLD: An ensemble approach integrating machine learning and deep learning for phishing URL classification

T Doshi, V Patel, N Shah, D Swain, D Swain, B Acharya - Franklin Open, 2025

Phishing continues to pose a significant cybersecurity threat by deceiving users into disclosing sensitive information through maliciously crafted URLs. Traditional detection methods, including blacklists and heuristic analyses, have proven …


Phishing Attack Detection Through Recursive Feature Elimination Via Cross Validation

S Masmoudi, HM Kammoun, M Charfeddine… - 2025 International Wireless …, 2025

Rising phishing attacks pose serious cybersecurity threats due to their use of fraudulent links to collect confidential user information. In this paper, we evaluate the performance of various Machine Learning (ML) models, including Decision Trees …


Phishing Attack Detection Using Whale Optimization Algorithm-Based Feature Selection

MM Abualhaj, S Al-Khatib, A Alalousi, MO Hiari… - 2025 5th International …, 2025

This study presents an optimized phishing attack detection model integrating the Whale Optimization Algorithm (WOA) for feature selection with XGBoost and Support Vector Machine (SVM) classifiers. The proposed approach enhances classification …


Phishing Detection in the Age of NLP: Leveraging Deep and Machine Learning for Enhanced Accuracy

MMH Melon, Y Arafat, S Zareen, RM Alsharfa…

Phishing remains a widespread cybersecurity attack, leveraging social engineering techniques to trick users and acquire sensitive information. Conventional detection mechanisms tend to fail to detect advanced phishing attacks, especially those that …


Phishing Detection Methods

SD Abualgasim, ZE Ahmed - Critical Phishing Defense Strategies and Digital Asset …, 2025

This chapter explores the evolution of phishing detection methods, present traditional, advanced, and hybrid approaches. Traditional methods provide a base layer of defense, but their effectiveness is limited against adaptive attacks. Advanced …


Phishing Email Detection With Data Augmentation Using LLMs

… PhishTank [13], OpenPhish [15], Alexa [16] and Common Crawl Archive [17]. This contained 50,000 phishing and 50,000 legitimate emails. … In [19] CNN was used with the PhishTank [13]dataset that contained 10,604 phishing websites and the …


Phishing Prevention in the Digital Age: An AI/ML Perspective

A Kadam, H Khirid, A Gawande, RB Chandrayan - Intelligent Strategies for ICT …, 2025

Phishing means cybercrime where attackers pose themselves as a known legitimate user to plifer sensitive information such as passwords and financial details through misleading emails or websites. Hence, this type of continuous threat persists despite …


Phishing URL Detection via Machine Learning: A Comprehensive Survey

J Islam, C Patra, PK Mani, S Biswas, D Giri, T Maitra - Proceedings of International …

… • Common Crawl [44]: Common Crawl provides a publicly accessible dataset that includes extensive metadata and content from benign websites collected through large-scale web crawling. This dataset is valuable for phishing detection research as …


Phishing Webpage Detection using URL and HTML Graphs based on a Multimodal AutoEncoder Ensemble

윤준호， 최석훈， 김혜정， 부석준 - Journal of KIISE, 2025

인터넷의 발전으로 인해 피싱 공격에 노출되는 사용자가 증가하고 있으며, 이를 예방 하기 위한 효과적인 탐지 방법이 필수적이다. 기존의 피싱 탐지 방법은 주로 URL의 문자 시퀀스를 분석하는 데 중점을 두었으나, 피싱 URL은 정상 URL과 유사한 패턴을 …


PhishKey: A Novel Centroid-Based Approach for Enhanced Phishing Detection Using Adaptive HTML Component Extraction

F Castaño, E Fidalgo, E Alegre, R Alaiz-Rodríguez… - arXiv preprint arXiv …, 2025

Phishing attacks pose a significant cybersecurity threat, evolving rapidly to bypass detection mechanisms and exploit human vulnerabilities. This paper introduces PhishKey to address the challenges of adaptability, robustness, and efficiency …


PhishSecure: Enhancing Web Safety

C Shravage, S Vairagar, P Metri, SC Jaygude… - Smart Trends in Computing and …

This project presents an innovative phishing detection system that addresses the limitations of traditional methods by combining URL-based and content-based features to accurately identify fraudulent websites. Unlike conventional approaches …


PhreshPhish: A Real-World, High-Quality, Large-Scale Phishing Website Dataset and Benchmark

T Dalton, H Gowda, G Rao, S Pargi, AH Khodabakhshi… - arXiv preprint arXiv …, 2025

… These user-sourced URLs provide a more realistic and representative sample of benign pages that users are likely to encounter on the web as opposed to those from other datasets such as Common Crawl [12]. The dataset was cleaned and curated to …


Phylolm: Inferring the phylogeny of large language models and predicting their performances in benchmarks

… own version of the Common Crawl dataset and thus share a similar training set. Lastly, some GPT-3 models (ada, babbage and curie) appear to be close to this OPT,Pythia and Falcon-RW cluster showing they may have been trained on a version of the …


PiKE: Adaptive Data Mixing for Multi-Task Learning Under Low Gradient Conflicts

Z Li, Y Deng, P Zhong, M Razaviyayn, V Mirrokni - arXiv preprint arXiv:2502.06244, 2025

… We evaluate PiKE in two multitask pretraining scenarios: 1) Pretraining language models on multilingual mC4 dataset [69], a dataset covering diverse languages from Common Crawl corpus. 2) Pretraining language models on the GLaM dataset [16] …


This thesis investigates the potential and limitations of using Generative AI (GenAI) in terms of quality and capability in agile web development projects using React. For this purpose, the Pipeline for Automated Code Generation from Backlog Items (PACGBI) …


Pirates of the RAG: Adaptively Attacking LLMs to Leak Knowledge Bases

C Di Maio, C Cosci, M Maggini, V Poggioni, S Melacci

The growing ubiquity of Retrieval-Augmented Generation (RAG) systems in several realworld services triggers severe concerns about their security. A RAG system improves the generative capabilities of a Large Language Models (LLM) by a …


PLaMo 2 Technical Report

P Networks, K Chubachi, Y Fujita, S Hemmi… - arXiv preprint arXiv …, 2025

… 100B, extracting data deemed particularly relevant to coding from CommonCrawl data. This time, we implemented the following methods: … Removal of irrelevant data through filtering: Since parsing all HTML content from CommonCrawl data to …


Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation

T Liu, J Qi, P He, A Bisazza, M Sachan, R Cotterell

Recent work suggests that large language models enhanced with retrieval-augmented generation are easily influenced by the order in which the retrieved documents are presented to the model when solving tasks such as question answering (QA) …


POLISH LANGUAGE MODELS IN BUSINESS AND PUBLIC SECTOR: A STRATEGIC PERSPECTIVE

Purpose: The aim of this article is to analyse the first two Polish language models (Large Language Models) from the point of view of strategic dimensions of implementing national (LLMs) in the business and public sector. The study outlines both the …


Political Leaning and Politicalness Classification of Texts

M Volf, J Simko - arXiv preprint arXiv:2507.13913, 2025

This paper addresses the challenge of automatically classifying text according to political leaning and politicalness using transformer models. We compose a comprehensive overview of existing datasets and models for these tasks, finding that …


PolyPrompt: Automating Knowledge Extraction from Multilingual Language Models with Dynamic Prompt Generation

N Roll - arXiv preprint arXiv:2502.19756, 2025

Large language models (LLMs) showcase increasingly impressive English benchmark scores, however their performance profiles remain inconsistent across multilingual settings. To address this gap, we introduce PolyPrompt, a novel …


PolyTruth: Multilingual Disinformation Detection using Transformer-Based Language Models

Z Gouliev, J Waters, C Wang - arXiv preprint arXiv:2509.10737, 2025

Disinformation spreads rapidly across linguistic boundaries, yet most AI models are still benchmarked only on English. We address this gap with a systematic comparison of five multilingual transformer models: mBERT, XLM, XLM-RoBERTa …


POS tagging of low-resource Pashto language: annotated corpus and BERT-based model

I Haq, Y Zhang, IA Qadri - Language Resources and Evaluation, 2025

This paper presents the development of a comprehensive part-of-speech (POS) annotated corpus for the low-resource Pashto language, along with a deep learning model for automatic POS tagging. The corpus comprises approximately 700K words (30K …


Position: Beyond Euclidean--Foundation Models Should Embrace Non-Euclidean Geometries

N He, J Liu, B Zhang, N Bui, A Maatouk, M Yang, I King… - arXiv preprint arXiv …, 2025

In the era of foundation models and Large Language Models (LLMs), Euclidean space has been the de facto geometric setting for machine learning architectures. However, recent literature has demonstrated that this choice comes with …


Position: Formal Mathematical Reasoning—A New Frontier in AI

K Yang, G Poesia, J He, W Li, KE Lauter, S Chaudhuri… - Forty-second International …

… 2024) as the base math LLM, which was trained on high-quality mathematical documents retrieved from Common Crawl through a carefully engineered data pipeline that combined automatic filtering and manual annotation. …


Position: The Most Expensive Part of an LLM should be its Training Data

N Kandpal, C Raffel - arXiv preprint arXiv:2504.12427, 2025

… Unlike other resources needed to produce LLMs, like hardware or energy, most training data has historically been collected for virtually no cost by mining text from the public Internet (Common Crawl). This webscraped data is foundational to LLMs …


Position: We need responsible, application-driven (RAD) AI research

S Hartman, CS Ong, J Powles, P Kuhnert - arXiv preprint arXiv:2505.04104, 2025

This position paper argues that achieving meaningful scientific and societal advances with artificial intelligence (AI) requires a responsible, application-driven approach (RAD) to AI research. As AI is increasingly integrated into society, AI …


Position: When Incentives Backfire, Data Stops Being Human

S Santy, P Bhattacharya, MH Ribeiro, KR Allen, S Oh - Forty-second International …

Progress in AI has relied on human-generated data, from annotator marketplaces to the wider Internet. However, the widespread use of large language models now threatens the quality and integrity of human-generated data on these very platforms …


Positional Fragility in LLMs: How Offset Effects Reshape Our Understanding of Memorization Risks

Y Xu, A Bosselut, I Schlag - arXiv preprint arXiv:2505.13171, 2025

Large language models are known to memorize parts of their training data, posing risk of copyright violations. To systematically examine this risk, we pretrain language models (1B/3B/8B) from scratch on 83B tokens, mixing web-scale data with public …


Post navigation

… LLMs and common crawl data sets are out there in the world now. The genie is very much out of the bottle and there’s not a great deal we can do to put it back, even if we wanted to. It’s also debatable what, if anything, content creators, organisations …


Practical Datasets for Analyzing LLM Corpora Derived from Common Crawl

N Hagar, J Bandy - Proceedings of the International AAAI Conference on …, 2025

Large language models (LLMs) rely heavily on web-derived training datasets, yet understanding how filtering and curation decisions affect these datasets remains challenging. This paper presents two complementary datasets designed to enable …


Practical Necromancy for Beginners: A Short Incomplete Opinionated Introduction to Artificial Intelligence for Archaeology and History Students

I will probably get this wrong: but maybe it will be wrong in useful and interesting ways. This book isn’ta hymn of praise to artificial intelligence. It’s not even all that scholarly a book. This is the book that I wish I had handy that day in September of …


PrahokBART: A Pre-trained Sequence-to-Sequence Model for Khmer Natural Language Generation

H Kaing, R Dabre, H Song, VH Tran, H Tanaka… - Proceedings of the 31st …, 2025

… We also find that some Khmer texts, particularly from Common Crawl (CC), were tokenized with spaces as word delimiters. While we cannot trace the exact source, these texts likely originated from preprocessed corpora. Additionally, the functional …