Martin Ruskov @mapto@qoto.org

The case of “vegetative electron microscopy” illustrated here shows what is badly needed in current #LLM research and has implications far beyond. We need tools that help us curate huge corpora. We need to be able to trace #hallucinations back to the training data and understand what are the specific (to a surprise, often #deterministic) reasons in the model input that cause that particular output.

If anyone is interested in collaborating on this, I'm in, have done some small-scale experiments and have already submitted a grant proposal.
https://theconversation.com/a-weird-phrase-is-plaguing-scientific-papers-and-we-traced-it-back-to-a-glitch-in-ai-training-data-254463

I think I now know where to draw the line between "good" and "bad" #GenAI, and possibly (or rather obviously) the same for #machineLearning. It's simply whether the input data has been constructed rigorously. Put this way it's the most obvious statement ever, but somehow #BigTech have convinced us all that they advance research by recklessly scraping #twitter, #4chan and who knows what else (they keep their training data secret).

What is good science in computational linguistics? Well, open data is a step towards it. But open and crap is not a solution. We need to actually _know_ and manage the data. And nobody in their right mind would want to plough through toxic data to clean it. We've all heard the horrors of Kenyan data workers who do it for money and still suffer doing it.

But better (yes, also smaller) corpora are of interest to scholars in the humanities and the social sciences. Think of https://textcreationpartnership.org or https://mlat.uzh.ch. Yes, they are too big for individual researchers or even teams to handle, but we have the organisational and technological infrastructure to work on them collectively. We've been doing it for ages and we will continue doing it. We just need to do it together.

And this is the goal of the European Research Council project proposal I'm submitting in this very moment.

Today at #CHR2025, I will be presenting our work on the evaluation of the historical adequacy of masked language models (MLMs) for #Latin. There are several models like this, and they represent the current state of the art for a number of downstream tasks, like semantic change and text reuse detection. However, a historical researcher, philologist or else would want to be sure that such models really represent the historical period of interest. For example, it would be an embarrasing hallucination if St. Augustine showed up in the context of the Roman senate.

Our evaluation confirms a known problem: LLMs and masked models in particular are trained on corpora without attention to historical periods. Unlike other research we've done on Early Modern English, this problem leads to models being barely distinguishable when it comes to their ability to generate based on a historical period. Even though history is a case where it is most obvious when models go wrong, this type of contamination is a known problem for LLM training overall, think of different legal jurisdictions using the same language, dialects in programming languages, etc.

This research was generously supported by AgileLab.

The full paper is available at:
https://anthology.ach.org/volumes/vol0003/the-latin-language-evolved-over-time-masked-models/

Our paper on the values found in fairy tales from some European countries has been published. We studied how values are explicitly present in tales from Germany, Italy and Portugal using various NLP techniques, but most notably Word2Vec and Word Embedding with a Compass. We visualise synchronic semantic variation to show certain differences based on observations of the corpus, some of them already observed in previous literature. A discussed example in our findings is how motherhood in Germany is strongly related to generosity, whereas in Italy and Portugal it has stronger relationship to wisdom.

Fulltext available at: https://aclanthology.org/2023.nlp4dh-1.8/

@folklore @linguistics @bookstodon

In the morning session today Sara Sullam and I will be presenting our work on exploring nominal (in our case study - bibliographical) data. We do it by borrowing a method from educational research - the notion of phenomenographic variation. #CHR2023🧵

Regarding the extremely insightful exchange between @tante and @pluralistic that unfolded yesterday, there is a bit too much of ideology in it for my taste. However, I couldn't help noticing that arguably the strongest point of Tante about the ideology of LLMs is based on an invalid argument he developed in 2024: the claim that "open-source LLMs do not really exist".

Tante makes a case that open weights is not open source, and that's a valid point. Back then it was probably difficult to see the open source (open weights, open data and more) LLMs that actually exist. Many of them are specialised, and commonly they'd perform even worse than open weights ones. Yet, they are out there and I'd claim they are inevitably going to be an important part of the future.

I've been studying particularly specialised models like MacBERTh, but there are also open autoregressive instruction-tuned (i.e. chatgpt-like) models. Now there's even the Model Openness Framework and the corresponding tool and ranking: https://mot.isitopen.ai

This might be seen as opening a conversation about whether we can separate affordances of technologies from their politics, but as I said, I have too many doubts about ideology to be willing to go down that path.

PS: I come late to the conversation and I'm a nobody in this community. Yet, if curious, I invite you to see my pinned posts to see what I'm doing around the topic.

Tante's original post: https://tante.cc/2024/10/16/does-open-source-ai-really-exist/

and here's the mechanism: they threaten and then shift the blame. European companies not planning to migrate away are falling behind.

https://www.theregister.com/2026/02/18/microsoft_asks_uk_parliament_to_correct_record/

US sanctions on ICC should be used as an indicator of where to direct efforts to grow independence from US businesses

https://www.euronews.com/my-europe/2026/02/18/us-sanctions-turn-international-criminal-court-judges-daily-life-into-a-nightmare

Il Pentagono vuole classificare Anthropic AI come "supply chain risk"

"La controversia sembra derivare dalla riluttanza di Anthropic a consentire un uso militare illimitato dei propri sistemi di IA. Attualmente, i modelli di Anthropic sono gli unici strumenti di IA disponibili all'interno di sistemi militari classificati tramite fornitori terzi, come Palantir Technologies, e anche in questo caso sono soggetti ad alcune restrizioni. Reuters ha precedentemente riferito che i dirigenti di Anthropic hanno comunicato ai funzionari militari di non volere che i propri sistemi fossero utilizzati per il puntamento di armi autonome o la sorveglianza interna."
https://gizmodo.com/pentagon-considers-designating-anthropic-ai-as-a-supply-chain-risk-report-2000722805
@aitech