My flow chart for reading a paper is like
Abstract -> Data availability

if code & data available: -> figures -> methods -> get the data -> run the code on the data...
if no data available: paper is effectively creative writing. Ignore or begin adversarial evaluation.

Why would you pique suspicion in your reader and signal something all may not be as it seems by not publishing the data? Why would anyone believe a paper that doesn't have code & data? You claim to have gotten a bunch of data and done a bunch of things to it in code, and you had to have done that in an organized enough way to yield a paper, so even if its not pretty, why wouldn't you just dump whatever you have on zenodo and remove an easy avenue for someone to dismiss your work?

Edit: clearly, caveats apply like if the data is privacy-sensitive health/PII or data under a strict license where you'd get sued if you post it. That's not what I'm talking about, and its fine if you explain that and post whatever derived data you can ethically and legally post. I'm talking about most primary research in my field which has no such limitations.

@jonny I'll try to answer, although it was probably a rethorical question but I think it's useful to see the point of view of experimentalists..

First, what data are we talking about? The widely-processed, very simplified tables that you use at the last step of analysis just to make your figures? Or the raw data that hasn't been spike-sorted, checked, or corrected when needed? Or something in between?

If we're talking about the first one, sure, it doesn't cost much to share it, but it's also very easy to fake, and checking that you can remake the figures from those is not going to catch any kind of error, since the author would have very likely noticed if there was an error at that stage. Just looking at the figure-making code would probably be more useful to catch errors. So: does it really change anything to publish that data? I'm not sure.

If we're talking about raw data, not only that's going to be much more difficult to share (e.g. size might be a problem, storage might have a cost, reuse might be a problem without also providing an explanation of all the processing steps done on the data), it's also going to take the time of the person dealing with the sharing. There is the added problem that the author might want to analyse more aspects of the data and sharing exposes them to being scooped. Of course that wouldn't be a problem if people analysing your data were sharing authorship with you (but they generally don't) or if the system wasn't stupidly geared towards encouraging high-impact, non-collaborative publications.

Sharing something in between raw and final will have problems from either stage, so once again: is it worth it?

So, you ask:
"Why would you pique suspicion in your reader and signal something all may not be as it seems by not publishing the data? Why would anyone believe a paper that doesn't have code & data?"

-> it might be worth more to the author to use their time & money for something else. Plus, do we really think that scientists who don't share data are faking the data? I don't. Do we think that scientists who share the data never fake the data? I don't either.

If a scientist wants to bend results some way there are many ways to do this other than faking data (p-hacking, cheating at the experimental stage, hiding results, over-inflating results..). I think it's better to read the reasoning of the paper, the methods, what kind of analyses were done, and look at the individual data distributions in the figures (these should definitely be shown).

Overall: obviously, it is better, if everything else was equal, to have as much data shared as possible. Unfortunately, once you take other parameters into account, there is a question of cost-benefit and it's not always in favour of sharing the data.

PS: it is almost never the professor that actually deals with data-sharing, but I'm sure you probably know that

#AcademicChatter

@elduvelle
yes i am talking about the raw data. there is a gradient there too, and given the state of the tooling it's often very hard to reproduce the "first step" of the analysis like extracting traces/spikes from raw recordings because it involves a lot of manual clicking around that isn't easily captured in code, and i have a bit of grace there since that's not the author's fault, that's the state of the tools. but yes, the complete, raw data should be shared including videos and raw recordings. even if that 'first step' cant be perfectly and automatically replicated, it's usually possible to get close enough for replication.

e.g. size might be a problem, storage might have a cost, reuse might be a problem without also providing an explanation of all the processing steps done on the data

again neuroscientists are never given training in this, but the data is almost never as big as they think once the data are properly encoded and compressed, and repositories like zenodo and dandi are effectively free-forever unlimited storage archives, as they should be. i'm not expecting perfectly curated analysis pipelines and am expecting to have to fight through the thicket a bit, that's all completely fine -- as long as it's there!

There is the added problem that the author might want to analyse more aspects of the data and sharing exposes them to being scooped.

I've never seen or heard of this happening, where open data caused some future result to get scooped, but a) it's fine to just say that in a data availability statement: "the data and code will be made available in 3 months after we finish up a secondary analysis of it," b) you have a huge head start so someone would have to be very quick to do that, and c) i actually think that's a good thing: it's ridiculously wasteful (and cruel, in the case of animal research) to need to re-do every experiment every time someone wants to investigate a phenomenon. someone re-analyzing and publishing on open data is a good thing. if i was an editor or reviewer, if i saw such a paper, i would require that the source of the data is properly cited and ask if the secondary authors had made contact with the primary authors and what their view on the secondary paper was. if the primary authors are like "hold up we're writing this exact thing" i'd suggest merging efforts.

it might be worth more to the author to use their time & money for something else

my standard is mere existence, i don't have an estimate of how much time and money it costs on an average case, but given that standard i would suspect "not much" since all im' asking for it "give it to me however you have it"

Plus, do we really think that scientists who don't share data are faking the data? [...] If a scientist wants to bend results some way there are many ways to do this other than faking data/

this is exactly the point. the argument isn't that the data is fake, but that the fine details of the analysis are cherrypicked to tell a particular story, and if the data is not shared, my first thought is that the authors don't want me looking too closely at it or else i'll see all the little tweaks and choices made in service of that story. the best way to demonstrate that your conclusions hold up to scrutiny and are not the result of some carefully picked garden path analysis is to share the data.

it is almost never the professor that actually deals with data-sharing

usually, yes, but oddly i am in the middle of a request where that is the case.

@jonny @elduvelle I might have been unlucky, but every time I've asked for data to the authors of a paper who said that data was available on request they either ignored me or the data had issues ...

Anyway, two interesting reads about this

Why don't we share data and code? Perceived barriers and benefits to public archiving practices - Gomes et al 2022
https://royalsocietypublishing.org/doi/10.1098/rspb.2022.1113

Why data is never raw - On the seductive myth of information free of human judgment
https://www.thenewatlantis.com/publications/why-data-is-never-raw