1/2 Looking at one of the #xz writeup, this struck my eye: “The release tarballs upstream publishes don't have the same code that GitHub has. This is common in C projects so that downstream consumers don't need to remember how to run autotools and autoconf.” Ah, GNU AutoHell, I remember it well. Tl;dr: With AutoHell, even if you're building for a 19-bit Multics variant from 1988, it’s got your back. Except for it’s just too hard to understand and use, thus the above.
2/2 Thus, another #xz lesson. Don’t rely on build tools you don’t understand generally, and don’t rely on GNU AutoHell specifically.
[Yes, I understand this hack has many more moving parts, most much more sophisticated. But I didn't see anyone else saying this.]
@timbray I'm also seeing surprisingly many lessons in there.
A partial list from me: Security through obscurity doesn't work. Everything is a security boundary, even build scripts and test files. Everything should be buildable from source and human-readable, even test files.
An indirect one that I remember sharply for having had to deal with it a lot in a past job: not everyone in an Open Source community has the same goals and priorities.
Re test files, I don't think that is desirable, especially for parser-like it compressor-like projects. In case of anything that smells of passing, fuzzer-generated regression tests have significant positive value (one can try to write regression tests by hand instead, but it's more work for imo an infrared chance of getting it wrong) and fuzzer-generated example inputs to parsing have a very large value. Example files that were generated using weird tools or extend weird ones are also important test cases (as opposed to previous ones, not just to assert lack of crashes, but to assert correct parsing).
I think that having a better split between building (which generates all non-test artifacts) and testing (that uses already-built everything else, generates test artifacts and runs them) solves the same problem: it allows build environments to ensure that testing doesn't affect the output and that test files are not inspected by the build process. If done sufficiently well (which is admittedly hard to do in the current world) this can even allow test-only dependencies to not be visible to the build stage.
@robryk @timbray You and I are reaching the same conclusion: chances are, building and testing should be separate phases, separate enough that test files aren't available while building.
That should be especially true for large binary-only test files that can't be reviewed and have to be blindly trusted - being binary-only, those have no reason to have to be processed through a build pipeline.