i know some people oppose the widespread use of CI on ideological grounds, so i think it's worth it thinking about why we value it
-
@whitequark @wwahammy My view is that any CI system that wants to be non-abusive to third parties' network resources needs to limit all fetching to content-addressed storage with caching in the CI host layer. No direct URL fetching/network access.
@whitequark @wwahammy I really really also don't like the "destroy the world and start over" that makes it take minutes to get CI results and know if your change needs revision to have a chance at being acceptable. It could and should be an incremental make that finishes in milliseconds when you've only made a localized change. By tracking cached CI results & artifacts linked to commits & configurations, a virtual overlay of the artifacts for the parent commit could always be used as the starting point for building with a proposed new commit, yielding near-instant results (assuming the build system is decent and handles incremental builds).
-
@whitequark @wwahammy I really really also don't like the "destroy the world and start over" that makes it take minutes to get CI results and know if your change needs revision to have a chance at being acceptable. It could and should be an incremental make that finishes in milliseconds when you've only made a localized change. By tracking cached CI results & artifacts linked to commits & configurations, a virtual overlay of the artifacts for the parent commit could always be used as the starting point for building with a proposed new commit, yielding near-instant results (assuming the build system is decent and handles incremental builds).
-
i know some people oppose the widespread use of CI on ideological grounds, so i think it's worth it thinking about why we value it
for me, aside from the obvious reliability aspect (that could probably be mostly achieved by every contributor having a pre-commit hook that runs tests in a nix shell or something), the key utility provided by CI is legibility:
- when a change is made, me and others are on the same page on how it's tested and what the acceptance criteria are
- when a release is made, me and others are on the same page on which steps are taken and in which environment to build and upload artifacts
this is almost more important than the added reliability:
- i do not want to have to maintain a special environment on my special machine that is able to do releases
- i want to empower contributors who aren't me to do releases on their own with just forge access
this is enough of a benefit that the risks from GitHub Actions design issues are worth mitigating in order to use the workflow
@whitequark No experience with Github Actions specifically, but even in a small (<10 person) company, having automatic builds were definitely worth the hassle, even if someone had to manually upload them to make the release available to customers.
At the very least, being able to trust that a given version/build number maps to a specified revision is huge.
-
right, so (for context) my responses to this would be:
I value reducing human misery a lot more than I value conserving energy, so I don't consider CI energy use "waste" unless there are specific ways in which it can be optimized but isn't.
I'm actively working on community run CI infrastructure so my position here should be obvious
"destroy the world" recipes are actively necessary to tackle this problem. GHA-style workflows, for all their faults, significantly limit how clever you can be in setting up your CI infrastructure, so the chances that you can turn a GHA workflow into a usable series of steps for your OS quickly is quite high—more so than if it was a heavily customized Buildbot workflow, for example
@whitequark @dalias @wwahammy Wait, you value reducing human misery? In this society, this economy and this industry?
(I mostly joke: I actually agree entirely with that value, but it does feel like a pretty marginal position these days)
-
@whitequark @wwahammy Well it's going to have to run on the CI side anyway when the PR is updated.
-
@dalias @wwahammy one thing I'm unsure about is downloads. the tradeoffs here aren't obvious; depending on what you're doing, caching stuff you download (even repeatedly) locally can end up using scarcer resources more aggressively (bandwidth can be a much more available resource).
the best case scenario here is something like Nix flakes which are intrinsically cacheable, but if you ever let external contributors run workflows, you run the risk of poisoning this cache (Nix isn't hardened enough against a malicious builder).
one day i'll have the answers to this
@whitequark @dalias @wwahammy if a malicious Nix builder can poison the cache, wouldn't that mean that a multiuser Nix system is insecure as well, since unprivileged users are allowed to build and install packages? -
@whitequark @wwahammy My view is that any CI system that wants to be non-abusive to third parties' network resources needs to limit all fetching to content-addressed storage with caching in the CI host layer. No direct URL fetching/network access.
@dalias @wwahammy I know this is a real problem (PyPI and Rubygems have both considered measures against excessive bandwidth use, mostly by CI services) but I don't think this is the solution; if someone says I should use a CI system where
git cloneandpip installdon't work I would simply consider it defective and pick a different one. and as stated, this seems like it would entirely prevent anything that uses HTTPS to talk to the network (so, basically everything) from working unless every individual tool is going to be upgraded with this system in mind which seems unlikely -
@whitequark @dalias @wwahammy Wait, you value reducing human misery? In this society, this economy and this industry?
(I mostly joke: I actually agree entirely with that value, but it does feel like a pretty marginal position these days)
@iris_meredith @dalias @wwahammy my entire motivation for building OSS (in the particular way that I do it) comes down to "the industry / the incumbents are making this miserable as fuck so I'll fix it"
see: Vivado, Verilog, etc
-
@whitequark @wwahammy Well it's going to have to run on the CI side anyway when the PR is updated.
-
@dalias @wwahammy I know this is a real problem (PyPI and Rubygems have both considered measures against excessive bandwidth use, mostly by CI services) but I don't think this is the solution; if someone says I should use a CI system where
git cloneandpip installdon't work I would simply consider it defective and pick a different one. and as stated, this seems like it would entirely prevent anything that uses HTTPS to talk to the network (so, basically everything) from working unless every individual tool is going to be upgraded with this system in mind which seems unlikely@whitequark @wwahammy Why would you ever do a git clone of third-party repos as part of CI? You just need the version you're building with, in which case you can request the archive of that, which can then be content-addressed by its hash. You don't need the entire history which is probably a few orders of magnitude larger.
-
@whitequark @wwahammy I think I'm one of them so I'll go over a few:
1. Resource usage externalities when this is done at scale, especially for large projects times large numbers of PR authors. This manifests as energy waste, hammering the servers/infrastructure of software you depend on and pull dynamically in standard "destroy the world and re-run everything from scratch" CI recipes, etc.
2. Dependency on subsidized compute resources from a capitalist platform with motivation to lock you in and enshittify.
3. Reducing or eliminating the mandate for your software to be independently buildable by people on their own systems without your CI infrastructure.
@dalias @whitequark @wwahammy ok so your not against CI/CD, your just against GitHub Actions specifically.
What would you recommend instead?
-
@whitequark @dalias @wwahammy if a malicious Nix builder can poison the cache, wouldn't that mean that a multiuser Nix system is insecure as well, since unprivileged users are allowed to build and install packages?
@noisytoot @dalias @wwahammy I was thinking about "substituters". as far as I'm aware nothing stops you from editing the stuff in the Nix store if you have the right privileges (directly or via a service) and it's pretty hard to detect if it's ever done, therefore I wouldn't rely just on Nix to prevent cache poisoning (especially in light of regularly dropping Linux LPEs)
-
@whitequark @wwahammy Why would you ever do a git clone of third-party repos as part of CI? You just need the version you're building with, in which case you can request the archive of that, which can then be content-addressed by its hash. You don't need the entire history which is probably a few orders of magnitude larger.
@dalias @wwahammy most of the time? because it's a submodule. sometimes a recursive submodule.
github's default actions/checkout does a shallow clone (which is just as efficient), but some packages do actually look at their own history in order to give accurate git-describe results or turn git distance numbers into version numbers. your workflow isn't my workflow
-
@dalias @whitequark @wwahammy ok so your not against CI/CD, your just against GitHub Actions specifically.
What would you recommend instead?
@valorzard @whitequark @wwahammy Well I'm against a number of standard CI/CD practices that are harmful to parties not even involved in the project using the CI/CD.
I don't have a specific recommendation for something I haven't wanted to use. I don't think the whole purpose of CI/CD is that important because I don't think we should be expecting non-developers to be using a continuous rolling main branch rather than discrete releases the maintainers have confidence in. If other people want to do that, fine, but finding the right tooling to do it without externalities impacting others is on them not me.
-
@dalias @wwahammy most of the time? because it's a submodule. sometimes a recursive submodule.
github's default actions/checkout does a shallow clone (which is just as efficient), but some packages do actually look at their own history in order to give accurate git-describe results or turn git distance numbers into version numbers. your workflow isn't my workflow
@dalias @wwahammy also I'm pretty sure that at least with Forgejo, it takes less resources to do a git shallow clone than it takes to download an archive of a commit (because the archive needs to be generated and then stored, and all of them are fully denormalized, while git does some sort of optimization with pack files I think?)
-
@noisytoot @dalias @wwahammy I was thinking about "substituters". as far as I'm aware nothing stops you from editing the stuff in the Nix store if you have the right privileges (directly or via a service) and it's pretty hard to detect if it's ever done, therefore I wouldn't rely just on Nix to prevent cache poisoning (especially in light of regularly dropping Linux LPEs)
@whitequark @dalias @wwahammy LPEs are certainly an issue (although they're also an issue for any CI that doesn't use proper VMs), but Nix doesn't just allow any random unprivileged user to configure a substituter, right? -
@dalias @wwahammy most of the time? because it's a submodule. sometimes a recursive submodule.
github's default actions/checkout does a shallow clone (which is just as efficient), but some packages do actually look at their own history in order to give accurate git-describe results or turn git distance numbers into version numbers. your workflow isn't my workflow
@whitequark @wwahammy OK, but that's the fault of the CI system doing a shallow clone rather than a fully recursive checkout from already-cloned-and-cached repositories. It's the fault of poor abstraction layers that behave as "just do whatever you want to script in this throwaway container" rather than something more structured.
-
@dalias @wwahammy also I'm pretty sure that at least with Forgejo, it takes less resources to do a git shallow clone than it takes to download an archive of a commit (because the archive needs to be generated and then stored, and all of them are fully denormalized, while git does some sort of optimization with pack files I think?)
@whitequark @wwahammy I don't see why the archive would need to be stored. Tarballs are fully streamable and the git-archive command emits them as a stream not with temporary storage.
-
@valorzard @whitequark @wwahammy Well I'm against a number of standard CI/CD practices that are harmful to parties not even involved in the project using the CI/CD.
I don't have a specific recommendation for something I haven't wanted to use. I don't think the whole purpose of CI/CD is that important because I don't think we should be expecting non-developers to be using a continuous rolling main branch rather than discrete releases the maintainers have confidence in. If other people want to do that, fine, but finding the right tooling to do it without externalities impacting others is on them not me.
@dalias @valorzard @wwahammy I think if you have significantly varying amounts of confidence in your
mainbranch there's something wrong with your approach to development, even if non-developers only ever use releases. releases are useful to indicate evolution of the support contract, sure; but if yourmainbranch is sometimes especially wonky because you landed a poorly tested change you should probably test your changes better -
@whitequark @dalias @wwahammy LPEs are certainly an issue (although they're also an issue for any CI that doesn't use proper VMs), but Nix doesn't just allow any random unprivileged user to configure a substituter, right?
@noisytoot @dalias @wwahammy nope. but if you're actively trying to cache intermediate products, you'd have to either allow persistent writes to /nix or allow writes to substituters, both of which seem like they'd allow for cache poisoning (or at least, they don't seem robust enough that I can guarantee absence of it)
