i know some people oppose the widespread use of CI on ideological grounds, so i think it's worth it thinking about why we value it
-
@noisytoot @dalias @wwahammy I was thinking about "substituters". as far as I'm aware nothing stops you from editing the stuff in the Nix store if you have the right privileges (directly or via a service) and it's pretty hard to detect if it's ever done, therefore I wouldn't rely just on Nix to prevent cache poisoning (especially in light of regularly dropping Linux LPEs)
@whitequark @dalias @wwahammy LPEs are certainly an issue (although they're also an issue for any CI that doesn't use proper VMs), but Nix doesn't just allow any random unprivileged user to configure a substituter, right? -
@dalias @wwahammy most of the time? because it's a submodule. sometimes a recursive submodule.
github's default actions/checkout does a shallow clone (which is just as efficient), but some packages do actually look at their own history in order to give accurate git-describe results or turn git distance numbers into version numbers. your workflow isn't my workflow
@whitequark @wwahammy OK, but that's the fault of the CI system doing a shallow clone rather than a fully recursive checkout from already-cloned-and-cached repositories. It's the fault of poor abstraction layers that behave as "just do whatever you want to script in this throwaway container" rather than something more structured.
-
@dalias @wwahammy also I'm pretty sure that at least with Forgejo, it takes less resources to do a git shallow clone than it takes to download an archive of a commit (because the archive needs to be generated and then stored, and all of them are fully denormalized, while git does some sort of optimization with pack files I think?)
@whitequark @wwahammy I don't see why the archive would need to be stored. Tarballs are fully streamable and the git-archive command emits them as a stream not with temporary storage.
-
@valorzard @whitequark @wwahammy Well I'm against a number of standard CI/CD practices that are harmful to parties not even involved in the project using the CI/CD.
I don't have a specific recommendation for something I haven't wanted to use. I don't think the whole purpose of CI/CD is that important because I don't think we should be expecting non-developers to be using a continuous rolling main branch rather than discrete releases the maintainers have confidence in. If other people want to do that, fine, but finding the right tooling to do it without externalities impacting others is on them not me.
@dalias @valorzard @wwahammy I think if you have significantly varying amounts of confidence in your
mainbranch there's something wrong with your approach to development, even if non-developers only ever use releases. releases are useful to indicate evolution of the support contract, sure; but if yourmainbranch is sometimes especially wonky because you landed a poorly tested change you should probably test your changes better -
@whitequark @dalias @wwahammy LPEs are certainly an issue (although they're also an issue for any CI that doesn't use proper VMs), but Nix doesn't just allow any random unprivileged user to configure a substituter, right?
@noisytoot @dalias @wwahammy nope. but if you're actively trying to cache intermediate products, you'd have to either allow persistent writes to /nix or allow writes to substituters, both of which seem like they'd allow for cache poisoning (or at least, they don't seem robust enough that I can guarantee absence of it)
-
@whitequark @wwahammy I don't see why the archive would need to be stored. Tarballs are fully streamable and the git-archive command emits them as a stream not with temporary storage.
-
@whitequark @wwahammy Gotta love how much of a regression all the fancy forges are versus plain cgi-bin cgit...
-
@whitequark @wwahammy Gotta love how much of a regression all the fancy forges are versus plain cgi-bin cgit...
@dalias @whitequark @wwahammy Do you include SourceHut in that analysis? In some ways it's even more minimalist than cgit.
-
@whitequark @wwahammy OK, but that's the fault of the CI system doing a shallow clone rather than a fully recursive checkout from already-cloned-and-cached repositories. It's the fault of poor abstraction layers that behave as "just do whatever you want to script in this throwaway container" rather than something more structured.
@dalias @wwahammy so I've been responsible for the operation of something more structured for a few years—in my case, a complex Buildbot CI workflow that was updating and building an LLVM/Clang/ARTIQ on a 10 Mbps link (not a typo). I actually did set up the caching system you're talking about here, which used nginx in a forward proxy mode to intercept and store Conda package requests, and it was one of my most nightmarish technical assignments. if I never have to do that again in my life it will be too soon. the correct amount of state in a CI system is zero, because this actually makes it knowable, instead of a bundle of surprises you never know will work from one build to the next because of changes you couldn't predict or track
this doesn't mean that redownloading the same static files over and over is necessary, but the basic principle of "preserve nothing from run to run" is the only way to stay sane
-
@whitequark @wwahammy Gotta love how much of a regression all the fancy forges are versus plain cgi-bin cgit...
@dalias see, I don't really like talking to you because of your tendency to arrogantly jump to conclusions without ever doing a bare minimum of research
-
@dalias see, I don't really like talking to you because of your tendency to arrogantly jump to conclusions without ever doing a bare minimum of research
@dalias not "Huh, I wonder why is it that Forgejo does that?" (I don't know but I suspect it has something to do with IO load from repeatedly requested archives), directly to "It's a regression compared to [favorite project]!". it's insufferable
-
@dalias @wwahammy so I've been responsible for the operation of something more structured for a few years—in my case, a complex Buildbot CI workflow that was updating and building an LLVM/Clang/ARTIQ on a 10 Mbps link (not a typo). I actually did set up the caching system you're talking about here, which used nginx in a forward proxy mode to intercept and store Conda package requests, and it was one of my most nightmarish technical assignments. if I never have to do that again in my life it will be too soon. the correct amount of state in a CI system is zero, because this actually makes it knowable, instead of a bundle of surprises you never know will work from one build to the next because of changes you couldn't predict or track
this doesn't mean that redownloading the same static files over and over is necessary, but the basic principle of "preserve nothing from run to run" is the only way to stay sane
@whitequark @wwahammy TBH if you can't trust your incremental builds to be incremental, that's something I'd want a good CI to test too. 🤪
Like, both preserving artifacts from parent commit, *and* running a new build from scratch, and asserting that the results are byte-for-byte identical.
No, that doesn't sound fun to implement.
-
@dalias @wwahammy so I've been responsible for the operation of something more structured for a few years—in my case, a complex Buildbot CI workflow that was updating and building an LLVM/Clang/ARTIQ on a 10 Mbps link (not a typo). I actually did set up the caching system you're talking about here, which used nginx in a forward proxy mode to intercept and store Conda package requests, and it was one of my most nightmarish technical assignments. if I never have to do that again in my life it will be too soon. the correct amount of state in a CI system is zero, because this actually makes it knowable, instead of a bundle of surprises you never know will work from one build to the next because of changes you couldn't predict or track
this doesn't mean that redownloading the same static files over and over is necessary, but the basic principle of "preserve nothing from run to run" is the only way to stay sane
@whitequark @dalias @wwahammy “10 Mbps link” That's a nice fast UART you've got there!
-
@whitequark @dalias @wwahammy “10 Mbps link” That's a nice fast UART you've got there!
-
@dalias see, I don't really like talking to you because of your tendency to arrogantly jump to conclusions without ever doing a bare minimum of research
@whitequark If this is a conversation you'd rather I not continue I'm fine with dropping it.
-
@dalias @wwahammy so I've been responsible for the operation of something more structured for a few years—in my case, a complex Buildbot CI workflow that was updating and building an LLVM/Clang/ARTIQ on a 10 Mbps link (not a typo). I actually did set up the caching system you're talking about here, which used nginx in a forward proxy mode to intercept and store Conda package requests, and it was one of my most nightmarish technical assignments. if I never have to do that again in my life it will be too soon. the correct amount of state in a CI system is zero, because this actually makes it knowable, instead of a bundle of surprises you never know will work from one build to the next because of changes you couldn't predict or track
this doesn't mean that redownloading the same static files over and over is necessary, but the basic principle of "preserve nothing from run to run" is the only way to stay sane
@dalias @wwahammy practically speaking, since most of the traffic is coming from npm/pip/cargo/etc I think you should be able to reduce load on external services without intercepting every network request, but by providing local on-demand caches of popular (thus, expensive to run) repositories. this is unlikely to make much of a difference because the supermajority of the load will continue to come from GitHub, but in a hypothetical world where GitHub implemented this, it would improve things a lot
of course GitHub doesn't care too much because npm traffic should be free for them and I guess they just don't think too much about the rest? gross behavior
-
@whitequark If this is a conversation you'd rather I not continue I'm fine with dropping it.
@dalias no, I would rather like to see you question your assumptions (that other people just don't know how to build software) more often. which I know is a lot more work, but still
-
@dalias no, I would rather like to see you question your assumptions (that other people just don't know how to build software) more often. which I know is a lot more work, but still
@whitequark I mean I feel like it's less of an "assumption" and more of a long history of unpleasant experiences.
-
@dalias @wwahammy practically speaking, since most of the traffic is coming from npm/pip/cargo/etc I think you should be able to reduce load on external services without intercepting every network request, but by providing local on-demand caches of popular (thus, expensive to run) repositories. this is unlikely to make much of a difference because the supermajority of the load will continue to come from GitHub, but in a hypothetical world where GitHub implemented this, it would improve things a lot
of course GitHub doesn't care too much because npm traffic should be free for them and I guess they just don't think too much about the rest? gross behavior
@dalias @wwahammy the unfortunate part about being a comparative drop in the bucket is that you could reduce your traffic by 99.9% and nobody on the other end would even notice. in general it doesn't look like a problem that will be solved unless e.g. PyPI starts responding with 429 to requests from Azure's ASN, and which will probably be solved quickly afterwards
from memory, the latest plan on this was to start charging the biggest bandwidth users, but I'm not sure where that's at. maybe @glyph knows?
-
@MrDOS @dalias @wwahammy I don't think I have words to adequately describe waiting for Conda to download a build of LLVM you just uploaded there minutes ago... for 90 minutes... then deciding to discard everything it's done and download it again, for some inscrutable dependency solver reasons I could never nail down
I think it may have improved since but it's why I still have a visceral reaction to Conda. it's basically like this
