i know some people oppose the widespread use of CI on ideological grounds, so i think it's worth it thinking about why we value it
-
@dalias @wwahammy so I've been responsible for the operation of something more structured for a few years—in my case, a complex Buildbot CI workflow that was updating and building an LLVM/Clang/ARTIQ on a 10 Mbps link (not a typo). I actually did set up the caching system you're talking about here, which used nginx in a forward proxy mode to intercept and store Conda package requests, and it was one of my most nightmarish technical assignments. if I never have to do that again in my life it will be too soon. the correct amount of state in a CI system is zero, because this actually makes it knowable, instead of a bundle of surprises you never know will work from one build to the next because of changes you couldn't predict or track
this doesn't mean that redownloading the same static files over and over is necessary, but the basic principle of "preserve nothing from run to run" is the only way to stay sane
@whitequark @dalias @wwahammy “10 Mbps link” That's a nice fast UART you've got there!
-
@whitequark @dalias @wwahammy “10 Mbps link” That's a nice fast UART you've got there!
-
@dalias see, I don't really like talking to you because of your tendency to arrogantly jump to conclusions without ever doing a bare minimum of research
@whitequark If this is a conversation you'd rather I not continue I'm fine with dropping it.
-
@dalias @wwahammy so I've been responsible for the operation of something more structured for a few years—in my case, a complex Buildbot CI workflow that was updating and building an LLVM/Clang/ARTIQ on a 10 Mbps link (not a typo). I actually did set up the caching system you're talking about here, which used nginx in a forward proxy mode to intercept and store Conda package requests, and it was one of my most nightmarish technical assignments. if I never have to do that again in my life it will be too soon. the correct amount of state in a CI system is zero, because this actually makes it knowable, instead of a bundle of surprises you never know will work from one build to the next because of changes you couldn't predict or track
this doesn't mean that redownloading the same static files over and over is necessary, but the basic principle of "preserve nothing from run to run" is the only way to stay sane
@dalias @wwahammy practically speaking, since most of the traffic is coming from npm/pip/cargo/etc I think you should be able to reduce load on external services without intercepting every network request, but by providing local on-demand caches of popular (thus, expensive to run) repositories. this is unlikely to make much of a difference because the supermajority of the load will continue to come from GitHub, but in a hypothetical world where GitHub implemented this, it would improve things a lot
of course GitHub doesn't care too much because npm traffic should be free for them and I guess they just don't think too much about the rest? gross behavior
-
@whitequark If this is a conversation you'd rather I not continue I'm fine with dropping it.
@dalias no, I would rather like to see you question your assumptions (that other people just don't know how to build software) more often. which I know is a lot more work, but still
-
@dalias no, I would rather like to see you question your assumptions (that other people just don't know how to build software) more often. which I know is a lot more work, but still
@whitequark I mean I feel like it's less of an "assumption" and more of a long history of unpleasant experiences.
-
@dalias @wwahammy practically speaking, since most of the traffic is coming from npm/pip/cargo/etc I think you should be able to reduce load on external services without intercepting every network request, but by providing local on-demand caches of popular (thus, expensive to run) repositories. this is unlikely to make much of a difference because the supermajority of the load will continue to come from GitHub, but in a hypothetical world where GitHub implemented this, it would improve things a lot
of course GitHub doesn't care too much because npm traffic should be free for them and I guess they just don't think too much about the rest? gross behavior
@dalias @wwahammy the unfortunate part about being a comparative drop in the bucket is that you could reduce your traffic by 99.9% and nobody on the other end would even notice. in general it doesn't look like a problem that will be solved unless e.g. PyPI starts responding with 429 to requests from Azure's ASN, and which will probably be solved quickly afterwards
from memory, the latest plan on this was to start charging the biggest bandwidth users, but I'm not sure where that's at. maybe @glyph knows?
-
@MrDOS @dalias @wwahammy I don't think I have words to adequately describe waiting for Conda to download a build of LLVM you just uploaded there minutes ago... for 90 minutes... then deciding to discard everything it's done and download it again, for some inscrutable dependency solver reasons I could never nail down
I think it may have improved since but it's why I still have a visceral reaction to Conda. it's basically like this
-
@MrDOS @dalias @wwahammy I don't think I have words to adequately describe waiting for Conda to download a build of LLVM you just uploaded there minutes ago... for 90 minutes... then deciding to discard everything it's done and download it again, for some inscrutable dependency solver reasons I could never nail down
I think it may have improved since but it's why I still have a visceral reaction to Conda. it's basically like this
@whitequark @MrDOS @wwahammy A similar visceral reaction is probably a large part of my rage at this kind of stuff.
-
@MrDOS @dalias @wwahammy I don't think I have words to adequately describe waiting for Conda to download a build of LLVM you just uploaded there minutes ago... for 90 minutes... then deciding to discard everything it's done and download it again, for some inscrutable dependency solver reasons I could never nail down
I think it may have improved since but it's why I still have a visceral reaction to Conda. it's basically like this
-
@dalias @wwahammy the unfortunate part about being a comparative drop in the bucket is that you could reduce your traffic by 99.9% and nobody on the other end would even notice. in general it doesn't look like a problem that will be solved unless e.g. PyPI starts responding with 429 to requests from Azure's ASN, and which will probably be solved quickly afterwards
from memory, the latest plan on this was to start charging the biggest bandwidth users, but I'm not sure where that's at. maybe @glyph knows?
@whitequark @dalias @glyph it really seems like it comes down to "GitHub doesn't want to fix it"
-
@whitequark @wwahammy TBH if you can't trust your incremental builds to be incremental, that's something I'd want a good CI to test too. 🤪
Like, both preserving artifacts from parent commit, *and* running a new build from scratch, and asserting that the results are byte-for-byte identical.
No, that doesn't sound fun to implement.
@dalias @whitequark @wwahammy I hope I didn't lose the right end of this thread with so many side replies it needs its own representational format. Only meta communication from my side:
Thanks for the civil discussion, a rarity in public Internet when different opinions clash. I know there's a lot of own experiences, assumptions, and opinions, some of them in comparable direction while others being diametrical.
Anyway, I enjoyed this discussion wholeheartedly. Not because it could also be a panel discussion but rather because they highlight reasons for actions being taken (with different findings but that is okay on my page)
I'd like to see more of this in the future
-
@whitequark @wwahammy I think I'm one of them so I'll go over a few:
1. Resource usage externalities when this is done at scale, especially for large projects times large numbers of PR authors. This manifests as energy waste, hammering the servers/infrastructure of software you depend on and pull dynamically in standard "destroy the world and re-run everything from scratch" CI recipes, etc.
2. Dependency on subsidized compute resources from a capitalist platform with motivation to lock you in and enshittify.
3. Reducing or eliminating the mandate for your software to be independently buildable by people on their own systems without your CI infrastructure.
@dalias @whitequark @wwahammy these can be solved by hosting your own GitLab, Forgejo, or Gitea instance, using an artifact storage (either built-in or something like Nexus) and not overcomplicating your CI setup (e.g. just calling the script/build system/test rather than having entire scripts in the CI)
-
R relay@relay.infosec.exchange shared this topic
