2️⃣ Here's the 2nd post highlighting key new features of the upcoming v261 release of systemd.
-
2️⃣ Here's the 2nd post highlighting key new features of the upcoming v261 release of systemd. #systemd261 #systemd
When you care for a more than a single system, it is often advisable to roll out new stuff in a staged fashion: first 1% of your fleet should get it, and only after the results are in and all is good, you up it to 10%, and then to 25% and finally to 100% (or in other steps like this).
With v261, systemd is going to help you with this to some degree:
-
2️⃣ Here's the 2nd post highlighting key new features of the upcoming v261 release of systemd. #systemd261 #systemd
When you care for a more than a single system, it is often advisable to roll out new stuff in a staged fashion: first 1% of your fleet should get it, and only after the results are in and all is good, you up it to 10%, and then to 25% and finally to 100% (or in other steps like this).
With v261, systemd is going to help you with this to some degree:
There's a new setting ConditionFraction= which allows to condition a unit to only run on a certain, randomly selected (but deterministic) subset of your fleet.
You can use it like this:
[Unit]
…
ConditionFraction=15%
… -
There's a new setting ConditionFraction= which allows to condition a unit to only run on a certain, randomly selected (but deterministic) subset of your fleet.
You can use it like this:
[Unit]
…
ConditionFraction=15%
…With that simple setting you just declared that the service shall be run on only ~15% of systems and be skipped on ~85% of them.
How does this work?
Basically, we derive an 32bit integer value from the machine ID (i.e. /etc/machine-id) by hashing, and if that integer is below the specified percentage of 2^32, the condition holds, and otherwise it doesn't.
You can also invert the check, so that you can condition some stuff to run on the in-group, and something else on the out-group.
-
With that simple setting you just declared that the service shall be run on only ~15% of systems and be skipped on ~85% of them.
How does this work?
Basically, we derive an 32bit integer value from the machine ID (i.e. /etc/machine-id) by hashing, and if that integer is below the specified percentage of 2^32, the condition holds, and otherwise it doesn't.
You can also invert the check, so that you can condition some stuff to run on the in-group, and something else on the out-group.
@pid_eins
This will be an essential feature for Works On My Machine certification! -
With that simple setting you just declared that the service shall be run on only ~15% of systems and be skipped on ~85% of them.
How does this work?
Basically, we derive an 32bit integer value from the machine ID (i.e. /etc/machine-id) by hashing, and if that integer is below the specified percentage of 2^32, the condition holds, and otherwise it doesn't.
You can also invert the check, so that you can condition some stuff to run on the in-group, and something else on the out-group.
As defined above the only input for the hash function is the machine ID, which would mean that for any unit you condition like this, it would be the very same subset of systems that would run it, and the same subset that wouldn't, regardless of the unit.
This typically is not what you want. Hence, you can also specify "ConditionFraction=foo 15%" where "foo" is an arbitrary string called the "tag". When specified it's mixed into the hash, and thus selects a random, yet deterministic subset…
-
As defined above the only input for the hash function is the machine ID, which would mean that for any unit you condition like this, it would be the very same subset of systems that would run it, and the same subset that wouldn't, regardless of the unit.
This typically is not what you want. Hence, you can also specify "ConditionFraction=foo 15%" where "foo" is an arbitrary string called the "tag". When specified it's mixed into the hash, and thus selects a random, yet deterministic subset…
…of systems that are selected. If multiple units use the same tag they'd be conditioned on the same subset of systems, and if they use a different tag the subset will be selected differently.
-
There's a new setting ConditionFraction= which allows to condition a unit to only run on a certain, randomly selected (but deterministic) subset of your fleet.
You can use it like this:
[Unit]
…
ConditionFraction=15%
…@pid_eins This is a very neat idea but I don't think it belongs in systemd. Tools like https://github.com/purpleidea/mgmt/ are where this belongs.
This design implies a static representation of the world, when in reality we might care about doing this in 15% of machines that are in a certain zone or with a certain version or something else.
So you need automation tools to deploy to that subset anyways, so the same tools can choose which % get the workload.
-
…of systems that are selected. If multiple units use the same tag they'd be conditioned on the same subset of systems, and if they use a different tag the subset will be selected differently.
@pid_eins neat feature. thank you for your work, sir
-
@pid_eins This is a very neat idea but I don't think it belongs in systemd. Tools like https://github.com/purpleidea/mgmt/ are where this belongs.
This design implies a static representation of the world, when in reality we might care about doing this in 15% of machines that are in a certain zone or with a certain version or something else.
So you need automation tools to deploy to that subset anyways, so the same tools can choose which % get the workload.
@purpleidea Yeah, well, that's just like, your opinion, man.
You can have a multitude of conditions in the same unit file, to further restrict things. And there's another condition type we added in this release that might fit will into the concept you describe, but that's for later.
-
@purpleidea Yeah, well, that's just like, your opinion, man.
You can have a multitude of conditions in the same unit file, to further restrict things. And there's another condition type we added in this release that might fit will into the concept you describe, but that's for later.
@pid_eins Well put it this way, what tooling would deploy the `ConditionFraction=` line?
Why wouldn't that tooling have the ability to choose which hosts?
I don't mind having this feature in, but I'm struggling to see the value. Like it's great to have a ladder, but if everything is already at eye level, how useful is it really?
-
@pid_eins Well put it this way, what tooling would deploy the `ConditionFraction=` line?
Why wouldn't that tooling have the ability to choose which hosts?
I don't mind having this feature in, but I'm struggling to see the value. Like it's great to have a ladder, but if everything is already at eye level, how useful is it really?
@pid_eins As an aside, I love the machine-id deterministic hashing... I think we should add the machine-id as a built in value users can query, and copy the hash function to be able to model the same distribution! I will eventually UTSL unless you have an API we can query directly.
-
2️⃣ Here's the 2nd post highlighting key new features of the upcoming v261 release of systemd. #systemd261 #systemd
When you care for a more than a single system, it is often advisable to roll out new stuff in a staged fashion: first 1% of your fleet should get it, and only after the results are in and all is good, you up it to 10%, and then to 25% and finally to 100% (or in other steps like this).
With v261, systemd is going to help you with this to some degree:
@pid_eins very nice! I assume there's a way to determine via DBus or Varlink if the randomized condition was true or not on a given unit file so that rollout agents can retrieve it?
-
@jamesh @purpleidea it's SHA256, not SHA1. It's 2026 after all
-
@pid_eins As an aside, I love the machine-id deterministic hashing... I think we should add the machine-id as a built in value users can query, and copy the hash function to be able to model the same distribution! I will eventually UTSL unless you have an API we can query directly.
@purpleidea we have no API for this, but it's trivial to reimplement. And this stuff is supposed to be deterministic and reproducible, hence I guess we really should try to keep it stable so that it doesn't lose it's reliable properties if you mix different systemd versions in the same datacenter. Hence I guess you can consider the algorithm the stable API here.
-
@pid_eins Well put it this way, what tooling would deploy the `ConditionFraction=` line?
Why wouldn't that tooling have the ability to choose which hosts?
I don't mind having this feature in, but I'm struggling to see the value. Like it's great to have a ladder, but if everything is already at eye level, how useful is it really?
@purpleidea well, people can push out unit files with any tool they like, it's really up to users.
The great thing about this is that it just works, i.e that there's no scalability problem. If you centrally determine a list of hosts to run something on, then you always have that: the algorithm needs to pick hosts, and compile things in a massive list, which could be quite large for a datacenter.
Also, you are assuming that there even *is* a central node that could do such scheduling. But…
-
@purpleidea well, people can push out unit files with any tool they like, it's really up to users.
The great thing about this is that it just works, i.e that there's no scalability problem. If you centrally determine a list of hosts to run something on, then you always have that: the algorithm needs to pick hosts, and compile things in a massive list, which could be quite large for a datacenter.
Also, you are assuming that there even *is* a central node that could do such scheduling. But…
@purpleidea …that's not a given: consider an image based desktop OS with a strong feedback path, which wants to deploy some tooling/metrics collection in a staged fashion. They typically don't have a list of their installations, but they still want to deploy piecemeal.
Hence, I am pretty sure this is very useful, and in many way systematically better than a centrally scheduled approach.
Also, systemd already had ConditionHost= (which can condition on /etc/machine-id) before. This feature…
-
@purpleidea …that's not a given: consider an image based desktop OS with a strong feedback path, which wants to deploy some tooling/metrics collection in a staged fashion. They typically don't have a list of their installations, but they still want to deploy piecemeal.
Hence, I am pretty sure this is very useful, and in many way systematically better than a centrally scheduled approach.
Also, systemd already had ConditionHost= (which can condition on /etc/machine-id) before. This feature…
@purpleidea … is now just adds a tiny bit of code on top which instead of doing memcmp() of the machine-id now calculates `HMAC(machine-id, tag) <= X`. Which is frankly a triviality.
-
@pid_eins very nice! I assume there's a way to determine via DBus or Varlink if the randomized condition was true or not on a given unit file so that rollout agents can retrieve it?
@raito yeah, you can query the result of conditions via D-Bus and "systemctl status"
-
@pid_eins Well put it this way, what tooling would deploy the `ConditionFraction=` line?
Why wouldn't that tooling have the ability to choose which hosts?
I don't mind having this feature in, but I'm struggling to see the value. Like it's great to have a ladder, but if everything is already at eye level, how useful is it really?
@purpleidea @pid_eins I think there’s room for both #configmanagememt driven sharding of populations and autonomous local stable sharding. The latter is useful when you want the local state to be persistent and stable even when #configmanagememt stops working or is not used in the first place.
-
R relay@relay.infosec.exchange shared this topic
