The root problem with a lot of Fediverse moderation is a problem that is well known the reputation-system literature:
-
The root problem with a lot of Fediverse moderation is a problem that is well known the reputation-system literature:
If the cost of creating a new identity is zero then a reputation system cannot usefully express a lower reputation than that of a new user.
A malicious actor can always create an account on a different instance, or spin up a new instance on a throw-away domain. The cost is negligible. This means that any attempt to find bad users and moderate them is doomed from the start. Unless detecting a bad user is instant, there is always a gap between a new fresh identity existing in the system and it being marked as such.
A system that expects to actually work at scale has to operate in the opposite direction: assume new users are malicious and provide a reputation system for allowing them to build trust. Unfortunately, this is in almost direct opposition to the desire to make the onboarding experience frictionless.
A model where new users are restricted from the things that make harassment easy (sending DMs, posting in other users’ threads) until they have established a reputation (other people in good standing have boosted their posts or followed them) might work.
-
R relay@relay.infosec.exchange shared this topic
-
The root problem with a lot of Fediverse moderation is a problem that is well known the reputation-system literature:
If the cost of creating a new identity is zero then a reputation system cannot usefully express a lower reputation than that of a new user.
A malicious actor can always create an account on a different instance, or spin up a new instance on a throw-away domain. The cost is negligible. This means that any attempt to find bad users and moderate them is doomed from the start. Unless detecting a bad user is instant, there is always a gap between a new fresh identity existing in the system and it being marked as such.
A system that expects to actually work at scale has to operate in the opposite direction: assume new users are malicious and provide a reputation system for allowing them to build trust. Unfortunately, this is in almost direct opposition to the desire to make the onboarding experience frictionless.
A model where new users are restricted from the things that make harassment easy (sending DMs, posting in other users’ threads) until they have established a reputation (other people in good standing have boosted their posts or followed them) might work.
@david_chisnall What bout malicious overzealous moderators?
-
The root problem with a lot of Fediverse moderation is a problem that is well known the reputation-system literature:
If the cost of creating a new identity is zero then a reputation system cannot usefully express a lower reputation than that of a new user.
A malicious actor can always create an account on a different instance, or spin up a new instance on a throw-away domain. The cost is negligible. This means that any attempt to find bad users and moderate them is doomed from the start. Unless detecting a bad user is instant, there is always a gap between a new fresh identity existing in the system and it being marked as such.
A system that expects to actually work at scale has to operate in the opposite direction: assume new users are malicious and provide a reputation system for allowing them to build trust. Unfortunately, this is in almost direct opposition to the desire to make the onboarding experience frictionless.
A model where new users are restricted from the things that make harassment easy (sending DMs, posting in other users’ threads) until they have established a reputation (other people in good standing have boosted their posts or followed them) might work.
@david_chisnall I'm seeing a lot of talk about reputation systems at the moment, applying to open source contributing and social media.
Every time, I'm reminded of how awful it was getting started on Stack Overflow.
I had an account for years before I ground through the painful process of building a reputation.
I'm not surprised that they're dying, it's not just AI; if you build walls in front of new users they'll give up and go somewhere else.
Much of my angst was that I'd put in the work elsewhere, but there was seemingly no means of transferring that reputation.
But there will always be new people trying to start from scratch, and somehow we need to welcome them whilst keeping out the abusers.
