1/ There has never been a more concentrated distillation of my teaching than this lesson: Algos, Bias, Due Process, & You.
-
9/ You’ll notice mention of a “Rival Clerk,” along with a reminder that we’re measuring the user’s speed et al. Dear reader, the “Rival Clerk” is not one of their peers. It’s a dark pattern⁶ designed to make them keep going. There’s so much in here ripe for discussion.
⁶ https://en.wikipedia.org/wiki/Dark_pattern
10/ When they finished, users were shown a results screen that explained a bit more about the exercise. There were three possible outcomes: (1) No clear evidence of automation bias (2) You may have fallen victim to automation bias; and (3) You likely fell victim to automation bias.
-
10/ When they finished, users were shown a results screen that explained a bit more about the exercise. There were three possible outcomes: (1) No clear evidence of automation bias (2) You may have fallen victim to automation bias; and (3) You likely fell victim to automation bias.
11/ Almost everyone fell victim to automation bias. The assistant's accuracy was 100% in phase 1 & 2, then dropped to 70%. Student performance started at 79% in phase 1, improved to 85% for a bit, but when the tool's accuracy declined, scores fell to 65%, worse than their initial performance.
-
11/ Almost everyone fell victim to automation bias. The assistant's accuracy was 100% in phase 1 & 2, then dropped to 70%. Student performance started at 79% in phase 1, improved to 85% for a bit, but when the tool's accuracy declined, scores fell to 65%, worse than their initial performance.
12/ Perhaps more telling is how often they relied on the assistant’s recommendation without consulting additional info (summary, authority, or excerpt). In phase 1, they avoided additional info 65% of the time. In phase 2, this went up to 80%, and in phase 3 it jumped to 84%.
-
12/ Perhaps more telling is how often they relied on the assistant’s recommendation without consulting additional info (summary, authority, or excerpt). In phase 1, they avoided additional info 65% of the time. In phase 2, this went up to 80%, and in phase 3 it jumped to 84%.
13/ We talked about what happened, and I hope the lesson sticks with them. Admittedly, the exercise was designed to push them to this result, but hopefully by giving it a name and being forced to face the reality that it can happen to them, this is a concern they will carry with them into practice.
-
13/ We talked about what happened, and I hope the lesson sticks with them. Admittedly, the exercise was designed to push them to this result, but hopefully by giving it a name and being forced to face the reality that it can happen to them, this is a concern they will carry with them into practice.
14/ Since we had just made use of a tool that purported to make predictions with some level of confidence, I suggested we might want to look more into what such tools are really telling us. So, I asked them the following.
-
14/ Since we had just made use of a tool that purported to make predictions with some level of confidence, I suggested we might want to look more into what such tools are really telling us. So, I asked them the following.
15/ Most ppl thought the answer was B. I suggested they think about that some more, divided them into groups of ~3, and asked that each group explore the following simulation together, after which we would talk. https://bail-risk-simulator-50382557550.us-west1.run.app/
-
15/ Most ppl thought the answer was B. I suggested they think about that some more, divided them into groups of ~3, and asked that each group explore the following simulation together, after which we would talk. https://bail-risk-simulator-50382557550.us-west1.run.app/
16/ TL;DR: high-performing tests can be wrong about most of their positive predictions if the thing they're trying to predict is rare. Context matters!! We reran the above poll, and thankfully most folks changed their answer to D (I don't know). Always consider the base rate.
-
16/ TL;DR: high-performing tests can be wrong about most of their positive predictions if the thing they're trying to predict is rare. Context matters!! We reran the above poll, and thankfully most folks changed their answer to D (I don't know). Always consider the base rate.
17/ The next sim generated some great conversations & helped students confront something that doesn't get said enough. There can be competing and mutually exclusive concepts of fairness, and the policies that seek to deliver on one measure of fairness might have to change when the context changes.
-
17/ The next sim generated some great conversations & helped students confront something that doesn't get said enough. There can be competing and mutually exclusive concepts of fairness, and the policies that seek to deliver on one measure of fairness might have to change when the context changes.
18/ It (https://fairness-simulator-the-toilet-seat-dilemma-50382557550.us-west1.run.app/) lets you simulate what happens when folks following different rules share a toilet. It assumes 2 populations, "sitters" & "standers" (folks who sometimes stand). It lets you see how different behavior effects 2 costs:
(1) the cost of having to change the seat's position before you use the toilet; and
(2) the cost of having to clean the seat if the last person failed to raise the seat when really they should have.
-
18/ It (https://fairness-simulator-the-toilet-seat-dilemma-50382557550.us-west1.run.app/) lets you simulate what happens when folks following different rules share a toilet. It assumes 2 populations, "sitters" & "standers" (folks who sometimes stand). It lets you see how different behavior effects 2 costs:
(1) the cost of having to change the seat's position before you use the toilet; and
(2) the cost of having to clean the seat if the last person failed to raise the seat when really they should have.
19/ This means you have to assign a relative value to these costs and make assumptions about how frequent certain behaviors are among your groups. After you've dialed these in, however, you can simulate the outcome for 100 users at a time to see what happens.
-
19/ This means you have to assign a relative value to these costs and make assumptions about how frequent certain behaviors are among your groups. After you've dialed these in, however, you can simulate the outcome for 100 users at a time to see what happens.
20/ There's a large universe of possible outcomes. If you're interested in what our groups found, here's a deeplink to my discussion of our debrief. TL;DR: There can be conflicting concepts of fairness, and the policies that deliver on one measure might have to change when context changes. https://suffolklitlab.org/algos-bias-due-process-you/#is-it-fair
-
20/ There's a large universe of possible outcomes. If you're interested in what our groups found, here's a deeplink to my discussion of our debrief. TL;DR: There can be conflicting concepts of fairness, and the policies that deliver on one measure might have to change when context changes. https://suffolklitlab.org/algos-bias-due-process-you/#is-it-fair
21/ Only now did I introduce the reporting on machine bias, sketching out the broad strokes. I focused on the fact that these tools can make different predictions for different populations based on their training data. Finally, I presented them with their role play.
-
21/ Only now did I introduce the reporting on machine bias, sketching out the broad strokes. I focused on the fact that these tools can make different predictions for different populations based on their training data. Finally, I presented them with their role play.
22/ Here's the simulation they were asked to explore while considering the above. https://facial-recognition-bias-sim-50382557550.us-west1.run.app/
-
22/ Here's the simulation they were asked to explore while considering the above. https://facial-recognition-bias-sim-50382557550.us-west1.run.app/
23/ Some folks suggested requiring equal treatment of populations before they would consider using the tech; others, setting high thresholds. We talked about requiring warrants before running a check, & I shared how MA has attempted to address these issues. https://www.nytimes.com/2021/02/27/technology/Massachusetts-facial-recognition-rules.html
-
23/ Some folks suggested requiring equal treatment of populations before they would consider using the tech; others, setting high thresholds. We talked about requiring warrants before running a check, & I shared how MA has attempted to address these issues. https://www.nytimes.com/2021/02/27/technology/Massachusetts-facial-recognition-rules.html
24/ Then I told them that there actually was a federal law enforcement agency actively using facial recognition out in the real world called ICE, and I asked what safeguards folks thought they had in place… Things got a bit quiet, and I shared the following reporting. https://www.404media.co/ices-facial-recognition-app-misidentified-a-woman-twice/
-
24/ Then I told them that there actually was a federal law enforcement agency actively using facial recognition out in the real world called ICE, and I asked what safeguards folks thought they had in place… Things got a bit quiet, and I shared the following reporting. https://www.404media.co/ices-facial-recognition-app-misidentified-a-woman-twice/
25/ It was only later in the week, after I taught the class, that a few Senators introduced the ICE Out of Our Faces Act. https://arstechnica.com/tech-policy/2026/02/ice-out-of-our-faces-act-would-ban-ice-and-cbp-use-of-facial-recognition/
-
25/ It was only later in the week, after I taught the class, that a few Senators introduced the ICE Out of Our Faces Act. https://arstechnica.com/tech-policy/2026/02/ice-out-of-our-faces-act-would-ban-ice-and-cbp-use-of-facial-recognition/
26/ I then asked if folks were familiar with the two incidents reported here:
ICE Arrest of a Citizen, Barely Dressed, Sows Fear in Twin Cities https://www.nytimes.com/2026/01/20/us/chongly-scott-thao-ice-arrest.html?smid=bs-share
and
ICE detains five-year-old Minnesota boy arriving home, say school officials https://www.theguardian.com/us-news/2026/jan/21/ice-arrests-five-year-old-boy-minnesota
-
26/ I then asked if folks were familiar with the two incidents reported here:
ICE Arrest of a Citizen, Barely Dressed, Sows Fear in Twin Cities https://www.nytimes.com/2026/01/20/us/chongly-scott-thao-ice-arrest.html?smid=bs-share
and
ICE detains five-year-old Minnesota boy arriving home, say school officials https://www.theguardian.com/us-news/2026/jan/21/ice-arrests-five-year-old-boy-minnesota
27/ They were. I speculated that the "targeted operations" cited were possibly driven by ‘ELITE’: The Palantir App ICE Uses to Find Neighborhoods to Raid⁷ and presented the following questions for them to consider in their groups.
⁷ https://www.404media.co/elite-the-palantir-app-ice-uses-to-find-neighborhoods-to-raid/
-
27/ They were. I speculated that the "targeted operations" cited were possibly driven by ‘ELITE’: The Palantir App ICE Uses to Find Neighborhoods to Raid⁷ and presented the following questions for them to consider in their groups.
⁷ https://www.404media.co/elite-the-palantir-app-ice-uses-to-find-neighborhoods-to-raid/
28/ They were provided w/ the following simulation comparing due process to medical screening & diagnostic tests. It lets you explore how changes to thresholds at different points in the process effect different measures of cost. https://screening-vs-diagnostic-tests-50382557550.us-west1.run.app/
-
28/ They were provided w/ the following simulation comparing due process to medical screening & diagnostic tests. It lets you explore how changes to thresholds at different points in the process effect different measures of cost. https://screening-vs-diagnostic-tests-50382557550.us-west1.run.app/
29/ As they clicked away at the simulation, I reminded them of Blackstone's ratio.
