1/ There has never been a more concentrated distillation of my teaching than this lesson: Algos, Bias, Due Process, & You.
-
13/ We talked about what happened, and I hope the lesson sticks with them. Admittedly, the exercise was designed to push them to this result, but hopefully by giving it a name and being forced to face the reality that it can happen to them, this is a concern they will carry with them into practice.
14/ Since we had just made use of a tool that purported to make predictions with some level of confidence, I suggested we might want to look more into what such tools are really telling us. So, I asked them the following.
-
14/ Since we had just made use of a tool that purported to make predictions with some level of confidence, I suggested we might want to look more into what such tools are really telling us. So, I asked them the following.
15/ Most ppl thought the answer was B. I suggested they think about that some more, divided them into groups of ~3, and asked that each group explore the following simulation together, after which we would talk. https://bail-risk-simulator-50382557550.us-west1.run.app/
-
15/ Most ppl thought the answer was B. I suggested they think about that some more, divided them into groups of ~3, and asked that each group explore the following simulation together, after which we would talk. https://bail-risk-simulator-50382557550.us-west1.run.app/
16/ TL;DR: high-performing tests can be wrong about most of their positive predictions if the thing they're trying to predict is rare. Context matters!! We reran the above poll, and thankfully most folks changed their answer to D (I don't know). Always consider the base rate.
-
16/ TL;DR: high-performing tests can be wrong about most of their positive predictions if the thing they're trying to predict is rare. Context matters!! We reran the above poll, and thankfully most folks changed their answer to D (I don't know). Always consider the base rate.
17/ The next sim generated some great conversations & helped students confront something that doesn't get said enough. There can be competing and mutually exclusive concepts of fairness, and the policies that seek to deliver on one measure of fairness might have to change when the context changes.
-
17/ The next sim generated some great conversations & helped students confront something that doesn't get said enough. There can be competing and mutually exclusive concepts of fairness, and the policies that seek to deliver on one measure of fairness might have to change when the context changes.
18/ It (https://fairness-simulator-the-toilet-seat-dilemma-50382557550.us-west1.run.app/) lets you simulate what happens when folks following different rules share a toilet. It assumes 2 populations, "sitters" & "standers" (folks who sometimes stand). It lets you see how different behavior effects 2 costs:
(1) the cost of having to change the seat's position before you use the toilet; and
(2) the cost of having to clean the seat if the last person failed to raise the seat when really they should have.
-
18/ It (https://fairness-simulator-the-toilet-seat-dilemma-50382557550.us-west1.run.app/) lets you simulate what happens when folks following different rules share a toilet. It assumes 2 populations, "sitters" & "standers" (folks who sometimes stand). It lets you see how different behavior effects 2 costs:
(1) the cost of having to change the seat's position before you use the toilet; and
(2) the cost of having to clean the seat if the last person failed to raise the seat when really they should have.
19/ This means you have to assign a relative value to these costs and make assumptions about how frequent certain behaviors are among your groups. After you've dialed these in, however, you can simulate the outcome for 100 users at a time to see what happens.
-
19/ This means you have to assign a relative value to these costs and make assumptions about how frequent certain behaviors are among your groups. After you've dialed these in, however, you can simulate the outcome for 100 users at a time to see what happens.
20/ There's a large universe of possible outcomes. If you're interested in what our groups found, here's a deeplink to my discussion of our debrief. TL;DR: There can be conflicting concepts of fairness, and the policies that deliver on one measure might have to change when context changes. https://suffolklitlab.org/algos-bias-due-process-you/#is-it-fair
-
20/ There's a large universe of possible outcomes. If you're interested in what our groups found, here's a deeplink to my discussion of our debrief. TL;DR: There can be conflicting concepts of fairness, and the policies that deliver on one measure might have to change when context changes. https://suffolklitlab.org/algos-bias-due-process-you/#is-it-fair
21/ Only now did I introduce the reporting on machine bias, sketching out the broad strokes. I focused on the fact that these tools can make different predictions for different populations based on their training data. Finally, I presented them with their role play.
-
21/ Only now did I introduce the reporting on machine bias, sketching out the broad strokes. I focused on the fact that these tools can make different predictions for different populations based on their training data. Finally, I presented them with their role play.
22/ Here's the simulation they were asked to explore while considering the above. https://facial-recognition-bias-sim-50382557550.us-west1.run.app/
-
22/ Here's the simulation they were asked to explore while considering the above. https://facial-recognition-bias-sim-50382557550.us-west1.run.app/
23/ Some folks suggested requiring equal treatment of populations before they would consider using the tech; others, setting high thresholds. We talked about requiring warrants before running a check, & I shared how MA has attempted to address these issues. https://www.nytimes.com/2021/02/27/technology/Massachusetts-facial-recognition-rules.html
-
23/ Some folks suggested requiring equal treatment of populations before they would consider using the tech; others, setting high thresholds. We talked about requiring warrants before running a check, & I shared how MA has attempted to address these issues. https://www.nytimes.com/2021/02/27/technology/Massachusetts-facial-recognition-rules.html
24/ Then I told them that there actually was a federal law enforcement agency actively using facial recognition out in the real world called ICE, and I asked what safeguards folks thought they had in place… Things got a bit quiet, and I shared the following reporting. https://www.404media.co/ices-facial-recognition-app-misidentified-a-woman-twice/
-
24/ Then I told them that there actually was a federal law enforcement agency actively using facial recognition out in the real world called ICE, and I asked what safeguards folks thought they had in place… Things got a bit quiet, and I shared the following reporting. https://www.404media.co/ices-facial-recognition-app-misidentified-a-woman-twice/
25/ It was only later in the week, after I taught the class, that a few Senators introduced the ICE Out of Our Faces Act. https://arstechnica.com/tech-policy/2026/02/ice-out-of-our-faces-act-would-ban-ice-and-cbp-use-of-facial-recognition/
-
25/ It was only later in the week, after I taught the class, that a few Senators introduced the ICE Out of Our Faces Act. https://arstechnica.com/tech-policy/2026/02/ice-out-of-our-faces-act-would-ban-ice-and-cbp-use-of-facial-recognition/
26/ I then asked if folks were familiar with the two incidents reported here:
ICE Arrest of a Citizen, Barely Dressed, Sows Fear in Twin Cities https://www.nytimes.com/2026/01/20/us/chongly-scott-thao-ice-arrest.html?smid=bs-share
and
ICE detains five-year-old Minnesota boy arriving home, say school officials https://www.theguardian.com/us-news/2026/jan/21/ice-arrests-five-year-old-boy-minnesota
-
26/ I then asked if folks were familiar with the two incidents reported here:
ICE Arrest of a Citizen, Barely Dressed, Sows Fear in Twin Cities https://www.nytimes.com/2026/01/20/us/chongly-scott-thao-ice-arrest.html?smid=bs-share
and
ICE detains five-year-old Minnesota boy arriving home, say school officials https://www.theguardian.com/us-news/2026/jan/21/ice-arrests-five-year-old-boy-minnesota
27/ They were. I speculated that the "targeted operations" cited were possibly driven by ‘ELITE’: The Palantir App ICE Uses to Find Neighborhoods to Raid⁷ and presented the following questions for them to consider in their groups.
⁷ https://www.404media.co/elite-the-palantir-app-ice-uses-to-find-neighborhoods-to-raid/
-
27/ They were. I speculated that the "targeted operations" cited were possibly driven by ‘ELITE’: The Palantir App ICE Uses to Find Neighborhoods to Raid⁷ and presented the following questions for them to consider in their groups.
⁷ https://www.404media.co/elite-the-palantir-app-ice-uses-to-find-neighborhoods-to-raid/
28/ They were provided w/ the following simulation comparing due process to medical screening & diagnostic tests. It lets you explore how changes to thresholds at different points in the process effect different measures of cost. https://screening-vs-diagnostic-tests-50382557550.us-west1.run.app/
-
28/ They were provided w/ the following simulation comparing due process to medical screening & diagnostic tests. It lets you explore how changes to thresholds at different points in the process effect different measures of cost. https://screening-vs-diagnostic-tests-50382557550.us-west1.run.app/
29/ As they clicked away at the simulation, I reminded them of Blackstone's ratio.
-
29/ As they clicked away at the simulation, I reminded them of Blackstone's ratio.
30/ I left them with my most charitable reading of the theory of fairness consistent with ICE actions. The last question, "how is this consistent with constitutional guarantees of due process?" was, as they say, "left as an exercise for the reader."
-
30/ I left them with my most charitable reading of the theory of fairness consistent with ICE actions. The last question, "how is this consistent with constitutional guarantees of due process?" was, as they say, "left as an exercise for the reader."
31/ As we packed up, I invited them to revisit the question that preceded the first exercise:
What makes something a good decision assistant?
I reminded them every sim was a decision assistant, told them it had been an honor to be their teacher for the day, & that I was always available to talk.
-
31/ As we packed up, I invited them to revisit the question that preceded the first exercise:
What makes something a good decision assistant?
I reminded them every sim was a decision assistant, told them it had been an honor to be their teacher for the day, & that I was always available to talk.
32/ Man, I love my job. Stay warm. Stay safe. Stay free.
-
32/ Man, I love my job. Stay warm. Stay safe. Stay free.
33/End
Thank you for making it all the way to the end. I'm pretty sure this is the longest thread I've ever written. If you came in halfway through & thought, "this should have been a blog post," I agree, and here it is: https://suffolklitlab.org/algos-bias-due-process-you/
