What a lot of people do not understand is the fragility of supply chains.
-
What a lot of people do not understand is the fragility of supply chains.
Setup
Let us assume there is a part X that is used by a large number of other companies. It is not glamorous, it is not expensive, and nobody outside a small circle of specialists ever thinks about it. But it is needed. Without it, other products cannot be finished.
Everything is fine
Part X is made by a small company with a few dozen employees and a machine that is several decades old.
Everything runs smoothly. The company knows how to maintain the machine. They know how to operate it. They know its strange noises, its moods, its undocumented rituals. They know which lever needs a bit of persuasion on cold mornings and which replacement part must be machined by hand because nobody has made the original since 1987. They also know how to train new employees, because the knowledge still exists inside the company.
Crisis
Then, suddenly, a few large customers run into a crisis.
Orders slow down. The warehouses fill up with part X. Prices fall.
The company downsizes. Then it downsizes again. But nobody really notices, because stocks are still full. Customers are not yet affected. Purchasing departments can still get part X from inventory. Balance sheets still look fine. The problem has not yet reached the spreadsheet.
If this goes on for long enough, the company goes broke.
Again, nobody really notices. Stocks are still full. Some people may worry, but as long as no current production line is stopping and no quarterly number is visibly bleeding, nothing urgent is done.
The employees move on. They retire, change industries, start new careers. The old machine is sold for scrap. The building is repurposed. The knowledge evaporates.
Recovery
A few years later, demand picks up again.
The warehouses slowly empty. Stocks run low. People start ordering part X again, only to discover that nobody is offering it anymore.
Now someone decides to restart production.
The first thing they discover is that there is no machine. Building a new one would be prohibitively expensive, assuming anyone still knows how to build it properly. So they desperately search for an old one.
They get lucky. In a scrapyard, they find a machine that used to produce part X. It is rusted, incomplete, and dysfunctional. Naturally, they buy it.
Now they try to get it working again.
But there is another problem. There are no people left who know how to maintain it. So they hire someone who understands industrial maintenance in general, but has never worked on this specific machine. That person does their best. They improvise. They read old manuals. They reverse-engineer undocumented fixes. They keep the machine alive with skill, patience, and increasing amounts of despair.
But it breaks down every few hours. Output is abysmal.
Bottlenecks
And now that one poor maintenance person is overworked. They need help. But training help requires time, and the only person who can train others is the same person needed to keep the machine barely running. Every hour spent teaching is an hour not spent preventing the next failure.
Very few businesses survive this phase.
There is no institutional knowledge anymore. New people are hired, begin training, look at the state of the machine, the chaos of the process, and the constant emergency mode, and conclude that the business is doomed anyway. Then they quit.
Churn becomes terrible. Even if the company survives financially, it remains fragile. It is always just one or two people quitting away from disaster.
At the end the world decides it needs to get rid of part X as the supply is too fragile.
Summary
This is still very much simplified. The reality is more complex, more ugly.
And that is the part many people miss: a supply chain is not just warehouses, contracts, prices, and transport routes. It is also people, habits, obsolete machines, informal knowledge, and boring little skills that nobody values until they are gone.
2021 demonstrated it's not even obscure parts - and that the entire supply chain can collapse for whole industries eg electronics.
and it's not even exotic parts - the mid 2010s capacitor shortage, blamed on a factory fire, was for everyday forgettable parts that cost fractions of a cent, suddenly became. Rumor has it that the fire had nothing to do with it, but the factories just wanted to up their prices from $0.003/unit.
-
@masek and the consequences of AI. Once the knowledge is gone, it's probably gone for good or, at least, for a long time.
@jt_rebelo This is why we had to learn how to build rockets again.
-
@jt_rebelo This is why we had to learn how to build rockets again.
@Gustodon don't remind me I should buy even more physical books...
-
@Stinson_108 Iow markets are fantastic at finding an optimal (i.e. most profitable) solution unless you expose them to the real world.
@masek -
@rcosta @eckes @jt_rebelo @masek
That's a rare example of 3D printing to the rescue. Helped due to low volume.No, it's not primarily environmental. It's lack of investment in training, lack of open source documentation (or even any under NDA) and lack of second sources.
Yes, building to last, repair and reuse are important. But only tangential to supply chain risks.
@raymaccarthy @eckes @jt_rebelo @masek "lack of open source documentation", I agree and that can be fixed without a big investment is more a matter of political will. But I maintain my views to systemic causes, specially the ones related with predatory economics. Greedy CEOs scrapping machines, ending internships and factory repair workshops, etc. Lack of parts is old as industrialization itself, is inevitable. We only can delay it and choose how to deal with it when finally happens.
-
What a lot of people do not understand is the fragility of supply chains.
Setup
Let us assume there is a part X that is used by a large number of other companies. It is not glamorous, it is not expensive, and nobody outside a small circle of specialists ever thinks about it. But it is needed. Without it, other products cannot be finished.
Everything is fine
Part X is made by a small company with a few dozen employees and a machine that is several decades old.
Everything runs smoothly. The company knows how to maintain the machine. They know how to operate it. They know its strange noises, its moods, its undocumented rituals. They know which lever needs a bit of persuasion on cold mornings and which replacement part must be machined by hand because nobody has made the original since 1987. They also know how to train new employees, because the knowledge still exists inside the company.
Crisis
Then, suddenly, a few large customers run into a crisis.
Orders slow down. The warehouses fill up with part X. Prices fall.
The company downsizes. Then it downsizes again. But nobody really notices, because stocks are still full. Customers are not yet affected. Purchasing departments can still get part X from inventory. Balance sheets still look fine. The problem has not yet reached the spreadsheet.
If this goes on for long enough, the company goes broke.
Again, nobody really notices. Stocks are still full. Some people may worry, but as long as no current production line is stopping and no quarterly number is visibly bleeding, nothing urgent is done.
The employees move on. They retire, change industries, start new careers. The old machine is sold for scrap. The building is repurposed. The knowledge evaporates.
Recovery
A few years later, demand picks up again.
The warehouses slowly empty. Stocks run low. People start ordering part X again, only to discover that nobody is offering it anymore.
Now someone decides to restart production.
The first thing they discover is that there is no machine. Building a new one would be prohibitively expensive, assuming anyone still knows how to build it properly. So they desperately search for an old one.
They get lucky. In a scrapyard, they find a machine that used to produce part X. It is rusted, incomplete, and dysfunctional. Naturally, they buy it.
Now they try to get it working again.
But there is another problem. There are no people left who know how to maintain it. So they hire someone who understands industrial maintenance in general, but has never worked on this specific machine. That person does their best. They improvise. They read old manuals. They reverse-engineer undocumented fixes. They keep the machine alive with skill, patience, and increasing amounts of despair.
But it breaks down every few hours. Output is abysmal.
Bottlenecks
And now that one poor maintenance person is overworked. They need help. But training help requires time, and the only person who can train others is the same person needed to keep the machine barely running. Every hour spent teaching is an hour not spent preventing the next failure.
Very few businesses survive this phase.
There is no institutional knowledge anymore. New people are hired, begin training, look at the state of the machine, the chaos of the process, and the constant emergency mode, and conclude that the business is doomed anyway. Then they quit.
Churn becomes terrible. Even if the company survives financially, it remains fragile. It is always just one or two people quitting away from disaster.
At the end the world decides it needs to get rid of part X as the supply is too fragile.
Summary
This is still very much simplified. The reality is more complex, more ugly.
And that is the part many people miss: a supply chain is not just warehouses, contracts, prices, and transport routes. It is also people, habits, obsolete machines, informal knowledge, and boring little skills that nobody values until they are gone.
RE: https://infosec.exchange/@masek/116656652453518116
If you know anything about what the #vinyl record producing industry went through when vinyl became a big thing again, a lot of this is going to sound familiar. Thankfully, not every press got junked, not every operator died and a market was created for high dollar "limited" pressings. Today, we have a mix of the old presses and brand new designs, but a lot of people will tell you it was a bitch coming back from the near dead.
-
What a lot of people do not understand is the fragility of supply chains.
Setup
Let us assume there is a part X that is used by a large number of other companies. It is not glamorous, it is not expensive, and nobody outside a small circle of specialists ever thinks about it. But it is needed. Without it, other products cannot be finished.
Everything is fine
Part X is made by a small company with a few dozen employees and a machine that is several decades old.
Everything runs smoothly. The company knows how to maintain the machine. They know how to operate it. They know its strange noises, its moods, its undocumented rituals. They know which lever needs a bit of persuasion on cold mornings and which replacement part must be machined by hand because nobody has made the original since 1987. They also know how to train new employees, because the knowledge still exists inside the company.
Crisis
Then, suddenly, a few large customers run into a crisis.
Orders slow down. The warehouses fill up with part X. Prices fall.
The company downsizes. Then it downsizes again. But nobody really notices, because stocks are still full. Customers are not yet affected. Purchasing departments can still get part X from inventory. Balance sheets still look fine. The problem has not yet reached the spreadsheet.
If this goes on for long enough, the company goes broke.
Again, nobody really notices. Stocks are still full. Some people may worry, but as long as no current production line is stopping and no quarterly number is visibly bleeding, nothing urgent is done.
The employees move on. They retire, change industries, start new careers. The old machine is sold for scrap. The building is repurposed. The knowledge evaporates.
Recovery
A few years later, demand picks up again.
The warehouses slowly empty. Stocks run low. People start ordering part X again, only to discover that nobody is offering it anymore.
Now someone decides to restart production.
The first thing they discover is that there is no machine. Building a new one would be prohibitively expensive, assuming anyone still knows how to build it properly. So they desperately search for an old one.
They get lucky. In a scrapyard, they find a machine that used to produce part X. It is rusted, incomplete, and dysfunctional. Naturally, they buy it.
Now they try to get it working again.
But there is another problem. There are no people left who know how to maintain it. So they hire someone who understands industrial maintenance in general, but has never worked on this specific machine. That person does their best. They improvise. They read old manuals. They reverse-engineer undocumented fixes. They keep the machine alive with skill, patience, and increasing amounts of despair.
But it breaks down every few hours. Output is abysmal.
Bottlenecks
And now that one poor maintenance person is overworked. They need help. But training help requires time, and the only person who can train others is the same person needed to keep the machine barely running. Every hour spent teaching is an hour not spent preventing the next failure.
Very few businesses survive this phase.
There is no institutional knowledge anymore. New people are hired, begin training, look at the state of the machine, the chaos of the process, and the constant emergency mode, and conclude that the business is doomed anyway. Then they quit.
Churn becomes terrible. Even if the company survives financially, it remains fragile. It is always just one or two people quitting away from disaster.
At the end the world decides it needs to get rid of part X as the supply is too fragile.
Summary
This is still very much simplified. The reality is more complex, more ugly.
And that is the part many people miss: a supply chain is not just warehouses, contracts, prices, and transport routes. It is also people, habits, obsolete machines, informal knowledge, and boring little skills that nobody values until they are gone.
@masek You're mostly not wrong, except for 'nobody outside a small circle of specialists ever thinks about it' and 'everything is fine'.
If your product or solution absolutely depends on a component supplied by one company a) they have you over a barrel and b) it's an existential risk to your company
As mentioned by another commentator, a business should look for a second source.
Ideally look at developing your own too. The only reliable ways of e.g. getting a decent PS/2 to USB converter these days is a) make your own b) get an open source alternative or c) test an offering from a manufacturer and sign a contract for a guaranteed supply.
You can get 'a converter' easily, but you won't know which chip is inside it, if that chip will continue to be inside it, and if the company will stay in business.
There isn't the market for large commercial companies anymore to mass produce, so it's largely left to cheap fly by night companies, and open source tested to work properly is the best way.
-
What a lot of people do not understand is the fragility of supply chains.
Setup
Let us assume there is a part X that is used by a large number of other companies. It is not glamorous, it is not expensive, and nobody outside a small circle of specialists ever thinks about it. But it is needed. Without it, other products cannot be finished.
Everything is fine
Part X is made by a small company with a few dozen employees and a machine that is several decades old.
Everything runs smoothly. The company knows how to maintain the machine. They know how to operate it. They know its strange noises, its moods, its undocumented rituals. They know which lever needs a bit of persuasion on cold mornings and which replacement part must be machined by hand because nobody has made the original since 1987. They also know how to train new employees, because the knowledge still exists inside the company.
Crisis
Then, suddenly, a few large customers run into a crisis.
Orders slow down. The warehouses fill up with part X. Prices fall.
The company downsizes. Then it downsizes again. But nobody really notices, because stocks are still full. Customers are not yet affected. Purchasing departments can still get part X from inventory. Balance sheets still look fine. The problem has not yet reached the spreadsheet.
If this goes on for long enough, the company goes broke.
Again, nobody really notices. Stocks are still full. Some people may worry, but as long as no current production line is stopping and no quarterly number is visibly bleeding, nothing urgent is done.
The employees move on. They retire, change industries, start new careers. The old machine is sold for scrap. The building is repurposed. The knowledge evaporates.
Recovery
A few years later, demand picks up again.
The warehouses slowly empty. Stocks run low. People start ordering part X again, only to discover that nobody is offering it anymore.
Now someone decides to restart production.
The first thing they discover is that there is no machine. Building a new one would be prohibitively expensive, assuming anyone still knows how to build it properly. So they desperately search for an old one.
They get lucky. In a scrapyard, they find a machine that used to produce part X. It is rusted, incomplete, and dysfunctional. Naturally, they buy it.
Now they try to get it working again.
But there is another problem. There are no people left who know how to maintain it. So they hire someone who understands industrial maintenance in general, but has never worked on this specific machine. That person does their best. They improvise. They read old manuals. They reverse-engineer undocumented fixes. They keep the machine alive with skill, patience, and increasing amounts of despair.
But it breaks down every few hours. Output is abysmal.
Bottlenecks
And now that one poor maintenance person is overworked. They need help. But training help requires time, and the only person who can train others is the same person needed to keep the machine barely running. Every hour spent teaching is an hour not spent preventing the next failure.
Very few businesses survive this phase.
There is no institutional knowledge anymore. New people are hired, begin training, look at the state of the machine, the chaos of the process, and the constant emergency mode, and conclude that the business is doomed anyway. Then they quit.
Churn becomes terrible. Even if the company survives financially, it remains fragile. It is always just one or two people quitting away from disaster.
At the end the world decides it needs to get rid of part X as the supply is too fragile.
Summary
This is still very much simplified. The reality is more complex, more ugly.
And that is the part many people miss: a supply chain is not just warehouses, contracts, prices, and transport routes. It is also people, habits, obsolete machines, informal knowledge, and boring little skills that nobody values until they are gone.
-
@jt_rebelo @masek and a few month later everybody has moved on to a better or cheaper process.. it’s not always good to rely on steam engine parts after their eol.
@eckes @jt_rebelo @masek I'm sure anyone could compress decades of research and experience into a few months... oh wait, instead we get product shortages and inferior solutions until the decades needed have passed.
-
What a lot of people do not understand is the fragility of supply chains.
Setup
Let us assume there is a part X that is used by a large number of other companies. It is not glamorous, it is not expensive, and nobody outside a small circle of specialists ever thinks about it. But it is needed. Without it, other products cannot be finished.
Everything is fine
Part X is made by a small company with a few dozen employees and a machine that is several decades old.
Everything runs smoothly. The company knows how to maintain the machine. They know how to operate it. They know its strange noises, its moods, its undocumented rituals. They know which lever needs a bit of persuasion on cold mornings and which replacement part must be machined by hand because nobody has made the original since 1987. They also know how to train new employees, because the knowledge still exists inside the company.
Crisis
Then, suddenly, a few large customers run into a crisis.
Orders slow down. The warehouses fill up with part X. Prices fall.
The company downsizes. Then it downsizes again. But nobody really notices, because stocks are still full. Customers are not yet affected. Purchasing departments can still get part X from inventory. Balance sheets still look fine. The problem has not yet reached the spreadsheet.
If this goes on for long enough, the company goes broke.
Again, nobody really notices. Stocks are still full. Some people may worry, but as long as no current production line is stopping and no quarterly number is visibly bleeding, nothing urgent is done.
The employees move on. They retire, change industries, start new careers. The old machine is sold for scrap. The building is repurposed. The knowledge evaporates.
Recovery
A few years later, demand picks up again.
The warehouses slowly empty. Stocks run low. People start ordering part X again, only to discover that nobody is offering it anymore.
Now someone decides to restart production.
The first thing they discover is that there is no machine. Building a new one would be prohibitively expensive, assuming anyone still knows how to build it properly. So they desperately search for an old one.
They get lucky. In a scrapyard, they find a machine that used to produce part X. It is rusted, incomplete, and dysfunctional. Naturally, they buy it.
Now they try to get it working again.
But there is another problem. There are no people left who know how to maintain it. So they hire someone who understands industrial maintenance in general, but has never worked on this specific machine. That person does their best. They improvise. They read old manuals. They reverse-engineer undocumented fixes. They keep the machine alive with skill, patience, and increasing amounts of despair.
But it breaks down every few hours. Output is abysmal.
Bottlenecks
And now that one poor maintenance person is overworked. They need help. But training help requires time, and the only person who can train others is the same person needed to keep the machine barely running. Every hour spent teaching is an hour not spent preventing the next failure.
Very few businesses survive this phase.
There is no institutional knowledge anymore. New people are hired, begin training, look at the state of the machine, the chaos of the process, and the constant emergency mode, and conclude that the business is doomed anyway. Then they quit.
Churn becomes terrible. Even if the company survives financially, it remains fragile. It is always just one or two people quitting away from disaster.
At the end the world decides it needs to get rid of part X as the supply is too fragile.
Summary
This is still very much simplified. The reality is more complex, more ugly.
And that is the part many people miss: a supply chain is not just warehouses, contracts, prices, and transport routes. It is also people, habits, obsolete machines, informal knowledge, and boring little skills that nobody values until they are gone.
this is good, but it doesn't include even one actual real concrete example
let me give you one
I worked for a small biotech that made a thing that is widely used in DNA sequencing, which in turn is now used in cancer diagnosticsso important
the thing we made used an $ electronic part that is made by one company in japan; there is no drop in replacment product available in the entire world
there are other products we could use to replace this part, but it would be a **LOT of time and money to redo everything - close to bankruptcy money
-
@masek much of what you describe has parallels in the US academic scientific enterprise. Labs are specialized. They use unique, often home-grown, assays and equipment. Technicians keep them running and train replacements. When there’s a lapse in government funding (ie grants and contracts), staff/knowledge are lost, equipment fails, and it’s remarkably difficult and costly to return to normal if funding resumes. #science
@SkeeCR @masek
Right, this happens.Another example: Dr. Gotagrant and his large team of graduate students are involved in a multi year study using a kit from company A, because it recovers the product that they are studying at the highest purity. Protocols are written around the use of this kit and things run smoothly until company Z buys A and discontinues the kit because they have one too, which is “just as good”.
-
What a lot of people do not understand is the fragility of supply chains.
Setup
Let us assume there is a part X that is used by a large number of other companies. It is not glamorous, it is not expensive, and nobody outside a small circle of specialists ever thinks about it. But it is needed. Without it, other products cannot be finished.
Everything is fine
Part X is made by a small company with a few dozen employees and a machine that is several decades old.
Everything runs smoothly. The company knows how to maintain the machine. They know how to operate it. They know its strange noises, its moods, its undocumented rituals. They know which lever needs a bit of persuasion on cold mornings and which replacement part must be machined by hand because nobody has made the original since 1987. They also know how to train new employees, because the knowledge still exists inside the company.
Crisis
Then, suddenly, a few large customers run into a crisis.
Orders slow down. The warehouses fill up with part X. Prices fall.
The company downsizes. Then it downsizes again. But nobody really notices, because stocks are still full. Customers are not yet affected. Purchasing departments can still get part X from inventory. Balance sheets still look fine. The problem has not yet reached the spreadsheet.
If this goes on for long enough, the company goes broke.
Again, nobody really notices. Stocks are still full. Some people may worry, but as long as no current production line is stopping and no quarterly number is visibly bleeding, nothing urgent is done.
The employees move on. They retire, change industries, start new careers. The old machine is sold for scrap. The building is repurposed. The knowledge evaporates.
Recovery
A few years later, demand picks up again.
The warehouses slowly empty. Stocks run low. People start ordering part X again, only to discover that nobody is offering it anymore.
Now someone decides to restart production.
The first thing they discover is that there is no machine. Building a new one would be prohibitively expensive, assuming anyone still knows how to build it properly. So they desperately search for an old one.
They get lucky. In a scrapyard, they find a machine that used to produce part X. It is rusted, incomplete, and dysfunctional. Naturally, they buy it.
Now they try to get it working again.
But there is another problem. There are no people left who know how to maintain it. So they hire someone who understands industrial maintenance in general, but has never worked on this specific machine. That person does their best. They improvise. They read old manuals. They reverse-engineer undocumented fixes. They keep the machine alive with skill, patience, and increasing amounts of despair.
But it breaks down every few hours. Output is abysmal.
Bottlenecks
And now that one poor maintenance person is overworked. They need help. But training help requires time, and the only person who can train others is the same person needed to keep the machine barely running. Every hour spent teaching is an hour not spent preventing the next failure.
Very few businesses survive this phase.
There is no institutional knowledge anymore. New people are hired, begin training, look at the state of the machine, the chaos of the process, and the constant emergency mode, and conclude that the business is doomed anyway. Then they quit.
Churn becomes terrible. Even if the company survives financially, it remains fragile. It is always just one or two people quitting away from disaster.
At the end the world decides it needs to get rid of part X as the supply is too fragile.
Summary
This is still very much simplified. The reality is more complex, more ugly.
And that is the part many people miss: a supply chain is not just warehouses, contracts, prices, and transport routes. It is also people, habits, obsolete machines, informal knowledge, and boring little skills that nobody values until they are gone.
@masek
Also opensource in a nutshell -
R relay@relay.an.exchange shared this topic
