I just concluded a decade long experiment.
-
@swope @Aaron_DeVries I once saw a datasheet for an Intel SSD, which basically said it likes being run hot and stored cold, and that combination gave the longest data retention lifetime. Hot/hot, not so much.
If I remember correctly, that's because higher temperatures make it easier for electrons to escape from the floating gates. Which is bad for data retention, but as long as the drive is powered on the firmware can handle that by re-writing the data occasionally. But it also means deliberate erasures are easier and cause less physical damage, so the SSD degrades slower if it's run hot. And after enough use, that matters more than the effect of temperature on the durability of any individual write.
This is interesting.
Sadly, we didn't get the internal NVMe temperatures during our environmental testing.
I think the drives' spec said 35°C ambient for the max operating temperature, and we would typically see 45°C internal from `nvme smart-log /dev/nvme0` (I might not be remembering the command correctly).
-
This is interesting.
Sadly, we didn't get the internal NVMe temperatures during our environmental testing.
I think the drives' spec said 35°C ambient for the max operating temperature, and we would typically see 45°C internal from `nvme smart-log /dev/nvme0` (I might not be remembering the command correctly).
I kinda supposed that there was an ideal temperature for writing that the drives try to maintain, but when ambient is too high they can't. Then what gets written is on the edge of being readable, and normal temperature storage losses degrade that in a few hours.
Lots of hand waving on my part. We just made a procedure and carried on. But it would be pretty insidious to look at your data right after recording and it looks good, but it's gone the next week.
-
@Aaron_DeVries
While doing environmental testing on a helicopter payload, I learned that NVMe drives (and perhaps all SSDs) can write data at high ambient temperatures >40 C, but the data is less permanent than if you write data at normal ambient temperatures.
So on hot days we had to hurry and copy our 4TB drives after the flight because the data had a half-life of a dozen hours or so.That phenomenon is already documented, but I don't think it's widely known.
@swope @Aaron_DeVries it’s inherent to all SSDs, to varying degrees, because it increases leakage from the cells — from memory offline storage at temperature is actually worse than online behaviour.
you can certainly design SSDs to be better at high temperature, mostly by doing the normal things to decrease leakage, like single layer cells, decreasing density of the cells, increasing the number of error correction bits, etc. you can also just mitigate the online case in firmware, by increasing cell refresh frequency.
such SSDs are manufactured, primarily for industrial, aviation, automobile, and similar applications, and especially targeting such markets you’ll often find data sheets giving a lot more detail about retention at temperature.
-
I just concluded a decade long experiment. I had a USB flash drive in a jar buried in my back yard since 2015. I dug it up, plugged it in and it suffered no data loss after 11 years idle underground.
It's a usless experiment but everyone needs hobbies.
@Aaron_DeVries I'm sure this will also work for any other functioning USB drive that's hermetically sealed against most elements, agents, and phenomena for an arbitrary length of time. Like if I put my car in a museum, instead of using it.
-
I just concluded a decade long experiment. I had a USB flash drive in a jar buried in my back yard since 2015. I dug it up, plugged it in and it suffered no data loss after 11 years idle underground.
It's a usless experiment but everyone needs hobbies.
@Aaron_DeVries ……….. phew, I’ll sleep a little more soundly tonight! (0;=
-
I just concluded a decade long experiment. I had a USB flash drive in a jar buried in my back yard since 2015. I dug it up, plugged it in and it suffered no data loss after 11 years idle underground.
It's a usless experiment but everyone needs hobbies.
It would have been more interstate if you had buried a control USB in a bottle full of water.
-
I just concluded a decade long experiment. I had a USB flash drive in a jar buried in my back yard since 2015. I dug it up, plugged it in and it suffered no data loss after 11 years idle underground.
It's a usless experiment but everyone needs hobbies.
@Aaron_DeVries
Size of drive? (Physical AND informational)I have this idea that drives with smaller individual memory cells are more susceptible to leaking.
-
@Aaron_DeVries
While doing environmental testing on a helicopter payload, I learned that NVMe drives (and perhaps all SSDs) can write data at high ambient temperatures >40 C, but the data is less permanent than if you write data at normal ambient temperatures.
So on hot days we had to hurry and copy our 4TB drives after the flight because the data had a half-life of a dozen hours or so.That phenomenon is already documented, but I don't think it's widely known.
@swope @Aaron_DeVries the main issue is the Tdelta between active and power off temps. if both are equal you end up with lower retention. Documented in JEP122
-
@Aaron_DeVries
While doing environmental testing on a helicopter payload, I learned that NVMe drives (and perhaps all SSDs) can write data at high ambient temperatures >40 C, but the data is less permanent than if you write data at normal ambient temperatures.
So on hot days we had to hurry and copy our 4TB drives after the flight because the data had a half-life of a dozen hours or so.That phenomenon is already documented, but I don't think it's widely known.
@swope @Aaron_DeVries
I should know this and I didn't.
Thank you! -
@Aaron_DeVries
While doing environmental testing on a helicopter payload, I learned that NVMe drives (and perhaps all SSDs) can write data at high ambient temperatures >40 C, but the data is less permanent than if you write data at normal ambient temperatures.
So on hot days we had to hurry and copy our 4TB drives after the flight because the data had a half-life of a dozen hours or so.That phenomenon is already documented, but I don't think it's widely known.
NVMe drives tend to get hot during usage - at least some models do at faster speeds. 40° C is not uncommon and by far not the highest temperature I have seen in working systems.
So what was the NVMe (not ambient) temperature that lead to this outcome? Should have been quite a lot hotter then?
CC: @Aaron_DeVries@mastodon.social
-
NVMe drives tend to get hot during usage - at least some models do at faster speeds. 40° C is not uncommon and by far not the highest temperature I have seen in working systems.
So what was the NVMe (not ambient) temperature that lead to this outcome? Should have been quite a lot hotter then?
CC: @Aaron_DeVries@mastodon.social@andi
Sorry I don't have the data available on the case or internal temperatures.I agree that it was probably quite high.
-
@andi
Sorry I don't have the data available on the case or internal temperatures.I agree that it was probably quite high.
Breath of relief
. If NVMe drives start loosing data at 40° C - that would be pretty bad!
CC: @Aaron_DeVries@mastodon.social
-
@Aaron_DeVries
While doing environmental testing on a helicopter payload, I learned that NVMe drives (and perhaps all SSDs) can write data at high ambient temperatures >40 C, but the data is less permanent than if you write data at normal ambient temperatures.
So on hot days we had to hurry and copy our 4TB drives after the flight because the data had a half-life of a dozen hours or so.That phenomenon is already documented, but I don't think it's widely known.
"@Aaron_DeVries
Mientras realizaba pruebas ambientales en la carga útil de un helicóptero, descubrí que las unidades NVMe (y tal vez todos los SSD) pueden escribir datos a temperaturas ambiente elevadas (>40 °C), pero que dichos datos resultan menos permanentes que si se escribieran a temperaturas ambiente normales.
Por ello, en los días calurosos teníamos que darnos prisa en copiar el contenido de nuestras unidades de 4 TB tras el vuelo, ya que los datos tenían una vida media de unas doce horas, aproximadamente.Este fenómeno ya está documentado, aunque no creo que sea ampliamente conocido."
-
@Aaron_DeVries
While doing environmental testing on a helicopter payload, I learned that NVMe drives (and perhaps all SSDs) can write data at high ambient temperatures >40 C, but the data is less permanent than if you write data at normal ambient temperatures.
So on hot days we had to hurry and copy our 4TB drives after the flight because the data had a half-life of a dozen hours or so.That phenomenon is already documented, but I don't think it's widely known.
@swope @Aaron_DeVries huh.. 40 ambient is not that much! People live in climates where it's close to 50 in heatwaves, I haven't heard of anyone losing data to heatwaves…
-
@swope @Aaron_DeVries huh.. 40 ambient is not that much! People live in climates where it's close to 50 in heatwaves, I haven't heard of anyone losing data to heatwaves…
That payload was writing continuously about as fast as the drives could go. Not a typical use case I think.
-
@swope @Aaron_DeVries the main issue is the Tdelta between active and power off temps. if both are equal you end up with lower retention. Documented in JEP122
@evey @swope @Aaron_DeVries so the storage unit needs to cool down for the bits to properly dry?
-
@Aaron_DeVries
I would also like to know.@Aaron_DeVries @swope A lot of physical processes are thermally activated. At higher temperatures the barrier between two (meta)stable states is often easier to cross, which is why things can go wrong at higher temperature.
If it is, as I read it, bits _written_ at higher temperature having a shorter lifetime even at lower temperature after, I can imagine that the state written at higher temp isn’t as far down into the local energy minimum.
-
I just concluded a decade long experiment. I had a USB flash drive in a jar buried in my back yard since 2015. I dug it up, plugged it in and it suffered no data loss after 11 years idle underground.
It's a usless experiment but everyone needs hobbies.
Was the flash memory in that USB Drive older SLC type?
In which case I would anticipate that it would have a longer data retention. 11 years is impressive though.
What make and model was that drive?
-
@swope @Aaron_DeVries huh.. 40 ambient is not that much! People live in climates where it's close to 50 in heatwaves, I haven't heard of anyone losing data to heatwaves…
@valpackett @swope @Aaron_DeVries I have heard some colloquial stories of "cold"/unpowered backups on flash becoming completely unusable after being stored in a hot safe. it would be interesting to see exactly how sensitive and reliable this effect is
-
@Aaron_DeVries @swope A lot of physical processes are thermally activated. At higher temperatures the barrier between two (meta)stable states is often easier to cross, which is why things can go wrong at higher temperature.
If it is, as I read it, bits _written_ at higher temperature having a shorter lifetime even at lower temperature after, I can imagine that the state written at higher temp isn’t as far down into the local energy minimum.
Yes, I imagine that something like that is going on. And the manufacturer is honest about the 35C max operating temperature.
For the user, they write a bunch of data and immediately verify the data are good -- not looking closely at the temperature log. But hours/days later their files become corrupted.
It's something to be aware of, and a curious physics question.
