Oh fuck #postgres.
-
Oh fuck #postgres. In 18 they made a change in data path from
/var/lib/postgresql/datato/var/lib/postgresql/<major_version>.They did exactly zero checks so users will shoot themselves in a leg in two ways:
- kinda benign: after 17 → 18 upgrade it won’t start
- fatal: user mounted docker volume at the old pathContainer restarts and data gone (that’s me)
What the actual fuck. Why on earth you couldn’t do a simple fucking check that old directory exists and has data, but new directory is empty, and then start with directory with data spitting warnings that default has been changed and you need to do something about that.
This is utter incompetence and giving exactly zero fucks about your user base
-
R relay@relay.an.exchange shared this topic
-
Oh fuck #postgres. In 18 they made a change in data path from
/var/lib/postgresql/datato/var/lib/postgresql/<major_version>.They did exactly zero checks so users will shoot themselves in a leg in two ways:
- kinda benign: after 17 → 18 upgrade it won’t start
- fatal: user mounted docker volume at the old pathContainer restarts and data gone (that’s me)
What the actual fuck. Why on earth you couldn’t do a simple fucking check that old directory exists and has data, but new directory is empty, and then start with directory with data spitting warnings that default has been changed and you need to do something about that.
This is utter incompetence and giving exactly zero fucks about your user base
@alex this is me migrating tonight to 18. Had to switch to a full pgdump for 12 To..
-
Oh fuck #postgres. In 18 they made a change in data path from
/var/lib/postgresql/datato/var/lib/postgresql/<major_version>.They did exactly zero checks so users will shoot themselves in a leg in two ways:
- kinda benign: after 17 → 18 upgrade it won’t start
- fatal: user mounted docker volume at the old pathContainer restarts and data gone (that’s me)
What the actual fuck. Why on earth you couldn’t do a simple fucking check that old directory exists and has data, but new directory is empty, and then start with directory with data spitting warnings that default has been changed and you need to do something about that.
This is utter incompetence and giving exactly zero fucks about your user base
@alex What is your environment or OS?
-
Oh fuck #postgres. In 18 they made a change in data path from
/var/lib/postgresql/datato/var/lib/postgresql/<major_version>.They did exactly zero checks so users will shoot themselves in a leg in two ways:
- kinda benign: after 17 → 18 upgrade it won’t start
- fatal: user mounted docker volume at the old pathContainer restarts and data gone (that’s me)
What the actual fuck. Why on earth you couldn’t do a simple fucking check that old directory exists and has data, but new directory is empty, and then start with directory with data spitting warnings that default has been changed and you need to do something about that.
This is utter incompetence and giving exactly zero fucks about your user base
@alex I hope you know, that a dump/restore is required to migrate data from any previous PostgreSQL major release.
https://www.postgresql.org/docs/current/upgrading.htmlAlso, with PostgreSQL 18, initdb creates new databases by default with enabled data checksums.
Based on this change, an in-place migration with pg_upgrade is only possible, when database running the old PostgreSQL version is also created with data checksums. -
@alex I hope you know, that a dump/restore is required to migrate data from any previous PostgreSQL major release.
https://www.postgresql.org/docs/current/upgrading.htmlAlso, with PostgreSQL 18, initdb creates new databases by default with enabled data checksums.
Based on this change, an in-place migration with pg_upgrade is only possible, when database running the old PostgreSQL version is also created with data checksums. -
@alex What is your environment or OS?
@ascherbaum @alex Yeah. I wonder what packages are being used?
-
@tomasv @alex Yes, I know.
I'm not sure if many people in the self-hosting sector are truly aware of some complexity when dealing with PostgreSQL DBs.My main intention was the advice, that it is not possible to mount the dabase files from a previous PostgreSQL major version without a migration with a new PostgreSQL major version.
Also using pg_upgrade is, when running PostgreSQL in a container, not so easy.
One possible way can be to use this container image:
https://github.com/tianon/docker-postgres-upgrade -
@ascherbaum @alex Yeah. I wonder what packages are being used?
@tomasv @ascherbaum nothing special: dockerized environment, official docker image. I’ve deployed one of the services with pg18 but used default mount path to
data. So I was unplesantly surprised to find empty database after 18.2 to 18.3 image upgrade. Old container was gc’ed because both pg and app healthchecks passed. Which is technically correct: app rolled bootstrap migration and went up in blank state and pg was indeed available. But data is gone. It was expected that there could be issues with 18, so it was first deployment with not that important data.Another another lesson that observability as useful as your risk assessment. Also lesson in trust in vendors: no matter how consistent in their changes they were in past, they absolutelly will do stupid things. That on me. I could’ve paid more attention that’s also on me.
But doing change of this magnitude so careless and without any proper discussion of what can go wrong for downstream users is on the people who made this damn pull request merged.
Worst part this change is purely about aesthetics and addresses exactly zero issues: pg sill gives no shit about operational complexity of major version upgrades. While I respect people around pg, database itself is an example of software that is plan hostile to users. It’s never joy, it always different from of pain.
-
@tomasv @ascherbaum nothing special: dockerized environment, official docker image. I’ve deployed one of the services with pg18 but used default mount path to
data. So I was unplesantly surprised to find empty database after 18.2 to 18.3 image upgrade. Old container was gc’ed because both pg and app healthchecks passed. Which is technically correct: app rolled bootstrap migration and went up in blank state and pg was indeed available. But data is gone. It was expected that there could be issues with 18, so it was first deployment with not that important data.Another another lesson that observability as useful as your risk assessment. Also lesson in trust in vendors: no matter how consistent in their changes they were in past, they absolutelly will do stupid things. That on me. I could’ve paid more attention that’s also on me.
But doing change of this magnitude so careless and without any proper discussion of what can go wrong for downstream users is on the people who made this damn pull request merged.
Worst part this change is purely about aesthetics and addresses exactly zero issues: pg sill gives no shit about operational complexity of major version upgrades. While I respect people around pg, database itself is an example of software that is plan hostile to users. It’s never joy, it always different from of pain.
@alex @tomasv What does "official docker image" mean?
If you refer to this:
https://hub.docker.com/_/postgres
This is kind of the official "Docker Postgres Image", but this is not maintained by the PG project.
Blaming Postgres for this is the wrong audience.
-
@alex @tomasv What does "official docker image" mean?
If you refer to this:
https://hub.docker.com/_/postgres
This is kind of the official "Docker Postgres Image", but this is not maintained by the PG project.
Blaming Postgres for this is the wrong audience.
thanks for pointing this out. My apologies for pg people. I stand corrected, it's maintained by "docker community" (what the fuck "official" means then)
