[ADD] various: deduplicate soon-to-be-unique fields before migration

Some target versions add a ``UNIQUE`` constraint on a field that allowed
duplicates in the source version (e.g. ``utm.source.name`` in 16.0). The
unique index creation then aborts the whole registry load with a
``psycopg2.errors.UniqueViolation``.

Declare such constraints per version in ``known_changes.yaml`` under the
``new_unique_constraints`` key (only ``model`` and ``fields`` needed).
During ``prepare_db.sh``, ``dedup_unique_constraints.py`` aggregates the
declarations of every traversed version and, on the source database,
renames duplicates by suffixing `` [<id>]`` while preserving foreign keys.

The actual PostgreSQL column type is used (not the ORM ``translate``
attribute) so both ``varchar`` and already-converted ``jsonb`` columns are
handled: ``jsonb`` duplicates are compared and renamed on every language
key (indexed ``en_US`` value). Renamed records are reported in a
timestamped JSON file and summarized at the end of ``migration.log``.
This commit is contained in:
Stéphan Sainléger
2026-07-13 15:23:26 +02:00
parent 781407fe4c
commit 718b367e86
6 changed files with 467 additions and 4 deletions

View File

@@ -232,4 +232,20 @@ log_step "POST-UPGRADE PROCESSES"
"${SCRIPT_DIR}/scripts/finalize_db.sh" "$FINALE_DB_NAME" "$FINALE_SERVICE_NAME"
log_step "UPGRADE PROCESS ENDED WITH SUCCESS"
# Report any records that were renamed to satisfy new unique constraints.
# The deduplication runs on the source DB (ou${ORIGIN_VERSION}) during
# prepare_db.sh and writes a timestamped JSON report to /tmp.
readarray -t dedup_reports < <(ls -1t /tmp/dedup_unique_constraints_"${COPY_DB_NAME}"_*.json 2>/dev/null || true)
if [[ ${#dedup_reports[@]} -gt 0 ]]; then
latest_report="${dedup_reports[0]}"
renamed_count=$(yq -p=json '.renamed_count' "$latest_report" 2>/dev/null || echo "?")
log_info "Unique-constraint deduplication: ${renamed_count} record(s) renamed in the source DB before migration."
log_info "Details: ${latest_report}"
if [[ "$renamed_count" =~ ^[0-9]+$ && "$renamed_count" -gt 0 ]]; then
yq -p=json -r '.renamed[] | " - \(.model) #\(.id) [\(.field)]: \(.old_value) -> \(.new_value)"' \
"$latest_report" 2>/dev/null || true
fi
fi
log_info "Full logs available at: ${LOG_FILE}"