Cleaning up duplicate packages when yum has failed

1. The problem

If a yum update run fails in some spectacular manner such as, but not limited to, your ssh session disconnecting in mid-update, yum being killed by the OOM killer, system powering off etc, then this can leave your system in an indeterminate state with two sets of packages installed. It is then impossible to run yum-complete-transaction or any new yum update as it complains about the duplicates.

2. Fix overview

To fix this you will need to perform a manual recovery. Before you start you are strongly advised to backup the contents of, at least, /var/lib/rpm and /var/lib/yum so that you have a way to get back to your currently broken state in case you break it even more while attempting to recover.

Updated info: it looks like the package-cleanup command got a new(ish) switch starting in 7.6: --removenewestdupes which does exactly what the rest of this massive article used to attempt to do manually. Try running package-cleanup --cleandupes --removenewestdupes as a first step before you carry on. If the switch is unknown then you might need to follow the rest of this article. Otherwise it probably already did what was needed and you can rerun yum update and this time, make sure the power's on/tmux is in use etc.

3. Old fix info - probably no longer required

So the process here is to make a list of all the duplicate packages. From this list we will remove just the database entries for only the newer versions and once all those are gone, re-run yum update and it will download the newer packages again and re-update your system and this will ensure that the newer ones are fully and correctly installed.

4. Fix details

The package-cleanup command is part of the yum-utils package so you may need to install that if it's not already available. Once it is installed, run package-cleanup --dupes --noplugins and this will give you output that looks somewhat like this, which I generated on a test VM by "powering it off" in mid-update:

[root@localhost ~]# package-cleanup --dupes
Loaded plugins: fastestmirror
nss-tools-3.28.4-8.el7.x86_64
nss-tools-3.28.4-15.el7_4.x86_64
grub2-common-2.02-0.64.el7.centos.noarch
grub2-common-2.02-0.65.el7.centos.2.noarch
wpa_supplicant-2.6-5.el7.x86_64
wpa_supplicant-2.6-5.el7_4.1.x86_64
grub2-tools-extra-2.02-0.65.el7.centos.2.x86_64
grub2-tools-extra-2.02-0.64.el7.centos.x86_64
nss-sysinit-3.28.4-15.el7_4.x86_64
nss-sysinit-3.28.4-8.el7.x86_64
grub2-tools-2.02-0.65.el7.centos.2.x86_64
grub2-tools-2.02-0.64.el7.centos.x86_64
bind-license-9.9.4-50.el7.noarch
bind-license-9.9.4-51.el7.noarch
nss-3.28.4-15.el7_4.x86_64
nss-3.28.4-8.el7.x86_64
kmod-20-15.el7_4.4.x86_64
kmod-20-15.el7.x86_64
systemd-219-42.el7.x86_64
systemd-219-42.el7_4.4.x86_64
selinux-policy-3.13.1-166.el7_4.5.noarch
selinux-policy-3.13.1-166.el7.noarch
grub2-pc-modules-2.02-0.65.el7.centos.2.noarch
grub2-pc-modules-2.02-0.64.el7.centos.noarch
NetworkManager-libnm-1.8.0-11.el7_4.x86_64
NetworkManager-libnm-1.8.0-9.el7.x86_64
NetworkManager-1.8.0-11.el7_4.x86_64
NetworkManager-1.8.0-9.el7.x86_64
nss-softokn-freebl-3.28.3-6.el7.x86_64
nss-softokn-freebl-3.28.3-8.el7_4.x86_64
grub2-pc-2.02-0.65.el7.centos.2.x86_64
grub2-pc-2.02-0.64.el7.centos.x86_64
openssh-server-7.4p1-13.el7_4.x86_64
openssh-server-7.4p1-11.el7.x86_64
systemd-libs-219-42.el7_4.4.x86_64
systemd-libs-219-42.el7.x86_64
cpio-2.11-24.el7.x86_64
cpio-2.11-25.el7_4.x86_64
kmod-libs-20-15.el7.x86_64
kmod-libs-20-15.el7_4.4.x86_64
ncurses-libs-5.9-13.20130511.el7.x86_64
ncurses-libs-5.9-14.20130511.el7_4.x86_64
nss-softokn-3.28.3-8.el7_4.x86_64
nss-softokn-3.28.3-6.el7.x86_64
ncurses-base-5.9-14.20130511.el7_4.noarch
ncurses-base-5.9-13.20130511.el7.noarch
bash-4.2.46-29.el7_4.x86_64
bash-4.2.46-28.el7.x86_64
NetworkManager-wifi-1.8.0-9.el7.x86_64
NetworkManager-wifi-1.8.0-11.el7_4.x86_64
kexec-tools-2.0.14-17.2.el7.x86_64
kexec-tools-2.0.14-17.el7.x86_64
NetworkManager-tui-1.8.0-11.el7_4.x86_64
NetworkManager-tui-1.8.0-9.el7.x86_64
openssh-7.4p1-11.el7.x86_64
openssh-7.4p1-13.el7_4.x86_64
grub2-tools-minimal-2.02-0.64.el7.centos.x86_64
grub2-tools-minimal-2.02-0.65.el7.centos.2.x86_64
NetworkManager-team-1.8.0-11.el7_4.x86_64
NetworkManager-team-1.8.0-9.el7.x86_64
systemd-sysv-219-42.el7_4.4.x86_64
systemd-sysv-219-42.el7.x86_64
binutils-2.25.1-32.base.el7_4.1.x86_64
binutils-2.25.1-31.base.el7.x86_64
selinux-policy-targeted-3.13.1-166.el7_4.5.noarch
selinux-policy-targeted-3.13.1-166.el7.noarch

From this list, you now need to take each one of the newer of the two duplicate packages and remove it from the rpm database while leaving it thinking that the older version is still installed. Now there is a --cleandupes option to package-cleanup but it does the opposite of the safe way to do this and removes the older version. The problem with this is that you have no idea if the newer version is correctly installed or not so removing the older version will indeed fix the duplicate but will not ensure that your system is in a consistent state.

You can remove the duplicates by using rpm -e --justdb --nodeps $newerpackage - using the example from above, that would be rpm -e --justdb --nodeps nss-tools-3.28.4-15.el7_4.x86_64 grub2-common-2.02-0.65.el7.centos.2.noarch wpa_supplicant-2.6-5.el7_4.1.x86_64 etc etc (don't blindly copy and paste that command, your package names and versions will be different).

5. Potential problems

Be aware that the newer package is not always listed second in the list - for example, selinux-policy-targeted-3.13.1-166.el7_4.5.noarch is listed first in my example but is newer than the other duplicate, selinux-policy-targeted-3.13.1-166.el7.noarch and NetworkManager-1.8.0-11.el7_4.x86_64 is newer than NetworkManager-1.8.0-9.el7.x86_64. This makes it non-trivial to automate this process so you are most likely stuck with running this list manually... sorry!

What might make it slightly easier is to pipe the output from package-cleanup to a file and then hack it around in your favorite editor to contain only the newer package names by removing the older ones from the list, e.g. package-cleanup --dupes --noplugins > pcd.txt then edit that, remove the older versions from the list and save it and then run something like for p in $(cat pcd.txt); do echo rpm -e --justdb --nodeps $p;done. Those paying attention will notice that this just echoed the commands it would have run to the terminal... you'll need to run that without the echo in there to actually do anything. It's your responsibility to make sure that what it is going to do is what you need it to do, please don't blame me if you get it wrong!

Now you see why it was important that you backed up /var/lib/yum and /var/lib/rpm before you started... you did back them up didn't you?