Sometimes docker is causing troubles,even stopped and all docker related packages are deleted from system. One example of such issue is
# rpm -qa | grep docker # #rm -rf docker/ rm: cannot remove ‘docker/devicemapper/mnt/c1b69563d2b817b729e875f50f9f5d29206d15f65d823c864c8444aa3c6030dd’: Device or resource busy rm: cannot remove ‘docker/containers/c1b69563d2b817b729e875f50f9f5d29206d15f65d823c864c8444aa3c6030dd/secrets’: Device or resource busy rm: cannot remove ‘docker/volumes/efd99751dce0cf97dd2a2f48ecc6ffa05d41b30938e08c9592b436bc3f858315/_data/secrets’: Device or resource busy rm: cannot remove ‘docker/volumes/efd99751dce0cf97dd2a2f48ecc6ffa05d41b30938e08c9592b436bc3f858315/_data/screen’: Device or resource busy
it is another topic why docker is not cleaning stuff once deleted packages! In this case, something is holding these files and lsof will not help you :). This long line
c1b69563d2b817b729e875f50f9f5d29206d15f65d823c864c8444aa3c6030dd reminds on docker container ID.Every process ( eg. nsenter, some command sent to inside container ) will get in /proc/$PID/mountinfo information about acquiring access to Mount Namespace.
In this case I did
# grep -l c1b69563 /proc/*/mountinfo /proc/8441/mountinfo /proc/8442/mountinfo root@ip-172-31-12-99: /var/lib # grep -l efd997 /proc/*/mountinfo /proc/8441/mountinfo /proc/8442/mountinfo
and from above we see that PIDs, 8441,8442 are still alive and holding pointers to files in /var/lib/docker.
# ps -f 8441 UID PID PPID C STIME TTY STAT TIME CMD root 8441 18680 0 04:16 pts/1 S 0:00 nsenter -m -u -n -i -p -t 19012
and killing process,
# kill -9 8441
released these and rm worked.
If you now think why I had to remove /var/lib/docker read changing docker storage backend you have to do that if you want to switch from / to loop lvm -> direct lvm -> overlay -> btrfs and visa versa.