docker complains about : cannot remove …. device or resource busy

Sometimes docker is causing troubles,even stopped and all docker related packages are deleted from system. One example of such issue is

# rpm -qa | grep docker
#
#rm -rf docker/
rm: cannot remove ‘docker/devicemapper/mnt/c1b69563d2b817b729e875f50f9f5d29206d15f65d823c864c8444aa3c6030dd’: Device or resource busy
rm: cannot remove ‘docker/containers/c1b69563d2b817b729e875f50f9f5d29206d15f65d823c864c8444aa3c6030dd/secrets’: Device or resource busy
rm: cannot remove ‘docker/volumes/efd99751dce0cf97dd2a2f48ecc6ffa05d41b30938e08c9592b436bc3f858315/_data/secrets’: Device or resource busy
rm: cannot remove ‘docker/volumes/efd99751dce0cf97dd2a2f48ecc6ffa05d41b30938e08c9592b436bc3f858315/_data/screen’: Device or resource busy

it is another topic why docker is not cleaning stuff once deleted packages! In this case, something is holding these files and lsof will not help you :). This long line c1b69563d2b817b729e875f50f9f5d29206d15f65d823c864c8444aa3c6030dd reminds on docker container ID.Every process ( eg. nsenter, some command sent to inside container ) will get in /proc/$PID/mountinfo information about acquiring access to Mount Namespace.
In this case I did

# grep -l c1b69563 /proc/*/mountinfo
/proc/8441/mountinfo
/proc/8442/mountinfo
root@ip-172-31-12-99: /var/lib # grep -l efd997 /proc/*/mountinfo
/proc/8441/mountinfo
/proc/8442/mountinfo

and from above we see that PIDs, 8441,8442 are still alive and holding pointers to files in /var/lib/docker.

# ps -f 8441
UID        PID  PPID  C STIME TTY      STAT   TIME CMD
root      8441 18680  0 04:16 pts/1    S      0:00 nsenter -m -u -n -i -p -t 19012

and killing process,

# kill -9 8441

released these and rm worked.

If you now think why I had to remove /var/lib/docker read changing docker storage backend you have to do that if you want to switch from / to loop lvm -> direct lvm -> overlay -> btrfs and visa versa.

Advertisements

#docker, #docker-storage, #docker-storage-setup, #linux, #nsenter