由于問(wèn)題發(fā)生的比較突然，業(yè)務(wù)催的比較急，所以沒(méi)來(lái)得及截圖案發(fā)現(xiàn)場(chǎng)，所以現(xiàn)場(chǎng)截圖為后期正常的環(huán)境。

周一上班，照例進(jìn)入服務(wù)器，進(jìn)入docker容器

>>> docker exec -i -t xxx /bin/bashDocker error : no space left on device沒(méi)空間了？

立馬想到查看系統(tǒng)的空間

>>> df -h

Filesystem Size Used Avail Use% Mounted ondevtmpfs 3.9G 0 3.9G 0% /devtmpfs 3.9G 0 3.9G 0% /dev/shmtmpfs 3.9G 984K 3.9G 1% /runtmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup/dev/vda1 50G 50G 0G 100% /overlay 50G 5.9G 41G 13% /var/lib/docker/overlay2/4d0941b78fa413f3b77111735e06045b41351748bcea7964205bcfbf9d4ec0b6/mergedoverlay 50G 5.9G 41G 13% /var/lib/docker/overlay2/54a7a7b66d9c12d7e42158d177a6b67321f7da1f223b301e838e9bc109a2bda4/mergedshm 64M 0 64M 0% /var/lib/docker/containers/712f65baea85d898f6c948e7149f84f2f5eaf3b1934540603f32ab278f9acec4/mounts/shmshm 64M 0 64M 0% /var/lib/docker/containers/a5fa673086c8f46ba98fc4425b353ed2e12de3277a5fe2dc5b8b7affa63b4518/mounts/shmoverlay 50G 5.9G 41G 13% /var/lib/docker/overlay2/7fbbc8a29119a1eaa1f212c50b75405a1f16fd68e3ae3949cc0c963d0727a9ab/mergedshm 64M 0 64M 0% /var/lib/docker/containers/011a83deceacecbacb4ef7eb06eb5b812babf9e83914a4fb33d4925cc1ad375b/mounts/shmtmpfs 783M 0 783M 0% /run/user/0原來(lái)是根目錄滿了

進(jìn)一步查找

>>> cd />>> du -h --max-depth=1

984K ./run16K ./opt13M ./root4.0K ./mediadu: cannot access ‘./proc/4382/task/4382/fd/4’: No such file or directorydu: cannot access ‘./proc/4382/task/4382/fdinfo/4’: No such file or directorydu: cannot access ‘./proc/4382/fd/3’: No such file or directorydu: cannot access ‘./proc/4382/fdinfo/3’: No such file or directory0 ./proc204M ./boot12K ./redis39M ./etc16K ./lost+found4.0K ./srv0 ./sys47G ./var2.8G ./usr4.0K ./mnt36K ./tmp0 ./dev4.0K ./home50G .

>>> cd var>>> du -h --max-depth=1

116M ./cache8.0K ./empty4.0K ./games4.0K ./opt24K ./db46G ./lib4.0K ./gopher4.0K ./adm4.0K ./crash12K ./kerberos4.0K ./preserve4.0K ./nis16K ./tmp4.0K ./yp4.0K ./local104K ./spool374M ./log47G .

>>> cd lib>>> du -h --max-depth=1

76K ./systemd24K ./NetworkManager4.0K ./tuned4.0K ./games248K ./cloud215M ./rpm8.0K ./plymouth46G ./docker248K ./containerd4.0K ./dbus4.0K ./initramfs4.0K ./os-prober8.0K ./rsyslog24K ./alternatives8.0K ./authconfig12K ./stateless4.0K ./misc4.0K ./ntp8.0K ./dhclient4.0K ./selinux8.0K ./chrony4.0K ./rpm-state12M ./yum8.0K ./postfix28K ./polkit-14.0K ./machines8.0K ./logrotate46G .

>>> cd docker>>> du -h --max-depth=1

84K ./network108K ./buildkit4.0K ./trust4.0K ./runtimes5.7M ./image42G ./volumes24K ./plugins4.0K ./tmp20K ./builder180K ./containers3.3G ./overlay24.0K ./swarm46G .

>>> cd volume>>> du -h --max-depth=1

172K ./kudu_to_jdy_kudu_to_jdy42G ./jdy_extensions_logs748K ./bot_etl_bot_etl42G .

原來(lái)是jdy_extensions_logs這個(gè)volume爆滿導(dǎo)致的，后面就是找代碼bug的過(guò)程了，略

其實(shí)docker提供了很多命令來(lái)對(duì)容器鏡像進(jìn)行管理，其中的docker system df就是一個(gè)比較有用的命令

# 顯示docker文件系統(tǒng)使用情況>>> docker system df

TYPE TOTAL ACTIVE SIZE RECLAIMABLEImages 7 3 2.146GB 693.2MB (32%)Containers 3 3 127.5kB 0B (0%)Local Volumes 3 3 1.511MB 0B (0%)Build Cache 0 0 0B 0B

# 顯示空間使用情況>>> docker system df -v

Images space usage:REPOSITORY TAG IMAGE ID CREATED SIZE SHARED SIZE UNIQUE SIZE CONTAINERSjdy_extension 1.7 e3be3be9664a 15 hours ago 627.6MB 73.86MB 553.7MB 1<none> <none> 4b5825747ae9 19 hours ago 74.02MB 73.86MB 158.8kB 0redis 6.2.1 f877e80bb9ef 2 weeks ago 105.3MB 0B 105.3MB 0kudu_to_jdy 2.9 888b72288bca 2 weeks ago 538.1MB 73.86MB 464.3MB 1jdy_to_db 1.9 c345c4e15c1a 7 months ago 587.7MB 73.86MB 513.9MB 0bot_etl 2.3 020d41691ec7 7 months ago 508.6MB 73.86MB 434.8MB 1ubuntu 20.04 adafef2e596e 8 months ago 73.86MB 73.86MB 0B 0Containers space usage:CONTAINER ID IMAGE COMMAND LOCAL VOLUMES SIZE CREATED STATUS NAMES011a83deceac jdy_extension:1.7 'supervisord -n -c /…' 1 17.8kB 15 hours ago Up 15 hours jdy_extension_1.7712f65baea85 kudu_to_jdy:2.9 'supervisord -n -c /…' 1 36.9kB 2 weeks ago Up 45 hours kudu_to_jdy_2.9a5fa673086c8 bot_etl:2.3 'supervisord -n -c /…' 1 72.8kB 7 months ago Up 45 hours bot_etl_2.3Local Volumes space usage:VOLUME NAME LINKS SIZEjdy_extensions_logs 1 658.5kBkudu_to_jdy_kudu_to_jdy 1 128kBbot_etl_bot_etl 1 724kBBuild cache usage: 0BCACHE ID CACHE TYPE SIZE CREATED LAST USED USAGE SHARED

# 刪除所有沒(méi)有在用的volume>>> docker volume rm $(docker volume ls -q)

補(bǔ)充：記一次docker磁盤占用100%情況

在執(zhí)行g(shù)it pull origin master的時(shí)候，拋出如下錯(cuò)誤：

error: RPC failed; HTTP 500 curl 22 The requested URL returned error: 500

fatal: the remote end hung up unexpectedly

百度之后好多說(shuō)是nginx文件大小限制，但是查看配置文件，大小限制已經(jīng)設(shè)置到了500M，應(yīng)該不是該原因造成的：

server{ listen 80; server_name localhost; client_max_body_size 500M; location / { proxy_redirect off; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_pass http://localhost:180; }}

然后使用df -h查看磁盤情況，發(fā)現(xiàn)data2磁盤已經(jīng)100%（docker存儲(chǔ)在這里）：

使用docker system prune -a釋放了很多空間，問(wèn)題解決。

注意：使用上述命令會(huì)刪除一下內(nèi)容

WARNING! This will remove: - all stopped containers - all networks not used by at least one container - all images without at least one container associated to them - all build cacheAre you sure you want to continue? [y/N] y

以上為個(gè)人經(jīng)驗(yàn)，希望能給大家一個(gè)參考，也希望大家多多支持好吧啦網(wǎng)。如有錯(cuò)誤或未考慮完全的地方，望不吝賜教。