The server's disk is full

When the server's disk fills up, databases stop, websites return errors and files can no longer be saved. You run the commands on the server, over SSH, as root (Connecting to your server over SSH (Windows, macOS, Linux)). If the server has the “Monitoring 24/7” service, you can also see how disk usage changes over time in the app.blackbox.md panel, on the “Monitoring” page (app.blackbox.md/monitoring, sidebar → “Infrastructure” → “Monitoring”).

Available to: anyone with root access over SSH to a BLACKBOX server. The charts on the “Monitoring” page: servers with the “Monitoring 24/7” service active. Upgrades from the panel: the server's owner.

The blackbox.md website is available in Romanian and Russian; this article quotes the Romanian labels with their English meaning.

Signs of a full disk

Step 1. Check space and inodes

df -h
df -i

• df -h: if the Use% column shows 100% (or close to it) for / or for another partition, the disk is full. Continue with steps 2–5.
• df -i: if the IUse% column shows 100%, no new files can be created, even if there is still free space. Go to the “Out of inodes” section below.

Step 2. Free up space quickly, with no risk

The system journal: the first command shows how much space it takes up, and the second deletes its archived files until no more than 200 MB remain.

journalctl --disk-usage
journalctl --vacuum-size=200M

The cache of downloaded packages:

Ubuntu and Debian

apt clean
apt autoremove --purge

The second command removes old kernels and packages that nothing depends on anymore. Read the list before you confirm.

AlmaLinux and CentOS Stream

dnf clean all

Step 3. Find what is taking up space

List the folders on the / partition by size (the largest are at the end; the last line is the total for the partition), then repeat the command for the largest folder, for example /var:

du -xh --max-depth=1 / 2>/dev/null | sort -h
du -xh --max-depth=1 /var 2>/dev/null | sort -h

Files larger than 500 MB:

find / -xdev -type f -size +500M -exec ls -lh {} + 2>/dev/null

The ncdu program is easier to use (move around with the arrow keys, quit with q). Install it after step 2, once the disk has a little free space again. On Ubuntu and Debian:

apt update
apt install -y ncdu

On AlmaLinux and CentOS Stream, the package comes from the EPEL repository:

dnf install -y epel-release
dnf install -y ncdu

Then start it:

ncdu -x /

Warning Do not delete database files (/var/lib/mysql, /var/lib/postgresql, /var/lib/pgsql) or system folders (/boot, /etc, /usr, /var/lib/dpkg, /var/lib/rpm) by hand. Clean up the data in /var/lib/docker only with docker commands; the only exception is emptying container logs, described below. Anything you delete without a backup cannot be recovered.

Step 4. Free up space

Large logs

Do not delete a log with rm while a program is still writing to it: the space does not come back until you restart the program. Empty it instead (replace FILE_PATH with the path you found in step 3):

truncate -s 0 FILE_PATH

Compressed archives of rotated logs that are older than 7 days:

find /var/log -type f -name "*.gz" -mtime +7 -delete

Docker

See how much space images, containers and volumes take up, then delete the images that no container uses and the build cache. The last two commands ask for confirmation before they delete anything.

docker system df
docker image prune -a
docker builder prune

Container logs have no size limit unless you have set one. Show the 5 largest logs, then empty the one you choose (replace FILE_PATH with its path from the list):

du -h /var/lib/docker/containers/*/*-json.log | sort -h | tail -n 5
truncate -s 0 FILE_PATH

Then limit them: Installing Docker and Docker Compose, section “Limit container logs”.

Risk Do not run docker volume prune, and do not add --volumes to docker system prune. The volumes of a project stopped with docker compose down count as unused, and their data (for example, the database) is lost for good.

Forgotten copies and archives

Database exports, website archives and copies made “just in case” often stay on the server. Copy them to your computer or to another server (Transferring files to the server (scp, rsync, SFTP)), check the copy, then delete them from the server:

rm FILE_PATH

Deleted files that are still open

If df -h shows the same usage after you delete files, a program is still holding them open. lsof +L1 lists them: the COMMAND column shows the program, and SIZE/OFF shows the size. Restart the service the program belongs to (replace SERVICE_NAME, for example with nginx):

lsof +L1
systemctl restart SERVICE_NAME

If lsof is missing, install it. On Ubuntu and Debian:

apt install -y lsof

On AlmaLinux and CentOS Stream:

dnf install -y lsof

Step 5. Restart services and check

df -h /
systemctl --failed

df -h / must show free space. systemctl --failed shows the services that have failed, for example mariadb, mysql, postgresql, nginx or php8.3-fpm. Restart each service in the list (replace SERVICE_NAME with its name), then open the website:

systemctl restart SERVICE_NAME

Keep some space in reserve: on the “Monitoring” page, the bar in the “Disk” column turns yellow from 80%.

Out of inodes

df -i shows IUse% at 100%, even though df -h still shows free space: there are too many small files on the disk. The usual causes are PHP sessions, application cache, the email queue and temporary files. Find the folders with the most files (the number on the left):

du --inodes -x --max-depth=3 / 2>/dev/null | sort -n | tail -n 20

If the folder you found contains sessions or temporary files, delete the old ones, for example those older than 7 days (replace FOLDER_PATH with the folder's path). Do not use this command in folders that hold website data (for example, files uploaded by users) or database data.

find FOLDER_PATH -type f -mtime +7 -delete

• PHP sessions are in /var/lib/php/sessions (Ubuntu, Debian) or /var/lib/php/session (AlmaLinux, CentOS Stream).

• Application cache: use the application's own cache-clearing command, if it has one.

• Email queue: if you use Postfix, the command below shows on its last line how many messages are waiting. Thousands of messages that you did not send are a sign of a compromised website: Your server was hacked or is under attack: what to do.

  postqueue -p | tail -n 1

How to prevent the problem

• Log rotation. logrotate already rotates the logs in /var/log from the system and from packages installed from the repository. For your application's logs in another folder, create the file /etc/logrotate.d/APP_NAME with the content below:

  /var/www/APP_NAME/logs/*.log {
      weekly
      rotate 4
      compress
      missingok
      notifempty
      copytruncate
  }

  Then check it (the command only simulates the rotation, without changing anything):

  logrotate -d /etc/logrotate.d/APP_NAME

• System journal limit. Keep the journal to 200 MB at most:

  mkdir -p /etc/systemd/journald.conf.d
  printf '[Journal]\nSystemMaxUse=200M\n' > /etc/systemd/journald.conf.d/size.conf
  systemctl restart systemd-journald

• Docker. Set the container log limit in /etc/docker/daemon.json and check docker system df from time to time: Installing Docker and Docker Compose.

• Backups off the server. BLACKBOX does not make automatic backups of your server, and the “Backups” button on the server card is not active. Send your backups to another server and do not keep old exports on the same disk: Backup strategy for your server.

• Keeping an eye on the disk. With the “Monitoring 24/7” service, you see the “Disk” column and the “Disk usage (%)” chart (Add-on services for your server). Without it, check df -h regularly.

A larger disk

The “Upgrade” button on the server card moves the server to a larger plan in the same family. The amount is taken from your balance, and the server reboots once: Server upgrade: moving to a larger plan.

• On CPU and High-Frequency Compute, an upgrade adds no disk space. Order a new server from a family with a larger disk (VPS plans: how to choose), move your data, then delete the old server.
• Free up a little space before the upgrade, so that services start normally after the reboot and the file system can be extended. When it's done, check df -h /; if it still shows the old size, write to us.
• For “VPS hosting” plans and dedicated servers, write to us: How to contact BLACKBOX support.

Frequently asked questions

I deleted large files, but df -h shows the same usage. Why? A program is still holding the files open. Find it with lsof +L1 and restart its service (step 4).

df -h shows free space, but I still get No space left on device. Why? Check df -i: the inodes may have run out. Also look at the other lines in df -h: another partition may be full, for example /boot or /tmp.

/var/lib/mysql takes up a lot of space, with many binlog.000… files. Can I delete them? Not with rm. They are MySQL binary logs, which MySQL 8 turns on by default and keeps for 30 days. If you don't use replication, delete the old ones from within MySQL:

mysql -e "PURGE BINARY LOGS BEFORE NOW() - INTERVAL 3 DAY;"

Will rebooting the server help? It frees only the space held by deleted files that are still open. Everything else stays in use, and with a full disk the database may not start. Free up space first, then reboot if needed: Rebooting, stopping and starting a server.

I can't connect over SSH. What should I do? The panel has no web console. Follow I can't connect over SSH; if that doesn't work, write to us with the server's IP address.

Can I ask the AI assistant to find what is filling the disk? Yes, if the assistant's API token has the “Rulează orice comandă ca root pe serverele mele” (Run any command as root on my servers) permission. Read each command before you approve it, especially the ones that delete files: Running commands on the server through AI.

Can the BLACKBOX team clean up the disk for me? You manage the server yourself, and support guides you. If you want the team to do the work, ask for a separate quote for server administration: Professional services on request: administration, DevOps, migration.