Your server was hacked or is under attack: what to do

If you suspect that someone has broken into your server or that the server is being attacked from outside, follow the steps below in order: secure your blackbox.md account, secure access to the server from the app.blackbox.md panel (the “My Servers” page, app.blackbox.md/my-servers, menu “Infrastructure” → “My Servers”), check the server and rebuild it from scratch. Run the commands on the server over SSH, as root (Connecting to your server over SSH (Windows, macOS, Linux)).

Available to: the holder of the blackbox.md account who ordered the server. The check commands: anyone with root SSH access to the server. On cards marked “Managed”, the panel buttons do not appear; for these servers, write to us directly.

The blackbox.md website is available in Romanian and Russian; this article quotes the Romanian labels with their English meaning.

Signs of a hacked server

Before you look for unfamiliar keys, identify the two keys that BLACKBOX puts on new servers: “Management key (blackbox.md)” and the AI agent's key, with the comment blackbox-agent-…. They are not signs of an intrusion: BLACKBOX keys on your server. Be careful: whoever adds a key chooses its comment, so an attacker can give their key the same name. If you see more than one blackbox-agent-… line, or if you are in doubt, remove them (step 2): the panel does not depend on the agent's key.

What to do first

If the server is attacking other systems or sending spam right now (a report from BLACKBOX, a growing email queue, heavy outgoing traffic):

1. In “My Servers”, on the server card, click “Stop” and confirm “Stop SERVER_NAME?”.
2. Write to support immediately (see “What to send to support” below).

While the server is stopped, billing continues, and “Reset password”, the “Installed keys” list and key removal do not work: they need the server to be running. Start it again with “Start” only when you are ready to do the steps below; when the server starts, the attacker's programs may start again too (Rebooting, stopping and starting a server).

Otherwise, do not stop the server: first do steps 1 and 2, while the server is running and responds over SSH.

Step 1. Secure your blackbox.md account

1. Change your account password: “Contul meu” (My account) → “Profil” (Profile) → the “Securitatea contului” (Account security) section → “Parolă” (Password) → “Schimbă parola” (Change password). Fill in “Parola curentă” (Current password) and “Parolă nouă” (New password), then click “Salvează parola” (Save password). The message “Parola a fost schimbată. Vă rugăm autentificați-vă din nou.” (Your password has been changed. Please sign in again.) appears, and you are signed out. All sessions on blackbox.md end, and all API tokens created before the change stop working (Changing your account password).
2. Sign in with your new password and open “Token-uri API” (API tokens). If the “Ultima utilizare” (Last used) column shows a recent date but you did not use the token yourself, the token has reached someone else. Click “Revocă” (Revoke) next to each old token (the list does not show that the password change has stopped them) and confirm “Revocați acest token? Orice client AI care îl folosește va pierde accesul imediat.” (Revoke this token? Any AI client that uses it will lose access immediately.) Create new tokens only for the AI clients you actually use (API tokens: creating, copying and revoking).
3. In “Chei SSH” (SSH keys), delete the keys you don't recognize with “Șterge” (Delete). Deleting a key here does not remove it from your servers (step 2).
4. In “Comenzi” (Orders) and “Sold” (Balance), check that there are no orders or payments that you did not make.
5. Open the panel with “Deschide panoul” (Open the panel) (or “Panou de administrare” (Admin panel) in the account menu). In the panel, go to “Settings” → “Sessions”, click “Sign Out Everywhere” and confirm “Sign out of every session for this account, including this one? You will need to sign in again.” All panel sessions end, and the devices remembered for 2FA are forgotten. Then open the panel again from your account (Panel settings: profile, password, and sessions).

Accounts created with Google or GitLab

The account has no password on blackbox.md, and the “Parolă” (Password) section does not appear. Change the password of your Google or GitLab account, turn on two-factor authentication there and end the active sessions. API tokens do not stop automatically: revoke them one by one with “Revocă” (Revoke). The other steps stay the same (Signing in with Google or GitLab).

Step 2. Secure access to the server from the panel

Work on the server card in “My Servers”. If someone else could also have used your account, repeat these steps for every server.

1. Click “Reset password” and confirm “Reset the password for SERVER_NAME? The current password stops working immediately.” The new root password appears on the card, in the “Username” and “Password” rows; keep it in a password manager. If reset_failed appears, the password has not changed: usually the panel could not connect to the server. See Resetting the root password and write to us.
2. Click “Add SSH key” and check every row in “Installed keys”. Keep “Management key (blackbox.md)” (it has no “Remove” button) and your own keys. For any other key you don't recognize, write down its fingerprint and comment, then click “Remove” and confirm “Remove key SHA256:… from this server?”. “Key removed.” appears.
3. Remove the AI agent's key (blackbox-agent-…) the same way if you don't use SSH access through AI, or if your API tokens could have reached someone else: an AI assistant with the “Vizualizează parolele/accesul serverelor” (View server passwords/access) permission can receive this key. If you still use SSH access through AI, request a new key with regenerate=true after the removal; without this parameter, you get the old key back (SSH access and SSH keys through the AI assistant).
4. If you don't have a key of your own on the server yet, add it from the same place (“Use a saved key” or “Paste a new key” → “Add key”). Connect with it in a new terminal window and leave that session open (Adding and removing SSH keys on the server).
5. Once your key works, click “Disable password login” and confirm. The card shows “Key-only” (Enabling and disabling SSH password login).

Important Do not delete the management key from authorized_keys (in the panel: “Management key (blackbox.md)”), and do not set PermitRootLogin no: either one blocks password resets and key management from the panel. Customers have no web or VNC console. Before you change SSH settings or the firewall, keep a second SSH session open; if you lose access, write to support.

Step 3. Check the server

The commands below only read information. Copy their output to your computer and write down what you find: processes, IP addresses, dates, file paths. Do not delete files or stop processes at random: attackers usually leave several ways back in, and whatever you delete can no longer be analyzed.

Ubuntu and Debian

who
last -n 30
lastb -n 30
journalctl -u ssh --since "7 days ago" | grep -E "Accepted|Failed password|Invalid user"
ls -la /var/spool/cron/crontabs

AlmaLinux and CentOS Stream

who
last -n 30
lastb -n 30
journalctl -u sshd --since "7 days ago" | grep -E "Accepted|Failed password|Invalid user"
ls -la /var/spool/cron

who shows who is connected right now, last shows successful logins, and lastb shows failed attempts (many failed attempts are normal). In the log, Accepted lines are successful logins, with the method (password or publickey) and the IP address; for publickey, the end of the line also shows the fingerprint of the key that was used (SHA256:…), which you can compare with the “Installed keys” list. If last is missing, use only journalctl. The last command shows which users have cron jobs: each file is named after a user.

Processes and connections

ps aux --sort=-%cpu | head -n 15
ls -l /proc/PID/exe
ss -tunap
ls -la /tmp /var/tmp /dev/shm

Replace PID with the number in the second column of a suspicious process: the exe link shows the real program. A path in /tmp or /dev/shm, or the note (deleted), is a bad sign. In the ss output, look for ESTAB connections to unknown addresses and ports, and check the program name in users:((…)).

Scheduled tasks, services, users and keys

crontab -l
cat /etc/crontab
ls -la /etc/cron.*
systemctl list-timers --all
systemctl list-units --type=service --state=running
awk -F: '$3 == 0 || $3 >= 1000 {print $1, $3, $6, $7}' /etc/passwd
getent group sudo wheel
cat /root/.ssh/authorized_keys
ls -la /home/*/.ssh/authorized_keys

Only root should have UID 0. Accounts with a UID of 1000 or higher are regular user accounts (nobody, with UID 65534, is normal): check that you know all of them, and the members of the sudo and wheel groups too. In cron and in services, lines that download and run scripts (curl, wget) are suspicious.

Site files

find /var/www -type f -mtime -7 -printf '%TY-%Tm-%Td %TH:%TM %p\n' | sort | tail -n 50
find /var/www -path "*uploads*" -name "*.php"

Replace /var/www with your site's directory if it is different; to cover a longer period, increase 7 (days). PHP files in upload directories (for example wp-content/uploads in WordPress) are highly suspicious: check each one.

Step 4. Rebuild the server from scratch

Risk A compromised server can no longer be considered safe, even if you deleted everything you found: keys, users, scheduled tasks or modified programs that you did not notice may still be there. The safe path is a server installed from scratch.

The panel has no button for reinstalling the operating system. The steps:

1. Order a new server (Placing an order step by step). In the “Cheie SSH pentru server” (SSH key for the server) step, attach your key and leave the “Dezactivează autentificarea prin parolă (doar cheie SSH)” (Disable password authentication (SSH key only)) box checked: the server will then accept only key authentication over SSH (The SSH key at order time and “key-only” servers).
2. Secure it and install packages from the distribution's official sources (Securing a new server: checklist).
3. Install the application from its official source: the CMS, modules and themes from the vendor, your own code from your git repository. Do not copy programs, scripts, cron jobs or configurations from the old server.
4. Restore only the data (databases, uploaded files, content) from a backup made before the compromise. BLACKBOX does not make automatic copies of the server; if you have no older backup, copy the data from the old server and check it first (Backup strategy for your server, Moving a site to a BLACKBOX server).
5. Change all of the application's secrets: database passwords, API keys, SMTP credentials, secret keys in the configuration, site administrator passwords. Also replace any private keys kept on the old server (for example, for deployment).
6. Point the domain to the new server (How to point your domain to the server). If you have connected Cloudflare or GoDaddy, you can do this from the panel's “DNS” page (Managing DNS records from the panel).
7. Copy whatever you still need from the old server, including the logs, then delete it with “Terminate”. The deletion runs after a 24-hour grace period and does not refund the period that has already started; until then, the server is billed. How this compares with canceling the order: Permanently deleting a server.

If you don't want a new server, ask support whether the operating system can be reinstalled on the same server. Reinstalling erases all data on the server, so save it first; then follow steps 2–5 of the list above.

Step 5. Close the way in

Before you put the new server into service, remove the cause, or the attack will happen again:

• Outdated CMS, modules and themes. Update them, delete the ones you don't use and turn on automatic security updates (System updates and automatic security updates).
• Weak or reused passwords. Use unique passwords generated by a password manager for root, site administration and databases; use keys for SSH.
• Services reachable from the internet. Bind databases to 127.0.0.1 and restrict access to admin panels (for example, phpMyAdmin). The network filter managed by BLACKBOX does not replace these measures (Firewall: network filtering and ufw/firewalld on the server).
• The panel. Turn on two-factor authentication in the panel (Two-factor authentication (2FA) in the panel).
• BLACKBOX access. For root, use PermitRootLogin prohibit-password, not PermitRootLogin no, and do not delete “Management key (blackbox.md)”.

Then go through the whole list: Securing a new server: checklist.

Your responsibility

• Under chapter 3 of the Terms and Conditions, you are fully responsible for your own activity and for the activity of everyone you give access to the services. You are also responsible for protecting passwords, SSH keys and API tokens. BLACKBOX is not liable for third parties' use of your accounts unless there is a proven breach of BLACKBOX's systems.
• Chapter 2 of the Terms prohibits, among other things, spam and taking part, “voluntar sau prin neglijență gravă” (voluntarily or through gross negligence), in DoS and DDoS attacks (Acceptable use policy: what is not allowed).
• When a violation is detected, for example through a report from outside, BLACKBOX may suspend or terminate the services, fully or partly, with or without prior notice, depending on how serious it is. Reply to support quickly and tell them what you have done.
• If the server held personal data of your users (for example, the customers of an online store), you are the controller of that data and responsible for complying with the law (Terms, chapter 8). Assess whether you need to notify the National Center for Personal Data Protection and the people affected (Articles 33–34 of Law No. 195/2024). BLACKBOX, as the processor acting on behalf of the controller, gives you reasonable assistance to the extent technically possible at the level of the hosting infrastructure: Data Processing Agreement (DPA) for companies.

Attack from outside: the server is not hacked

If in step 3 you found no unfamiliar successful logins and no unknown processes or keys, the server is probably just the target of an attack from outside.

SSH password-guessing attempts

Thousands of Failed password or Invalid user lines in the log and a long lastb list are normal for any server with a public IP address: bots scan the internet constantly. Protect the server with fail2ban (Protection against password-guessing attempts with fail2ban) and with key-only login (“Disable password login”, step 2).

Excessive HTTP traffic to the site

1. Put the domain behind the Cloudflare proxy (the orange cloud). If you have connected Cloudflare in the panel, check “Proxied” on the site's A, AAAA and CNAME records on the “DNS” page (Managing DNS records from the panel); otherwise, turn on the proxy directly in your Cloudflare account. Web traffic then goes through Cloudflare, which can filter requests.

2. Limit the number of requests per IP address in nginx. Create the file /etc/nginx/conf.d/ratelimit.conf:

   limit_req_zone $binary_remote_addr zone=perip:10m rate=10r/s;
   limit_req_status 429;

   Add this to the site's server { } block:

   limit_req zone=perip burst=20 nodelay;

   Check the configuration and reload nginx:

   nginx -t
   systemctl reload nginx

   Each IP address can make 10 requests per second, with a burst allowance of 20; extra requests get error 429. Adjust the values to your site. Behind Cloudflare, nginx sees Cloudflare's addresses: first set up nginx to see your visitors' real addresses (Cloudflare documentation).

3. If the attack continues or comes directly to the server's IP address, write to support.

What to send to support

Write to support@blackbox.md, open a ticket from “Contul meu” (My account) → “Ajutor și suport” (Help and support) (Opening a support ticket) or, for emergencies, call +373 78 003948, Monday–Friday, 09:00–18:00 (Moldova time). Include:

• the server's IP address;
• the order number (BBX-…);
• when you noticed the problem (date, time and time zone);
• what you have already done (server stopped, password reset, keys removed, new server ordered);
• what you found in step 3, as text: processes, IP addresses, log lines, fingerprints of the keys you removed.

Security Never send passwords, private keys or API tokens to anyone, not even to BLACKBOX support. A key's fingerprint and the public key are safe to send.

Frequently asked questions

Can I clean up the server without reinstalling it? You can try, but you won't know for sure that you found everything. For a server with a site or important data, we recommend a new server (step 4).

Can BLACKBOX clean up the server for me? BLACKBOX does not access or read the content of your application, except for technical support at your explicit request or where the law requires us to (Data Processing Agreement (DPA) for companies). For help with the cleanup or with moving to a new server, ask support: Professional services on request: administration, DevOps, migration.

I received an abuse report, but I can't find anything on the server. What do I do? Reply to us anyway, as soon as possible: tell us what you checked in step 3 and what you found. Sometimes the source is an application on the server, for example a contact form that is being used to send spam.

Can I get a different IP address? The IP address cannot be changed from the panel. A newly ordered server gets its own IP address; for other options, write to us.