Survive being hacked by using write-only backups
One of the functions of backups is to be able to restore the system and data in the case of a successful hacking attack. Without write-only backups, if the servers have access to the backups, then the hacker who has access to the servers has access to the backups too.
It is generally understood that backups which don't happen automatically get forgotten at some point. Without automatic backups, when a restore is required, the team realizes that a backup hasn't happened recently, and the backups can't recover current or near-current data. Therefore, automated jobs are set up on the servers to perform backups.
The problem becomes one of security. If the backups are automatic, the keys to access the backup are stored on the server. If a hacker compromises the server, they also have access to the backup, and can delete it.
The solution, which I recommend to all of my customers, is to use a backup environment which can be accessed by two credentials:
One credential which can only create new backups (but which cannot delete or alter existing backups)
A different credential for reading the backups
The key for reading is given to the admins (ideally this key is written down on paper only), and the key which can only create additional backups is stored on the server and used by the automated backup process on the server.
I have had good experiences over the last two years with https://www.tarsnap.com/. It supports multiple credentials, including credentials which can only create new backups. In addition:
The data is encrypted meaning that tarsnap themselves cannot read the backups (nor a hacker who successfully penetrates tarsnap's infrastructure).
The client is open-source so it can be verified that indeed only encrypted contents are sent to the tarsnap servers.
Tarsnap sends only deltas to the server, meaning that e.g. full database backups may be backed up each time, without worry that too much data need be stored at tarsnap or too much data may be sent.
It is in a different hosting environment to the application software, thus successful access by the hacker to the data center administration system will not enable deleting of the backups. This is not just a theoretical possibility, for example https://threatpost.com/hacker-puts-hosting-service-code-spaces-out-of-business/
Because the backup is off-site, the data must be transferred over the internet. In the case of a restore, it must be understood that the restore might take some time. (That is why backups within the data center are performed as well, for other scenarios, such as non-malicious operations accidents, or destructive bugs.) If all data within the data center has been deleted by a hacker, having a slow backup is preferable to having no backup at all.
The term "write-only backup" is the standard term but it isn't really accurate, a more accurate term would be "backup, with many keys, including a write-only key".
I have no affiliation with tarsnap and I have used no affiliate links in this article.