Large backups to S3

Dan

Dan

Moderator
Hi there!

I previously used a script I wrote with the help of some others on the forum to run backups and then upload them to S3, recently I had the backup file for my own domain start failing to transfer to S3 because it has gotten fairly large (over 13GB). Sometimes it would transfer and other times it wouldn't.

A few years back cPanel added the ability to transfer backups to S3 as well however I knew it wouldn't work very well at the point so after my own backups started failing to transfer I decided to give cPanel's backup and transfer a try. Lo and behold it also fails to transfer large files. The first week it did 4 out of 13 backups and the next week it did 8 out of 13. With all of the failed files being 340MB+ although one did transfer that was 355MB. Looking at the transport log it never says that anything has failed it just gets to a point and stops.

So the question is this: what do you do for backing up large accounts and where do you transfer them to? How do you transfer them or do you even transfer them at all?

Any input is of course welcome regarding backup & transfer solutions!

Thanks!
 
I'm curious if there's any indication of a timeout in the transport logs? That's an issue we've seen when the cPanel "Maximum Destination Backup Timeout" isn't set high enough to accommodate the large backups.

Default is 2700 seconds, which may not be long enough for it to transport.
 
KH-JonathanKW said:
I'm curious if there's any indication of a timeout in the transport logs? That's an issue we've seen when the cPanel "Maximum Destination Backup Timeout" isn't set high enough to accommodate the large backups.

Default is 2700 seconds, which may not be long enough for it to transport.
Click to expand...
JonathanKW,

Nope, no timeout indicated anywhere. I'll attach a copy of my log here, the two added to the queue that only have three lines are the backups that were added and failed. However doesn't the destination timeout really mean the maximum time without any response back or is it actually maximum time to send the file to the desination?

Thanks!

PS: Odd, .log is not an allowed extension to upload on the forum?
 

Attachments

  • cpbackup_transporter.zip
    1.9 KB · Views: 9,179
Here's where those settings that JonathanKW mentioned are located:
1582222629874.png


If I'm reading it right the abrupt end between these two logs...
Code: 
[2020-02-15 21:39:56 -0800] info [cpbackup_transporter] Upload attempt #1 starting for /backup/weekly/2020-02-15/accounts/account2.tar.gz to /weekly/2020-02-15/accounts/account2.tar.gz for destination:  Destination
[2020-02-15 21:59:34 -0800] info [cpbackup_transporter] Initializing log file
...looks like a 20 minute time out occurred during uploading. (21:39 - 21:59) Maybe your settings is set to timeout after 1200 seconds somewhere.

This one though...
Code: 
[2020-02-15 21:59:37 -0800] info [cpbackup_transporter] Upload attempt #1 starting for /backup/weekly/2020-02-15/accounts/account.tar.gz to /weekly/2020-02-15/accounts/account.tar.gz for destination:  Destination
[2020-02-15 22:00:33 -0800] info [cpbackup_transporter] Initializing log file
...didn't even seem to start uploading and stopped after attempting for 1 minute. That would seem to me like it couldn't reach the server to start uploading.
 
Hi Josh,

The Maximum destination backup timeout was at the default setting, I think it was 2700 which is 45 minutes. I have now changed it to 6 hours.

The whole log from beginning to end is little more than 20 minutes, 21:39 to 22:02 so, yeah, certainly no where near the 45 minute timeout limit. If it couldn't reach S3 then you would think it would actually log that fact...not much of a log if it doesn't LOL

Thanks!

Dan
 
There's another timeout under the remote destination configuration too, you may want to check it as well.

To confirm on my original thought, I did some digging.

Older Dedicated KnownHost servers (pre-atlanta) were on a 100mbps port so a backup at 340GB was experiencing transport errors to a remote backup destination via FTP.

We had to extend the timeout to over 10+ hours to give it enough time to transfer, otherwise it stopped.

However, the errors were pretty clear when this happens..

==
Cpanel::LoggerAdapter::warn(Cpanel::LoggerAdapter=HASH(0x19e2538), "Upload attempt failed: Time out reached for upload attempt #2\x{a}") called at /usr/local/cpanel/Cpanel/Backup/Queue.pm line 641
==
 
Hi JonathanKW!

KH-JonathanKW said:
There's another timeout under the remote destination configuration too, you may want to check it as well.
Click to expand...
That timeout is set to 30 seconds and has a maximum of 300, it looks like it's just looking for a response from the remote server and is not a transfer timeout.
Timeout
The number of seconds that the system will wait for a response from the remote server.
Click to expand...

I'm not seeing any of the text from your log file snippet at all in my log. No 'LoggerAdapter' or 'warn' or anything similar to that. There are no failures or warnings in any log at all that I can find.

Thanks!
 
Backups ran on Saturday and the change that I made to Maximum Destination Backup Timeout made no difference at all. Still only 8 of the backups were transferred. Total elapsed time of the transporter log was less than 30 minutes.

Thanks for the help everyone but it looks like I should just switch back to my old script for the time being and maybe I'll look into Jetbackup.
 
Hey @Dan
It may be worth opening a ticket with us so we can get cPanel involved directly. They're not without bugs with their transfer scripts.
 
Yup, I concur with James. That was going to be my recommendation as well. If there is an issue with backups to S3 then it needs to be investigated and made aware so that it can be corrected.
 
You must log in or register to reply here.
Top