amazon s3 performance
I've been using Amazon S3 for backups. Yesterday I started a really big backup which I expect to be transferring data for hours or possibly days. I noticed something odd in my server activity graph, though:
The green data is outgoing. The baseline is visible on the left at about 2.2 Mbits/s, everything above that is S3 activity. After transferring data continuously for a few hours at full speed (7.4 Mbits/s in this case), it appears as though my transfer rate decreased steadily over time. What caused this? Is this something local on my server? Is this Amazon S3 throttling my transfers?
Before I jump to conclusions, I should note that since approximately the point where the transfer rate started decreasing, duplicity (the backup program I'm using) has been processing a really large single file (8.3 GB). Duplicity is currently using up quite a chunk of my CPU time, so this slowdown could certainly be related to an inefficiency in processing large data files. I guess I'll know more once this file is done.
Update: It looks like my guess was right. After finishing processing that huge file, the transfer rate went back up to somewhere near normal. Looks like purely a local issue.
The green data is outgoing. The baseline is visible on the left at about 2.2 Mbits/s, everything above that is S3 activity. After transferring data continuously for a few hours at full speed (7.4 Mbits/s in this case), it appears as though my transfer rate decreased steadily over time. What caused this? Is this something local on my server? Is this Amazon S3 throttling my transfers?
Before I jump to conclusions, I should note that since approximately the point where the transfer rate started decreasing, duplicity (the backup program I'm using) has been processing a really large single file (8.3 GB). Duplicity is currently using up quite a chunk of my CPU time, so this slowdown could certainly be related to an inefficiency in processing large data files. I guess I'll know more once this file is done.
Update: It looks like my guess was right. After finishing processing that huge file, the transfer rate went back up to somewhere near normal. Looks like purely a local issue.
Is the file sparse? rsync does the same, cpu usage goes up, bandwidth goes down.
![[info]](http://stat.livejournal.com/img/userinfo.gif){kind=small}
ghewgill
: Re: sparse files?2007-11-18T02:31:23Z
It's not a sparse file, it's a big tarball of a previous home directory backup. Since duplicity uses librsync, it makes sense that the same behaviour carries over from rsync. Perhaps it's just not limited to sparse files. :)
2007-11-18T00:40:19Z