Troubleshooting Data Transfers

There is nothing worse than sitting around waiting for data to finish transferring across the network, that is if you are not paid hourly. For the rest of us, waiting eight hours to move 10 GB of data is crazy in this day and age of standard 1 Gbps bandwidth. The trouble is what piece of the entire transport process is to blame for the slow speed?

The first thing everyone wants to blame is the network, I am guilty of this myself. However, when you start testing the many components that work to transfer a file from point A to point B you find there are many factors to this seemingly simple task. Network Administrators know this pain best, as their networks are typically blamed first. They are put into a position where the burden of proof is on them and have to show the problem is not coming from the network. Being on the receiving end of this has encouraged me to put together a list of the components that can be the cause of poor data transfer performance.

How to Start Troubleshooting Data Transfers

Here are some questions to ask yourself when beginning to troubleshoot data transfer problems.

Storage Hardware Limitations

How fast can I transfer a file from the local disk to another local disk? Some modern networks have the ability to move data faster than the storage media the data is held on can be read from. If your server is using Hard Disk Drives (HDD) and not Solid State Drives(SSD) this may be the cause of poor transfer speeds.

Benchmarking Disk Performance

Understanding Disk Performance

Number Of Files To Process

Are you transferring a few large files or lots of small files? Moving a single 1GB file will be faster than moving 1024 x 1 MB files. This is due to the work the CPU has to do to process each file through the transfer process.

File Size Transfer Lab

Transfer Protocol & Settings

What protocol or software are you using to initiate the transfer? This is the thing I see overlooked the most. Some protocols handle transfers quickly and efficiently, while others do not. More importantly what settings are you using with the protocol?

The Network

Lastly, if you are sure everything running up to your network is good, then it’s time to test the network. There are a few things to keep in mind here. For example, are you passing through a VPN? VPNs will add a lot of overhead to a data transfer, as the data will be encrypted. The Internet is made up of many smaller networks, so there are a lot of places a slowdown can happen. You also need to be mindful of what direction the data is flowing. Are you uploading from your home to a server or downloading from a server to your home?

Benchmarking Network Performance

Wrapping Up Troubleshooting Data Transfers

This is not an exhaustive list, just the major points I see most often that cause issues. It is easy to forget or take for granted all the moving parts of simply transferring data from one location to another. In one extreme outlier, the performance issue came down to one badly punched-down Cat6 block. It is not the first thing most would think to check, yet changing the cable resolved the issue.

The next time you run into data transfer problems, start by walking through every step a single byte of data takes to be moved from one location to another. The issue may be somewhere you would never think to look.