Globus is widely used in the research ecosystem for reliable, secure, and high-performance data transfer. The service has two key aspects: an orchestration layer that manages these transfers, and GridFTP, a highly performant protocol that moves the bits on the wire between source and destination storage systems. The GridFTP protocol, developed over 25 years ago now, was designed and built to manage ever increasing scientific data volumes across highly distributed, yet connected sets of systems. In this blog post, we reflect on the evolution of GridFTP, and the drivers of a system designed to serve as the workhorse for scientific data movement.
In the late 1990s significant hardware resources were coming online that made grid computing a necessary reality. Important scientific instruments were created in geographically distributed areas. The best example of this is the Large Hadron Collider at CERN, but there were many others like the Advanced Photon Source at Argonne National Lab. These instruments were crucial (and still are today) to a wide swath of scientific research across the globe. They produced immense amounts of data, on the order of multiple petabytes per month. There were also powerful data centers at various laboratories and other locations capable of processing teraflops. Additionally, graphics centers where the processed data could be visualized existed in limited locations, e.g.,the CAVE at the University of Illinois Chicago.
