About Zettar
Zettar delivers Zettar zx, a "Hyperscale Data Distribution" software platform for moving data at scale and speed. It supports both file based and cloud storage.
The term “Hyperscale Data” means the overall data size ≥ 1 TB; file/object sizes: a few KB – 1 TB; number of files/objects: from a few to multiple hundreds of millions; "distribution": with zx, it's feasible to build up various data distribution patterns or grids; "platform": zx can be managed programmatically. It is GA-grade, scale-out, petascale-proven, all-in-one, multi-100+Gbps capable. Furthermore, as a single application, Zettar zx efficiently tackles the following three "seemingly unrelated" data movement needs:
- Bulk transfer/stream data over digital connections (multiple of: Mbps, Gbps, 10Gbps, 100Gbps; LAN, Metro, WAN)
- Replicate data sets incrementally, even those with hundreds of millions of small files/objects
- Accelerate the data loading/unloading for data shipping devices (AWS Snowball alike)
Zettar zx's performance is unaffected by distance, encryption, and checksumming. This is a goal that numerous parties, both academic and commercial, have struggled to reach for decades but failed. The tremendous value to the business efficiency of all distributed data-intensive enterprises should be evident.
In March 2019, Zettar won the Supercomputing Asia 2019 (SCA19) Data Mover Challenge (DMC), a grueling 2-month long international competition at the highest level. The Zettar team beat out six other elite national teams from the U.S. and Japan (slide 17) by a wide margin.
Unlike most other commercial data mover software, Zettar zx's modern, ground-breaking design can comfortably address the world's exponential data growth, for which modern IT needs a new pillar of support: data movement at scale and speed, along with storage, computing, and networking. Zettar aims to make zx the foundation of this fourth pillar.
Zettar has deployed its solution at a large global biopharmaceutical company, an European Oil & Gas major, a US DOE Exascale Computing Project (ECP) project, a hyperscale Web property, several supercomputing centers (domestic and international), and more. Many use cases involve the moving of multiple petabytes of data over distance.
Zettar is a revenue-supported startup in Palo Alto, California. It has also been awarded grants by both the U.S. National Science Foundation (NSF) and the Department of Energy (DOE) Office of Science.
With an appropriate infrastructure (storage, computing, networking), zx has the following accomplishments and more
May 2017 Zettar Moves Petabyte Datasets at Unprecedented Speed via ESnet
Each end only employs a modest two-node cluster consisting of two inexpensive 1U commodity servers with 4x10Gbps unbonded Ethernet ports (thus 2 x 4 x10Gbps = 80Gbps - the bandwidth cap)
September 2018 ESnet's Network, Software Help SLAC Researchers in Record-Setting Transfer of 1 Petabyte of Data
With the same data transfer nodes as the 2017 trial, and the 4 AIC SB122A-PH 1U 10-bay storage servers updated with Intel E5-2699v4 CPUs along with 16 Intel Optane P4800X 375GB SSDs, Zettar transferred, with encryption and checksumming, one petabyte of data in just 29 hours over the same connection. The average transfer rate is 75Gbps, or 94% utilization of the available bandwidth.
March 2019 Zettar won the Supercomputing Asia 2019 (SCA19) Data Mover Challenge (DMC)
Quoting the SCA19 DMC Press Release, "Overall Winner Award: Zettar Inc. They achieve an amazing 68 Gbps transfer rate between Chicago and NSCC (Singapore)." - note that this feat was accomplished with a very restrictive environment, so proper tuning is not feasible. Also see below for the extremely demanding network topology.
November 2019 Zettar zx enabled the historical 1st Poland - Singapore data transfer production trial over the new CAE-1 100Gbps link
In early November, barely two weeks prior to SC19, Zettar zx was used to established the historical 1st Poland to Singapore data transfer production trial over the new CAE-1 100Gbps connection linking London and Singapore, across a vast distance 12,375 miles, achieving a world-leading average transfer rate ~60 Gbps using modest production set up and hardware. The outcome was shown in the ICM SC19 booth 1393 and the A*STAR NSCC Singapore booth 2043. Dr. Marek Michalewicz, Director of ICM, presented the result in the well-attended SC19 Global Research Platform BoF session. The trial was also presented in the Supermicro’s SC19 booth as an invited talk.
The May 2017 trial results visualized
The September 2018 trial results visualized
The extremely demanding SCA19 DMC network topology
The November 2019 Poland-Singapore data transfer path: 12,375 miles
News
- Chin Fang, How To Improve Biopharmaceutical Research Data Utilization?, Bio-IT World, Contributed commentary, January 08, 2020
- ICM University of Warsaw, Poland news release, Poland-Singapore data transfer over new CAE-1 100G trans-continental link, November 13, 2019
- InsideHPC article, Production Trial Shows Global Science Possible with CAE-1 100Gbps link, November 11, 2019
- Chin Fang, Is A Science DMZ A Key To Solving Poor Data Utilization?, Bio-IT World, Contributed commentary, Apr 16, 2019
- ESnet News & Publications, ESnet’s Networking Prowess on Display at Singapore Conference, April 8, 2019
- Supercomputing Asia 2019, Data Mover Challenger Winners Announced!, National Supercomputing Centre, Singapore, Mar 13, 2019
- Thomas Coughlin, Wicked Fast Data Transport, Frobes, Feb 11, 2019
- Press release, BeeGFS based burst buffer enables world record hyperscale data distribution, Oct 25, 2018
- Machine Design, DoE Tests Newest Information Superhighway, October 23, 2018
- ESnet News & Publications, ESnet's Network, Software Help SLAC Researchers in Record-Setting Transfer of 1 Petabyte of Data, OCTOBER 17, 2018
- InsideHPC article, Big Data over Big Distance: Zettar Moves a Petabyte over 5000 Miles in 29 Hours, October 5, 2018
- Press release, Zettar Transferred, with Encryption, One Petabyte of Data in Just 29 Hours Using AIC Servers, October 04, 2018
- Chin Fang, Dealing With Fast Growing Data With Hyperscale Data Distribution, Bio-IT World, Contributed commentary, August 17, 2018