Zettar delivers Zettar zx, a "Hyperscale Data Distribution" software platform for moving data at scale and speed. It supports both file based and cloud storage.
The term “Hyperscale Data” means the overall data size ≥ 1 TB; file/object sizes: a few KB – 1 TB; number of files/objects: from a few to multiple hundreds of millions; "distribution": with zx, it's feasible to build up various data distribution patterns or grids; "platform": zx can be managed programmatically. It is GA-grade, scale-out, petascale-proven, all-in-one, multi-100+Gbps capable. Furthermore, as a single application, Zettar zx efficiently tackles the following three "seemingly unrelated" data movement needs:
Zettar zx's performance is unaffected by distance, encryption, and checksumming. This is a goal that numerous parties, both academic and commercial, have struggled to reach for decades but failed. The tremendous value to the business efficiency of all distributed data-intensive enterprises should be evident.
In March 2019, Zettar won the Supercomputing Asia 2019 (SCA19) Data Mover Challenge (DMC), a grueling 2-month long international competition at the highest level. The Zettar team beat out six other elite national teams from the U.S. and Japan (slide 17) by a wide margin.
Unlike most other commercial data mover software, Zettar zx's modern, ground-breaking design can comfortably address the world's exponential data growth, for which modern IT needs a new pillar of support: data movement at scale and speed, along with storage, computing, and networking. Zettar aims to make zx the foundation of this fourth pillar.
Zettar has deployed its solution at a large global biopharmaceutical company, an European Oil & Gas major, a US DOE Exascale Computing Project (ECP) project, a hyperscale Web property, several supercomputing centers (domestic and international), and more. Many use cases involve the moving of multiple petabytes of data over distance.
Zettar is a revenue-supported startup in Palo Alto, California. It has also been awarded grants by both the U.S. National Science Foundation (NSF) and the Department of Energy (DOE) Office of Science.
Each end only employs a modest two-node cluster consisting of two inexpensive 1U commodity servers with 4x10Gbps unbonded Ethernet ports (thus 2 x 4 x10Gbps = 80Gbps - the bandwidth cap)
With the same data transfer nodes as the 2017 trial, and the 4 AIC SB122A-PH 1U 10-bay storage servers updated with Intel E5-2699v4 CPUs along with 16 Intel Optane P4800X 375GB SSDs, Zettar transferred, with encryption and checksumming, one petabyte of data in just 29 hours over the same connection. The average transfer rate is 75Gbps, or 94% utilization of the available bandwidth.
Quoting the SCA19 DMC Press Release, "Overall Winner Award: Zettar Inc. They achieve an amazing 68 Gbps transfer rate between Chicago and NSCC (Singapore)." - note that this feat was accomplished with a very restrictive environment, so proper tuning is not feasible. Also see below for the extremely demanding network topology.
In early November, barely two weeks prior to SC19, Zettar zx was used to established the historical 1st Poland to Singapore data transfer production trial over the new CAE-1 100Gbps connection linking London and Singapore, across a vast distance 12,375 miles, achieving a world-leading average transfer rate ~60 Gbps using modest production set up and hardware. The outcome was shown in the ICM SC19 booth 1393 and the A*STAR NSCC Singapore booth 2043. Dr. Marek Michalewicz, Director of ICM, presented the result in the well-attended SC19 Global Research Platform BoF session. The trial was also presented in the Supermicro’s SC19 booth as an invited talk.
The May 2017 trial results visualized
The September 2018 trial results visualized
The extremely demanding SCA19 DMC network topology
The November 2019 Poland-Singapore data transfer path: 12,375 miles