As part of an initiative to expand their presence in cloud/datacenter, video and telecommunication verticals, digital solutions provider EchoStreams was looking for a new system to add to their FlacheSAN line. These new servers would provide high performance SSD storage for content delivery networks (CDNs), and video production and post-production applications — all of which typically require both high transfer rates and high numbers of input/output operations per second (IOps). With video file sizes reaching over a terabyte per hour of video, they required some serious bandwidth. On top of all that, customers wanted newer solutions that used NVMe SSDs to achieve higher performance over the existing SATA-based SSDs, while reducing the size and power requirements of the storage systems.
The Challenge
The Solution
Systems engineers began development of two systems. The first used 20 U.2 NVMe PCIe SSDs. The second included 36 M.2 NVMe PCIe SSDS. One of the first hurdles was creating a system that could connect the extensive number of NVMe devices. For that, they turned to the new AMD EPYC processor that can support 128 PCIe lanes, and to two Samsung SSDs — the Samsung 983 DCT and the 970 PRO.
The new system being developed increased throughput by over 300% and IOps by 400%, while lowering power and cooling requirements.
The Results
The previous FlacheSAN1L-D4 system, featuring 20 SATA SSDs and dual Xeon E5-2600 v4 (Broadwell), achieved approximately 10GB/s throughput with about 1 million IOps (4K block size QD 32). The new system under development, the FlacheSAN1N20U-UN with 20 Samsung 983 DCT NVMe drives and a single AMD EPYC, hit 34GB/s — a more than 300 percent improvement — and about 4 million IOps — a 400 percent improvement. The first system prototype is currently shipping to limited customers.
EchoStreams Innovative Solutions, LLC is an original equipment manufacturer (OEM) and original design manufacturer (ODM) solutions provider focused specifically on server and storage technologies for the fastest growing verticals in today’s digital era, such as CDNs, cloud and datacenter, video production and post-production and telecommunications. Located in City of Industry, California, their engineering team works on the development of new servers for these and other IT industries, testing the systems with multiple industry-standard benchmarkers to ensure performance is optimized for each of the various supported applications.
The Challenge
Expanding Storage Speeds for Maximum Transfer Rate Throughput
EchoStreams’ customers were requesting faster systems for their CDNs, video production and post-production storage, and big data systems. The OEM/ODM solutions provider determined to meet customer needs with a new version of their FlacheSAN hardware incorporating the latest high-performance SSDs. These new systems would have to exhibit the highest available transfer rates as well as above-average IOps.
Video production and post-production in particular demand very high transfer rates throughput. With higher resolution video (4K or 8K) and more color depth (10bit or more), HDR and higher frame rates (60fps or even 120fps), movie studios need super fast storage to transfer files for editors in post-production. Requirements for CDNs are more mixed — some will need high throughput, some will need high IOps and some will need both.
Previous solutions had relied on lots of drives to deliver aggregate performance with high enough transfer rates and IOps to satisfy application demands, but systems with large numbers of SATA drives also produce a lot of heat and require larger power supplies.
There are several things that limit the number of drives that can fit in a single enclosure. Some are obvious, like the volume of each drive or the amount of heat that each drive generates, which must be dissipated. Other issues, like the total number of connections the interface can support, are less obvious. Echostreams had to find a way to increase the speed and power of their systems without increasing the number of drives.
The Solution
Samsung NVMe SSDs Provide More Speed for Less Power
EchoStreams engineered a system that allows two NVMe drives to share one 4x PCIe connection, allowing 20 U.2 drives with Intel motherboards, and up to 36 M.2 drives with an AMD EPYC system. The lower power usage of NVMe drives enables systems that run relatively cool and use less power than the SATA drives that were previously in use.
While they design and manufacture all other boards such as backplane, risers and the chassis and system packaging, the EchoStreams NVMe systems were designed using a Tyan/Mitac motherboard. With SATA, it was not uncommon to stuff 60 or more drives onto a backplane. With NVMe, the speed of the drives requires a high-speed backplane, with many motherboards. With Xeon CPUs, PCIe is currently limited to 40 lanes, while the new AMD EPYC system supports 128.
Both systems feature Samsung NVMe drives. The first system to be released features up to 20 Samsung 983 DCT drives; the second will feature up to 36 Samsung 970 PRO SSDs. The latter, using M.2 SSDs, will take advantage of the greater power efficiency, higher performance and lower prices of the 970 PRO units. The systems use the NVMeoF protocol by default, but also support iSCSI with RoCE, and Infiniband.
The servers include three 100Gb Ethernet network interface controllers (NICs), with options for either Mellanox or Qlogic (now Cavium) NICs. There are two PCIe3 x16 slots for 100G NICs, and 1 slot on an open compute project (OCP 2.0) mezzanine card, a PCIe3 x16 slot for an additional NIC. The default configuration is three 100G NICs, but other NICs can be ordered if desired. The motherboard also includes two 1GigE ports and a baseboard management controller (BMC) management port.
The systems continue to evolve as NVMe standards change. Over the last two years, NVMe has gone from 1.1 to 1.2 to 1.3, adding features such as a standard way to blink LEDs to identify specific drives.
The Technology
983 DCT 960GB
High performance, highly reliable and secure NVMe U.2 SSD designed for servers that need enhanced data reliability to safeguard critical data.
SSD 970 PRO NVMe M.2 1TB
Ultimate NVMe Read/Write performance. V-NAND reliability backed by up to 5 yr. warranty.
The Results
Three Times the Throughput and Four Times the IOps With Equal Power Consumption
The old SATA-based systems could only sustain 550MB/s, while the new systems support 3GB/s read with the Samsung 983 DCTs. The old systems could hit writes 500MB/s, while the new ones hit 1.9 GB/s with the 983 DCTs. IOps went from 100,000 random reads to 500,000 random reads (4K block size) per drive.
The previous FlacheSAN1L-D4 with 20 SATA SSDs and dual Xeon E5-2600 v4 (Broadwell) achieved about 10GB/s sustained throughput with about 1 million IOps (4K block size QD 32). The new FlacheSAN1N20U-UN (featuring 20 Samsung 983 DCT NVMe drives and a single AMD EPYC 7401 24-core CPU with 132GB RAM) was able to hit 34GB/s — an improvement of more than 300 percent — and about 4 million IOps, an improvement of about 400 percent.
The IOps per dollar and performance per watt are also both greatly improved. Power consumption is about the same for the whole system, but generates much higher performance, rising from 500Mbps to 3000MB/s with the same power consumption.
CDNs typically use many relatively small drives (rather than a few large drives), since that results in less data loss per drive in the event of a failure, and more drives provide better results. Media systems supporting 4K to 8K production and HDR need the most capacity: one movie in production can use one terabyte per hour of video.
The old system was a 2U chassis with 40 SATA SSDs and had hit a maximum throughput of 18GB/s; the new system is 40GB/s with half the power consumption. With three 100Gb NICs, 16 to 18 Samsung 983 DCT SSDs can be used to saturate the NICs. More drives can reduce the latency and increase IOps, but higher throughput would require faster NICs.
The new systems are still being developed and evaluated, and will continue to improve. However, even at this preliminary stage, it’s clear that the new NVMe SSDs offer higher performance at a lower operating cost, with the ability to add more SSDs to a single chassis without increasing power consumption or cooling requirements.
