#58 - Storage Vendor Hero Numbers | Storage Unpacked Podcast

This week, Chris Mellor is back and the team take a dive into the subject of storage vendor hero numbers. What are hero numbers? We’ve all seen them, they’re the huge performance figures quoted by all-flash storage vendors aimed at putting their products forward in the best light possible.

Are hero numbers believable or should we be looking at certified vendor benchmark testing as a guide to capability? Do users even look at benchmark or hero numbers in the first place? Could the whole exercise be a waste of time? The conversation moves to talk about STAC (Securities Technologies Analysis Center) and whether more application specific benchmarks have greater value. This includes the approach Pure Storage appears to have taken with FlashArray and FlashBlade, only quoting application-specific testing like that of AIRI.

As the discussion moves on, the team talk about more consistent deterministic latency (more on that here) and whether public cloud service providers will ever subject their service offerings to benchmark testing (TL;DR, probably not). As a final wrap up, the team asks, what can customers actually do to ensure they are seeing the real truth behind benchmark claims.

Elapsed Time: 00:20:48

Timeline

00:00:00 – Intro
00:01:00 – What are vendor hero numbers – ten meeelion IOPS!!
00:02:00 – But certified benchmarks are expensive to perform.
00:04:00 – Storage arrays aren’t interesting, do hero numbers help?
00:05:00 – Do users even care about industry standard benchmarks?
00:06:00 – STAC – financial application benchmarks
00:07:48 – Should vendors be more realistic about hero numbers?
00:09:50 – Why has Pure Storage stopped producing FlashArray performance data?
00:11:40 – However Pure do produce numbers for CI like AIRI (see Soundbytes #008).
00:14:30 – Do people do proofs of concept any more?
00:16:00 – Maybe arrays are massively over specified anyway?
00:16:30 – Or actually, it’s about consistent deterministic performance.
00:17:44 – Do we need performance benchmarks for cloud vendors?
00:19:00 – So what should customers do?
00:20:08 – Wrap Up