• Skip to main content
  • Skip to primary sidebar
  • About This Podcast
  • Hosts
  • Channels
  • Sponsorship
  • Podroll
  • Feedback

Storage Unpacked

A weekly podcast on deploying and managing enterprise storage and data

Home » #55 – Storage for Hyperscalers

#55 – Storage for Hyperscalers

29 June 2018 by Chris Evans

This week we talk to Mark Carlson, co-chair of the SNIA Technical Council, about the storage needs of hyperscalers.  Mark defines hyperscalers as those companies opening multiple data centres a year, most notably Amazon, Microsoft Azure, Google and Facebook in the US and Baidu, Alibaba and Tencent in China.  These vendors are deploying petabytes of storage a year, with specific requirements on storage media.  The hyperscaler applications have issues with HDD and SSD performance characteristics, such as tail latency and the effects of garbage collection.  As a result, drive manufacturers are building in new features to meet the needs of these companies.

What exactly are the issues?  The conversation covers some of the typical problems, like reusing existing partially failed media, the problems caused by tail latency, write amplification and why non-deterministic latency is such an issue.  Some vendor solutions include Depop, Fast Fail and Open Channel.  Not sure what these are?  Listen in to find out!

In the podcast, Mark mentions a white paper on hyperscaler storage.  You can find it here (PDF).  The SNIA YouTube channel can be found here.  Mark mentions his presentation at Tech Field Day, which can be found here.

Elapsed Time:  00:27:54

Timeline

  • 00:00:00 – Intro
  • 00:01:30 – Who do we mean by hyper-scalers?
  • 00:02:30 – ODM (Not OEM) – Original Design Manufacturers.
  • 00:04:00 – Aren’t HPE and Dell selling to hyper-scalers?
  • 00:05:30 – Why do hyper-scalers need different storage media?
  • 00:07:00 – Differences of approach to resiliency compared to Enterprise.
  • 00:09:00 – Don’t we want to have partial device failures?
  • 00:10:20 – Depop – depopulate and reset drive to factory settings.
  • 00:11:30 – Back to tail latency – extended response times.
  • 00:12:30 – NVM Sets to logically partition a drive for each application.
  • 00:13:40 – Fast Fail for HDD reads.
  • 00:15:00 – putting drives into a deterministic window – putting maintenance off.
  • 00:16:30 – Open Channel approach – let the host do the maintenance work.
  • 00:17:30 –  How are changes introduced?  ECNs and Technical Proposals.
  • 00:22:00 – Will hyper-scaler features filter down to the enterprise?
  • 00:26:00 – Wrap Up

Mark’s Bio

Mark A. Carlson, Principal Engineer, Industry Standards at Toshiba, has more than 35 years of experience with Networking and Storage development and more than 20 years experience with Java technology. Mark was one of the authors of the CDMI Cloud Storage standard. He has spoken at numerous industry forums and events. He is the co-chair of the SNIA Cloud Storage and Object Drive technical working groups, and serves as co-chair of the SNIA Technical Council.

Copyright (c) 2016-2018 Storage Unpacked.  No reproduction or re-use without permission. Podcast Episode HIJ4.
 

 

Podcast: Play in new window | Download

(Visited 353 times, 1 visits today)

Filed Under: Guest Speakers Tagged With: HDD, hyperscalers, Mark Carlson, SNIA, SSD

Reader Interactions

Trackbacks

  1. Why Deterministic Storage Performance is Important | Architecting IT says:
    2 July 2018 at 10:31 am

    […] On average, hard drives are relatively consistent, but for each individual I/O the response time could be very different indeed.  Hard drives also have lots of error recovery features that attempt to fix transient I/O errors.  These are the kinds of problems that cause tail latency – something the hyper-scale cloud service providers want to desperately avoid.  We talk about this problem with Mark Carlson on this week’s Storage Unpacked podcast. […]

  2. #59 - Ethernet vs Fibre Channel | Storage Unpacked Podcast says:
    1 August 2018 at 10:36 am

    […] #55 – Storage for Hyperscalers […]

  3. #60 - New Data Economy with Derek Dicker | Storage Unpacked Podcast says:
    18 August 2018 at 7:42 am

    […] #55 – Storage for hyperscalers […]

  4. Pure Storage - Seeding the NVMe Market | Architecting IT says:
    21 September 2018 at 7:15 am

    […] #55 – Storage for Hyperscalers […]

  5. Pure Storage - Seeding the NVMe Market - Architecting IT says:
    26 June 2019 at 12:51 pm

    […] #55 – Storage for Hyperscalers […]

  6. Why Deterministic Storage Performance is Important - Architecting IT says:
    18 February 2020 at 12:26 pm

    […] On average, hard drives are relatively consistent, but for each individual I/O the response time could be very different indeed.  Hard drives also have lots of error recovery features that attempt to fix transient I/O errors.  These are the kinds of problems that cause tail latency – something the hyper-scale cloud service providers want to desperately avoid.  We talk about this problem with Mark Carlson on this week’s Storage Unpacked podcast. […]

  7. #161 - Seagate MACH.2 Dual Actuator Drive Deep Dive | Storage Unpacked Podcast says:
    12 June 2020 at 12:01 pm

    […] #55 – Storage for Hyperscalers […]

  8. #62 - The Future of Data Infrastructure with Scott Hamilton | Storage Unpacked Podcast says:
    1 July 2020 at 11:57 am

    […] #55 – Storage for Hyperscalers […]

  9. #94 - Backblaze HDD Reliability Stats with Andy Klein | Storage Unpacked Podcast says:
    1 July 2020 at 1:01 pm

    […] #55 – Storage for Hyperscalers […]

Primary Sidebar

Subscribe to Podcast

Apple PodcastsGoogle PodcastsSpotifyAndroidStitcherby EmailRSS

Categories

  • All-Flash
  • AWS
  • AWS Reinvent
  • Cloud
  • CloudFlare
  • Commvault
  • Containers
  • Data Management
  • Data Protection
  • Databases
  • Events
  • Everspin Technologies
  • Fauna
  • File Systems
  • Flash Memory Summit 2022
  • Fungible
  • Garbage Collection
  • General
  • Guest Speakers
  • Hitachi NEXT 2019
  • Hitachi Vantara
  • HPE
  • HYCU Inc
  • IBM
  • In-Memory Computing
  • Infinidat
  • Intel
  • Liqid
  • MemVerge
  • Midrange
  • NVIDIA
  • NVMe
  • Object Storage
  • Persistent Memory
  • Pliops
  • Pure Accelerate techfest22
  • Pure Storage
  • Pure//Launch
  • Scality
  • Serverless
  • SmartNICs
  • Softiron
  • Software-Defined Storage
  • Soundbytes
  • Sponsored
  • Standards
  • Storage Hardware
  • Storage Management
  • StorONE
  • Tintri
  • VAST Data
  • Vendors
  • VMware
  • Western Digital

Copyright (c) 2021 Brookend Ltd, no reproduction without permission.