Over the past decade, cloud technologies have evolved from simple Infrastructure as a Service offerings to diversified services and middleware portfolios. One of the major areas of advancement is towards the cloud storage offerings.
When discussing cloud storage, one of the common questions comes up is that what does cloud storage means to someone.
If we look at modern cloud storage, it ranges from general purpose block storage comes for servers to higher level storage services such as Relational databases, NoSQL. In addition, the cloud offers different storage configurations for underlying infrastructure. As you can see, cloud storage is a broader topic which needs further understanding when selecting storages for different use cases.
This article focuses on providing a higher level overview of different cloud storage options and application context so that it will help you to take effective decisions on storage selection in the cloud. I will be also using AWS cloud storage services as examples for cloud storage options so that it will be helpful to find further details about practical use cases.
Physical Storage Options (Foundational Storage)
This is the most common characteristics which we understand when it comes to storage looking from a physical storage standpoint. If we look at different physical storage options, magnetic hard disk storage, solid state drives and tape drives are the common configurations comes into mind. In the context of cloud, these storage configurations offered to consumers either as dedicated offerings or as shared disk storage where the inputs and outputs (IOPS) and storage capacity is shared with others.
When starting a server in a cloud, it’s more likely you are either using pre-defined physical storage configurations or customized configurations for the disk storage, where the operating system runs. It’s also possible to attach addition storages or expand the storage to these servers when needed.
If we take AWS for example, it provides solid state storage (SSD), magnetic hard disk storage (two subtypes called throughput optimized and cold start storage) and NVMe storage.
Virtual Storage Options (Storage Building Blocks)
When using cloud services, one of its strengths comes with the virtualization by providing convenient building blocks for application development. This is also the same for the storage. Although physical storage configurations are needed, most of the applications require higher level capabilities like availability, fault tolerance, durability, security & etc. Since most of the cloud providers use commodity hardware it is of utmost importance that these abstractions are provided to simplify the use cases.
Few common storage building blocks most of the cloud service providers offers are block storage (reliability with redundancy and reusable), object storage (reliable storage with redundancy and higher-level APIs), network file systems (reliable shared storage).
Some of these storage building blocks even provide the ability to select the physical storage configurations like physical storage type (e.g; SSD or Magnetic Hard Disk).
In AWS, EBS is the block storage offering which could be attached to EC2 instances (Virtual Machines). Since it has redundancy built in, a physical disk failure won’t affect the availability of storage. AWS S3 is one of the popular options for object storage where it exposes a REST API to communicate with the service where multiple data center level redundancy is there. AWS EFS is their network file system offering for shared storage that could be shared across several EC2 instances at runtime. Amazon Glacier is a tape-based storage option for data archival. AWS KMS is special purpose storage offerings with virtual and physical hardware security module support for storing encryption keys. You can find some applications of these storage options in AWS EBS vs Instance Storage Patterns for Application Use Cases article.
Application Storage Services
These storage services are mostly considered as a platform as service offerings by various cloud providers which includes, managed or cloud-native relational database services, NoSQL database services, caching storage, queue storage services and etc.
Although I have classified Amazon S3 storage building blocks considering it as an object storage option, it is difficult to draw the line between some of the storage building blocks like S3 and application storage services due to the wide range of functionalities they provide.
These storage services are directly usable as application-level building blocks to design solutions. If we look at the underlying infrastructure, most of these use virtual storage options underneath to provide the reliability, security & etc.
If you think of setting up your own relational database system instead of using these application storage services, it should be possible. However, it might become challenging when it comes to managing those where it requires to do security patching, implement support for disaster recovery, implement fault tolerance and high availability & etc where it requires expert knowledge and continuous commitment which adds up for the total cost of ownership.
In AWS, Amazon RDS provides ranges of managed relational database offerings including MySQL, MariaDB, Oracle, Microsoft SQL Server, PostgreSQL and cloud-native Amazon Aurora database. AWS DynamoDB is the NoSQL offering provided by Amazon which is a simple key-value based database which is massively scalable. Amazon ElastiCache offers fully managed Redis and Memcached for cache storage.