The Simple Storage Service (S3) protocol is a standard interface developed by Amazon Web Services (AWS) for storing and retrieving data in the cloud. It’s a key-value store, meaning that each piece of data (an object) is associated with a unique identifier (a key) that you use to access that data. The S3 protocol is crucial in cloud storage services for several reasons:
- Scalability
S3 allows for virtually unlimited storage, making it ideal for applications that need to scale rapidly or handle large amounts of data. - Durability and Availability
S3 is designed for 99.999999999% (11 9’s) of durability, meaning the chances of losing data are extremely low. It also guarantees 99.99% availability, ensuring that your data is accessible when you need it. - Security
S3 provides robust security features, including encryption for data at rest and in transit, access control policies, and logging capabilities. - Interoperability
Many cloud storage providers, not just AWS, support the S3 protocol, making it a versatile choice for developers. - Cost-Effective
With S3, you only pay for the storage you use, making it a cost-effective solution for businesses of all sizes.
In summary, the S3 protocol is a reliable, secure, and scalable solution for storing data in the cloud, making it a cornerstone of modern cloud-based applications.
Setting Up Go Environment
install AWS SDK into our project
go get github.com/aws/aws-sdk-go-v2/aws
go get github.com/aws/aws-sdk-go-v2/config
go get github.com/aws/aws-sdk-go-v2/credentials
go get github.com/aws/aws-sdk-go-v2/feature/s3/manager
go get github.com/aws/aws-sdk-go-v2/service/s3
Simple project to upload file
AWS SDK Wrapper
package s3client
import (
"context"
"github.com/aws/aws-sdk-go-v2/aws"
"github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/feature/s3/manager"
"github.com/aws/aws-sdk-go-v2/service/s3"
"io"
"log"
"sync"
)
var (
uploader *manager.Uploader
once sync.Once
)
func setup() {
endpoint := "http://s3-test-endpoint.com"
accessKey := "XVXQ8Q3QGQ9QXQ8QXQ8Q"
secretKey := "/XQ8Q3QGQ9QXQ8QXQ8QXQ/Q3QGQ9QXQ8QXQ8QXQ8"
resolver := aws.EndpointResolverWithOptionsFunc(func(service, region string, options ...interface{}) (aws.Endpoint, error) {
return aws.Endpoint{
URL: endpoint,
HostnameImmutable: true,
Source: aws.EndpointSourceCustom,
}, nil
})
awsConfig, err := config.LoadDefaultConfig(context.TODO(),
config.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(accessKey, secretKey, "")),
config.WithEndpointResolverWithOptions(resolver),
config.WithRegion("auto"),
)
if err != nil {
log.Fatalf("failed to load config, %v", err)
}
uploader = manager.NewUploader(s3.NewFromConfig(awsConfig)) // Initialize S3 client
}
func UploadFile(ctx context.Context, bucket, filename string, file io.Reader) (string, error) {
once.Do(setup)
result, err := uploader.Upload(ctx, &s3.PutObjectInput{
Bucket: aws.String(bucket),
Key: aws.String(filename),
Body: file,
ACL: "public-read",
})
if err != nil {
log.Fatalf("failed to upload file, %v", err)
}
return result.Location, nil
}
Why we need to wrap third-party library?
The purpose wrapper is to simplify the library’s interface, adapt it to fit our application’s specific needs, or add additional functionality. In this case we simplifies the process of uploading a file to S3 by providing a singleUploadFile function that handles the setup and upload process. This makes it easier for the rest of our application to upload files to S3, as it only needs to call s3client.UploadFilewith the appropriate parameters.
Anyway, let’s explain the code a little bit
We need to setup our project to connect with the cloud storage services, in the setupfunction there is 2 key steps that we need to highlight.
First, inresolver parameter we need to pass aws.EndpointSourceCustomto indicate we didn’t connect to AWS endpoint.
Secondly,config.WithRegion("auto")when load default config.
Then inUploadFile function we use once.Do(setup) to make sure setup function only called once.
Main Function
We will create simple server using chi, then create POST API to upload file to our S3 Cloud Storage. Also Hello World endpoint to make sure our server is working 😆.
package main
import (
"github.com/go-chi/chi/v5"
"github.com/yudaph/s3-integration/s3client"
"net/http"
)
func main() {
r := chi.NewRouter()
r.Get("/", func(w http.ResponseWriter, r *http.Request) {
w.Write([]byte("Hello, World!"))
})
r.Post("/", HandleUpload)
http.ListenAndServe(":8080", r)
}
func HandleUpload(w http.ResponseWriter, r *http.Request) {
file, header, err := r.FormFile("image")
if err != nil {
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
defer file.Close()
// TODO: Do Validation
// Upload file to S3
filename := "suffix_" + header.Filename
url, err := s3client.UploadFile(r.Context(), "bucket-name", filename, file)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.Write([]byte(url))
}
Let’s explain the HandleUploadfunction:
We will user.FormFile("image") to get file from request, in other framework likemux and fiber we only get the header, so if we need to use header.Open() to get the file from header(CMIIW).
Then do some validation, after that we simply call our wrapper functions3client.UploadFile to upload to our cloud storage.
Conclusion
In this article we learn about S3 protocol, how to Integrate with Non-AWS Cloud Storage Service that supports S3 protocol, and a little bit about wrapper.
Hopefully, this article can be helpful. Don’t hesitate to give criticism and suggestions if you find any mistakes in this article.
Let’s connect on https://www.linkedin.com/in/yudaph/ and https://github.com/yudaph