{
"slug": "Connect-to-google-cloud-object-storage-with-Golang-Language-a30a24e84601",
"title": "Connect to google cloud object storage with Golang Language",
"subtitle": "Google Cloud Object Storage is the one big and best device for storing sequences of bytes of files. It can handle it easily and store data…",
"excerpt": "Google Cloud Object Storage is the one big and best device for storing sequences of bytes of files. It can handle it easily and store data…",
"date": "2023-09-14",
"tags": [
"Go"
],
"readingTime": "6 min",
"url": "https://medium.com/@mobinshaterian/connect-to-google-cloud-object-storage-with-golang-language-a30a24e84601",
"hero": "https://cdn-images-1.medium.com/max/800/1*rs9aQdm9F2DHHhL9Lvb-Uw.png",
"content": [
{
"type": "heading",
"level": 2,
"text": "Connect to google cloud object storage with golang Language"
},
{
"type": "paragraph",
"html": "Google Cloud Object Storage is the one big and best device for storing sequences of bytes of files. It can handle it easily and store data in distributed locations. working with the SDK is so simple and they had an enormous library for handling the data."
},
{
"type": "paragraph",
"html": "Below explain how to make Storage with the UI of GCS."
},
{
"type": "heading",
"level": 2,
"text": "Creating a Client"
},
{
"type": "paragraph",
"html": "To start working with this package, We need to create a client.The client will use your default application credentials."
},
{
"type": "code",
"lang": "python",
"code": "import \"cloud.google.com/go/storage\"ctx := context.Background()client, err :=\nstorage.NewClient(ctx)if err != nil {\n // TODO: Handle error.}"
},
{
"type": "paragraph",
"html": "For Authentication and to connect securely to the GCS use the below document."
},
{
"type": "paragraph",
"html": "In our system, we make a JSON authentication and log in with a credential file."
},
{
"type": "image",
"src": "https://cdn-images-1.medium.com/max/800/1*rs9aQdm9F2DHHhL9Lvb-Uw.png",
"alt": "https://medium.com/@rahulrumalla/patterns-to-setup-secure-your-gcp-credentials-fo-go-apps-fe798bbf29e9",
"caption": "https://medium.com/@rahulrumalla/patterns-to-setup-secure-your-gcp-credentials-fo-go-apps-fe798bbf29e9",
"width": 752,
"height": 408
},
{
"type": "code",
"lang": "go",
"code": "filename := \"./address_of_json.json\" client, err := storage.NewClient(ctx,\noption.WithCredentialsFile(filename)) if err != nil { logger.Fatal(err) }"
},
{
"type": "heading",
"level": 2,
"text": "Buckets"
},
{
"type": "paragraph",
"html": "A Google Cloud Storage bucket is a collection of objects. To work with a bucket, make a bucket handle:"
},
{
"type": "code",
"lang": "go",
"code": "bucketName := \"my-first-bucket\"bkt := client.Bucket(bucketName)"
},
{
"type": "paragraph",
"html": "A handle is a reference to a bucket. You can have a handle even if the bucket doesn’t exist yet. To create a bucket in Google Cloud Storage, call BucketHandle.Create:"
},
{
"type": "code",
"lang": "go",
"code": "if err := bkt.Create(ctx, projectID, nil); err != nil { // TODO: Handle error.}"
},
{
"type": "paragraph",
"html": "Each bucket has associated metadata, represented in this package by BucketAttrs."
},
{
"type": "code",
"lang": "go",
"code": "attrs, err := bkt.Attrs(ctx)if err != nil { // TODO: Handle\nerror.}fmt.Printf(\"bucket %s, created at %s, is located in %s with storage class %s\\n\",\nattrs.Name, attrs.Created, attrs.Location, attrs.StorageClass)"
},
{
"type": "heading",
"level": 2,
"text": "Object"
},
{
"type": "paragraph",
"html": "An object holds arbitrary data as a sequence of bytes, like a file. You refer to objects using a handle, just as with buckets, but unlike buckets you don’t explicitly create an object."
},
{
"type": "code",
"lang": "go",
"code": "obj := bkt.Object(\"new-first-one\")"
},
{
"type": "heading",
"level": 2,
"text": "Write Bytes into Object"
},
{
"type": "paragraph",
"html": "GCS handles the file as a sequence of bytes and puts it into the object. In our case, we need to store bytes of input inside of the object."
},
{
"type": "code",
"lang": "go",
"code": "// Create a byte slice. bytes := []byte(\"Hello, world!\") writer := obj.NewWriter(ctx) _, err =\nwriter.Write(bytes) if err != nil { log.Fatal(err) } // Close the writer. err = writer.Close() if\nerr != nil { log.Fatal(err) }"
},
{
"type": "heading",
"level": 2,
"text": "Read Bytes From Object"
},
{
"type": "paragraph",
"html": "GCS can read bytes from an object at once or paginate the bytes."
},
{
"type": "code",
"lang": "go",
"code": "// Create a reader. reader, err := obj.NewReader(ctx) if err != nil { log.Fatal(err) } // Read the\nbytes from the file. bytes, err := io.ReadAll(reader) if err != nil { log.Fatal(err) } // Close the\nreader. err = reader.Close() if err != nil { log.Fatal(err) } // Convert the bytes to a string and\nprint it. fmt.Println(string(bytes))"
},
{
"type": "heading",
"level": 2,
"text": "Delete Object"
},
{
"type": "paragraph",
"html": "GCS can easily delete the object."
},
{
"type": "code",
"lang": "go",
"code": "err = obj.Delete(ctx) if err != nil { logger.Fatal(err) }"
},
{
"type": "heading",
"level": 2,
"text": "List of Object"
},
{
"type": "paragraph",
"html": "For getting a list of objects inside the bucket they prepare the iterator functionality."
},
{
"type": "code",
"lang": "go",
"code": "query := &storage.Query{Prefix: \"\"} var names []string it := bkt.Objects(ctx, query) for { attrs,\nerr := it.Next() if err == iterator.Done { break } if err != nil { log.Fatal(err) } names =\nappend(names, attrs.Name) }"
},
{
"type": "heading",
"level": 2,
"text": "CheckSum upload file"
},
{
"type": "paragraph",
"html": "check file upload successfully on server."
},
{
"type": "code",
"lang": "go",
"code": "ctx := context.Background() client, err := storage.NewClient(ctx) if err != nil { // TODO: handle\nerror. } data := []byte(\"verify me\") wc :=\nclient.Bucket(\"bucketname\").Object(\"filename1\").NewWriter(ctx) wc.CRC32C = crc32.Checksum(data,\ncrc32.MakeTable(crc32.Castagnoli)) wc.SendCRC32C = true if _, err :=\nwc.Write([]byte(\"hello world\")); err != nil { // TODO: handle error. // Note that Write may return\nnil in some error situations, // so always check the error from Close. } if err := wc.Close(); err\n!= nil { // TODO: handle error. } fmt.Println(\"updated object:\", wc.Attrs())"
},
{
"type": "heading",
"level": 2,
"text": "Set time out for client connction"
},
{
"type": "code",
"lang": "go",
"code": "ctx := context.Background() client, err := storage.NewClient(ctx) if err != nil { // TODO: handle\nerror. } tctx, cancel := context.WithTimeout(ctx, 30*time.Second) defer cancel() // Cancel when\ndone, whether we time out or not. wc :=\nclient.Bucket(\"bucketname\").Object(\"filename1\").NewWriter(tctx) wc.ContentType = \"text/plain\" wc.ACL\n= []storage.ACLRule{{Entity: storage.AllUsers, Role: storage.RoleReader}} if _, err :=\nwc.Write([]byte(\"hello world\")); err != nil { // TODO: handle error. // Note that Write may return\nnil in some error situations, // so always check the error from Close. } if err := wc.Close(); err\n!= nil { // TODO: handle error. } fmt.Println(\"updated object:\", wc.Attrs())"
},
{
"type": "heading",
"level": 2,
"text": "Set key for each object"
},
{
"type": "code",
"lang": "go",
"code": "func main() {\n ctx := context.Background() client, err := storage.NewClient(ctx) if err != nil {\n // TODO: handle error.\n }\n obj := client.Bucket(\"my-bucket\").Object(\"my-object\") // Encrypt the\n object's contents. w := obj.Key(secretKey).NewWriter(ctx) if _, err := w.Write([]byte(\"top secret\")); err != nil { // TODO: handle error. } if err := w.Close(); err != nil { // TODO: handle error. }}"
},
{
"type": "heading",
"level": 2,
"text": "Get attribute of object"
},
{
"type": "code",
"lang": "go",
"code": "func main() {\n ctx := context.Background() client, err := storage.NewClient(ctx) if err != nil {\n // TODO: handle error.\n }\n objAttrs, err := client.Bucket(\"my-bucket\").Object(\"my-object\").Attrs(ctx) if err != nil {\n // TODO: handle error.\n }\n fmt.Println(objAttrs)\n}"
},
{
"type": "heading",
"level": 2,
"text": "Update Object Meta data"
},
{
"type": "code",
"lang": "go",
"code": "func main() {\n ctx := context.Background() client, err := storage.NewClient(ctx) if err != nil {\n // TODO: handle error.\n }\n objAttrs, err := client.Bucket(\"my-bucket\").Object(\"my-object\").Attrs(ctx) if err != nil {\n // TODO: handle error.\n }\n fmt.Println(objAttrs)\n}"
},
{
"type": "heading",
"level": 2,
"text": "Using GoLang to create Google Cloud Signed URLs to upload images using ReactJS and dealing with CORS"
},
{
"type": "heading",
"level": 2,
"text": "Signed URLs"
},
{
"type": "paragraph",
"html": "it ‘s give time-limited access to a specific Cloud Storage resource. Anyone in possession of the signed URL can use it while it’s active, regardless of whether they have a Google account."
},
{
"type": "paragraph",
"html": "A signed URL is a URL that provides limited permission and time to make a request. Signed URLs contain authentication information in their query string, allowing users without credentials to perform specific actions on a resource. When you generate a signed URL, you specify a user or service account which must have sufficient permission to make the request that the signed URL will make. After you generate a signed URL, anyone who possesses it can use the signed URL to perform specified actions, such as reading an object, within a specified period of time."
},
{
"type": "paragraph",
"html": "https://cloud.google.com/storage/docs/access-control/signing-urls-with-helpers#storage-signed-url-object-go"
},
{
"type": "heading",
"level": 2,
"text": "Serialize a struct to bytes to send GCS"
},
{
"type": "heading",
"level": 3,
"text": "The naive approach: structured text as bytes"
},
{
"type": "paragraph",
"html": "The most straightforward but pretty inefficient method to convert a struct into a byte slice is to marshal it to JSON/YAML or any other markup language format but these formats are human-readable.So if performance or size matters, it’s better to utilize something that is faster and more compact rather than JSON or YAML."
},
{
"type": "heading",
"level": 3,
"text": "The more efficient approach: structured binary encoding"
},
{
"type": "paragraph",
"html": "The encoding/gob package of the Go standard library provides you an easy way to serialize your struct to bytes. It consumes less CPU power and memory than JSON but still preserves the field names which makes the output data length greater than it could be. What is really unpleasant is the size of the data sequence. It might be a bit greater than the length of the same struct object in the JSON format."
},
{
"type": "paragraph",
"html": "The encoding/binary package of the Go standard library allows you to convert structs or single values to byte sequences. You should also specify the endianness. The binary.BigEndian byte order is usually applied to transmit data over the network and the binary.LittleEndian byte order is mostly utilized to store data in the RAM and CPU registers. Also we will use our BytesReadWriteSeeker type for read and write operations."
},
{
"type": "heading",
"level": 2,
"text": "Encoding Data with the Go Binary Package"
},
{
"type": "heading",
"level": 2,
"text": "Sample code for GOB"
},
{
"type": "paragraph",
"html": "Special thanks with Mohsen Mottaghi"
}
]
}