With the evolving architectural design of web applications, microservices have been a successful new trend in architecting the application landscape. Along with the advancements in application architecture, transport method protocols, such as REST and gRPC are getting better in efficiency and speed. Also, containerizing microservice applications help greatly in agile development and high-speed delivery.

In this blog, I will try to showcase how simple it is to build a cloud-native application on the microservices architecture using Go.

We will break the solution into multiple steps. We will learn how to:

1) Build a microservice and set of other containerized services which will have a very specific set of independent tasks and will be related only with the specific logical component.

2) Use go-kit as the framework for developing and structuring the components of each service.

3) Build APIs that will use HTTP (REST) and Protobuf (gRPC) as the transport mechanisms, PostgreSQL for databases and finally deploy it on Azure stack for API management and CI/CD.

Note: Deployment, setting up the CI-CD and API-Management on Azure or any other cloud is not in the scope of the current blog.

Prerequisites:

• A beginner’s level of understanding of web services, Rest APIs and gRPC
• GoLand/ VS Code
• Properly installed and configured Go. If not, check it out here
• Set up a new project directory under the GOPATH
• Understanding of the standard Golang project. For reference, visit here
• PostgreSQL client installed
• Go kit

What are we going to do?

We will develop a simple web application working on the following problem statement:

• A global publishing company that publishes books and journals wants to develop a service to watermark their documents. A document (books, journals) has a title, author and a watermark property
• The watermark operation can be in Started, InProgress and Finished status
• The specific set of users should be able to do the watermark on a document
• Once the watermark is done, the document can never be re-marked

Example of a document:

{content: “book”, title: “The Dark Code”, author: “Bruce Wayne”, topic: “Science”}

For a detailed understanding of the requirement, please refer to this.

Architecture:

In this project, we will have 3 microservices: Authentication Service, Database Service and the Watermark Service. We have a PostgreSQL database server and an API-Gateway.

‍

Authentication Service:

The application is supposed to have a role-based and user-based access control mechanism. This service will authenticate the user according to its specific role and return HTTP status codes only. 200 when the user is authorized and 401 for unauthorized users.

APIs:

• /user/access, Method: GET, Secured: True, payload: user: <name></name>
  It will take the user name as an input and the auth service will return the roles and the privileges assigned to it
• /authenticate, Method: GET, Secured: True, payload: user: <name>, operation: <op></op></name>
  It will authenticate the user with the passed operation if it is accessible for the role
• /healthz, Method: GET, Secured: True
  It will return the status of the service

Database Service:

We will need databases for our application to store the user, their roles and the access privileges to that role. Also, the documents will be stored in the database without the watermark. It is a requirement that any document cannot have a watermark at the time of creation. A document is said to be created successfully only when the data inputs are valid and the database service returns the success status.

We will be using two databases for two different services for them to be consumed. This design is not necessary, but just to follow the “Single Database per Service” rule under the microservice architecture.

APIs:

• /get, Method: GET, Secured: True, payload: filters: []filter{“field-name”: “value”}
  It will return the list of documents according to the specific filters passed
• /update, Method: POST, Secured: True, payload: “Title”: <id>, document: {“field”: “value”, ...}</id>
  It will update the document for the given title id
• /add, Method: POST, Secured: True, payload: document: {“field”: “value”, ...}
  It will add the document and return the title-ID
• /remove Method: POST, Secured: True, payload: title: <id></id>
  It will remove the document entry according to the passed title-id
• /healthz, Method: GET, Secured: True
  It will return the status of the service

Watermark Service:

This is the main service that will perform the API calls to watermark the passed document. Every time a user needs to watermark a document, it needs to pass the TicketID in the watermark API request along with the appropriate Mark. It will try to call the database Update API internally with the provided request and returns the status of the watermark process which will be initially “Started”, then in some time the status will be “InProgress” and if the call was valid, the status will be “Finished”, or “Error”, if the request is not valid.

APIs:

• /get, Method: GET, Secured: True, payload: filters: []filter{“field-name”: “value”}
  It will return the list of documents according to the specific filters passed
• /status, Method: GET, Secured: True, payload: “Ticket”: <id></id>
  It will return the status of the document for watermark operation for the passed ticket-id
• /addDocument, Method: POST, Secured: True, payload: document: {“field”: “value”, ...}
  It will add the document and return the title-ID
• /watermark, Method: POST, Secured: True, payload: title: <id>, mark: “string”</id>
  It is the main watermark operation API which will accept the mark string
• /healthz, Method: GET, Secured: True
  It will return the status of the service

Operations and Flow:

Watermark Service APIs are the only ones that will be used by the user/actor to request watermark or add the document. Authentication and Database service APIs are the private ones that will be called by other services internally. The only URL accessible to the user is the API Gateway URL.

1. The user will access the API Gateway URL with the required user name, the ticket-id and the mark with which the user wants the document to apply watermark
2. The user should not know about the authentication or database services
3. Once the request is made by the user, it will be accepted by the API Gateway. The gateway will validate the request along with the payload
4. An API forwarding rule of configuring the traffic of a specific request to a service should be defined in the gateway. The request when validated, will be forwarded to the service according to that rule.
5. We will define an API forwarding rule where the request made for any watermark will be first forwarded to the authentication service which will authenticate the request, check for authorized users and return the appropriate status code.
6. The authorization service will check for the user from which the request has been made, into the user database and its roles and permissions. It will send the response accordingly
7. Once the request has been authorized by the service, it will be forwarded back to the actual watermark service
8. The watermark service then performs the appropriate operation of putting the watermark on the document or add a new entry of the document or any other request
9. The operation from the watermark service of Get, Watermark or AddDocument will be performed by calling the database CRUD APIs and forwarded to the user
10. If the request is to AddDocument then the service should return the “TicketID” or if it is for watermark then it should return the status of  the operation

Note:

Each user will have some specific roles, based on which the access controls will be identified for the user. For the sake of simplicity, the roles will be based on the type of document only, not the specific name of the book or journal

Getting Started:

Let's start by creating a folder for our application in the $GOPATH. This will be the root folder containing our set of services.

Project Layout:

The project will follow the standard Golang project layout. If you want the full working code, please refer here

• api: Stores the versions of the APIs swagger files and also the proto and pb files for the gRPC protobuf interface.
• cmd: This will contain the entry point (main.go) files for all the services and also any other container images if any
• docs: This will contain the documentation for the project
• config: All the sample files or any specific configuration files should be stored here
• deploy: This directory will contain the deployment files used to deploy the application
• internal: This package is the conventional internal package identified by the Go compiler. It contains all the packages which need to be private and imported by its child directories and immediate parent directory. All the packages from this directory are common across the project
• pkg: This directory will have the complete executing code of all the services in separate packages.
• tests: It will have all the integration and E2E tests
• vendor: This directory stores all the third-party dependencies locally so that the version doesn’t mismatch later

We are going to use the Go kit framework for developing the set of services. The official Go kit examples of services are very good, though the documentation is not that great.

Watermark Service:

1. Under the Go kit framework, a service should always be represented by an interface.

Create a package named watermark in the pkg folder. Create a new service.go file in that package. This file is the blueprint of our service.

CODE: https://gist.github.com/velotiotech/6334b7537f0ac19211913b65d6ef49e1.js

2. As per the functions defined in the interface, we will need five endpoints to handle the requests for the above methods. If you are wondering why we are using a context package, please refer here. Contexts enable the microservices to handle the multiple concurrent requests, but maybe in this blog, we are not using it too much. It's just the best way to work with it.

3. Implementing our service:

CODE: https://gist.github.com/velotiotech/ef64f825586347a2693b501942aac783.js

We have defined the new type watermarkService empty struct which will implement the above-defined service interface. This struct implementation will be hidden from the rest of the world.

NewService() is created as the constructor of our "object". This is the only function available outside this package to instantiate the service.

4. Now we will create the endpoints package which will contain two files. One is where we will store all types of requests and responses. The other file will be endpoints which will have the actual implementation of the requests parsing and calling the appropriate service function.

- Create a file named reqJSONMap.go. We will define all the requests and responses struct with the fields in this file such as GetRequest, GetResponse, StatusRequest, StatusResponse, etc. Add the necessary fields in these structs which we want to have input in a request or we want to pass the output in the response.

CODE: https://gist.github.com/velotiotech/c246fa13bb6c15bb2b7db4aa36520854.js

- Create a file named endpoints.go. This file will contain the actual calling of the service implemented functions.

CODE: https://gist.github.com/velotiotech/d686e4eb2e7adba220b2a6ac11042f2a.js

In this file, we have a struct Set which is the collection of all the endpoints. We have a constructor for the same. We have the internal constructor functions which will return the objects which implement the generic endpoint. Endpoint interface of Go kit such as MakeGetEndpoint(), MakeStatusEndpoint() etc.

In order to expose the Get, Status, Watermark, ServiceStatus and AddDocument APIs, we need to create endpoints for all of them. These functions handle the incoming requests and call the specific service methods

5. Adding the Transports method to expose the services. Our services will support HTTP and will be exposed using Rest APIs and protobuf and gRPC.

Create a separate package of transport in the watermark directory. This package will hold all the handlers, decoders and encoders for a specific type of transport mechanism

6. Create a file http.go: This file will have the transport functions and handlers for HTTP with a separate path as the API routes.

CODE: https://gist.github.com/velotiotech/515896f98f8ae1a03a901b8b7c8bbe8c.js

This file is the map of the JSON payload to their requests and responses. It contains the HTTP handler constructor which registers the API routes to the specific handler function (endpoints) and also the decoder-encoder of the requests and responses respectively into a server object for a request. The decoders and encoders are basically defined just to translate the request and responses in the desired form to be processed. In our case, we are just converting the requests/responses using the json encoder and decoder into the appropriate request and response structs.

We have the generic encoder for the response output, which is a simple JSON encoder.

7. Create another file in the same transport package with the name grpc.go. Similar to above, the name of the file is self-explanatory. It is the map of protobuf payload to their requests and responses. We create a gRPC handler constructor which will create the set of grpcServers and registers the appropriate endpoint to the decoders and encoders of the request and responses

CODE: https://gist.github.com/velotiotech/bcaaf75154b7fc59a964d49d9e490bac.js

- Before moving on to the implementation, we have to create a proto file that acts as the definition of all our service interface and the requests response structs, so that the protobuf files (.pb) can be generated to be used as an interface between services to communicate.

- Create package pb in the api/v1 package path. Create a new file watermarksvc.proto. Firstly, we will create our service interface, which represents the remote functions to be called by the client. Refer to this for syntax and deep understanding of the protobuf.

We will convert the service interface to the service interface in the proto file. Also, we have created the request and response structs exactly the same once again in the proto file so that they can be understood by the RPC defined in the service.

CODE: https://gist.github.com/velotiotech/d90a6d3ff5cc747ffa1169f3ed7464a1.js

Note: Creating the proto files and generating the pb files using protoc is not the scope of this blog. We have assumed that you already know how to create a proto file and generate a pb file from it. If not, please refer protobuf and protoc gen

I have also created a script to generate the pb file, which just needs the path with the name of the proto file.

CODE: https://gist.github.com/velotiotech/55a7808e8c92bee11d1968f4140db5b1.js

8. Now, once the pb file is generated in api/v1/pb/watermark package, we will create a new struct grpcserver, grouping all the endpoints for gRPC. This struct should implement pb.WatermarkServer which is the server interface referred by the services.

To implement these services, we are defining the functions such as func (g *grpcServer) Get(ctx context.Context, r *pb.GetRequest) (*pb.GetReply, error). This function should take the request param and run the ServeGRPC() function and then return the response. Similarly, we should implement the ServeGRPC() functions for the rest of the functions.

These functions are the actual Remote Procedures to be called by the service.

We will also need to add the decode and encode functions for the request and response structs from protobuf structs. These functions will map the proto Request/Response struct to the endpoint req/resp structs. For example: func decodeGRPCGetRequest(_ context.Context, grpcReq interface{}) (interface{}, error). This will assert the grpcReq to pb.GetRequest and use its fields to fill the new struct of type endpoints.GetRequest{}. The decoding and encoding functions should be implemented similarly for the other requests and responses.

CODE: https://gist.github.com/velotiotech/0edc25d2ec60c68cf3a9f1f2ed621b58.js

9. Finally, we just have to create the entry point files (main) in the cmd for each service. As we already have mapped the appropriate routes to the endpoints by calling the service functions and also we mapped the proto service server to the endpoints by calling ServeGRPC() functions, now we have to call the HTTP and gRPC server constructors here and start them.

Create a package watermark in the cmd directory and create a file watermark.go which will hold the code to start and stop the HTTP and gRPC server for the service

CODE: https://gist.github.com/velotiotech/c59fce3cc0687c142aae10bb97377219.js

Let’s walk you through the above code. Firstly, we will use the fixed ports to make the server listen to them. 8081 for HTTP Server and 8082 for gRPC Server. Then in these code stubs, we will create the HTTP and gRPC servers, endpoints of the service backend and the service.

CODE: https://gist.github.com/velotiotech/75d7e165f0b146b3a447567b23d422ea.js

Now the next step is interesting. We are creating a variable of oklog.Group. If you are new to this term, please refer here. Group helps you elegantly manage the group of Goroutines. We are creating three Goroutines: One for HTTP server, second for gRPC server and the last one for watching on the cancel interrupts. Just like this:

CODE: https://gist.github.com/velotiotech/dfa30ce306597a7761ae54da2eb9d812.js

Similarly, we will start a gRPC server and a cancel interrupt watcher.
Great!! We are done here. Now, let's run the service.

CODE: https://gist.github.com/velotiotech/2c9daafb88dd301ada5c22aaf3eb4a2f.js

The server has started locally. Now, just open a Postman or run curl to one of the endpoints. See below:
We ran the HTTP server to check the service status:

CODE: https://gist.github.com/velotiotech/0a586a35741061622ef74b3e4d775dfc.js

We have successfully created a service and ran the endpoints.

Further:

I really like to make a project complete always with all the other maintenance parts revolving around. Just like adding the proper README, have proper .gitignore, .dockerignore, Makefile, Dockerfiles, golang-ci-lint config files, and CI-CD config files etc.

I have created a separate Dockerfile for each of the three services in path /images/.

I have created a multi-staged dockerfile to create the binary of the service and run it. We will just copy the appropriate directories of code in the docker image, build the image all in one and then create a new image in the same file and copy the binary in it from the previous one. Similarly, the dockerfiles are created for other services also.

In the dockerfile, we have given the CMD as go run watermark. This command will be the entry point of the container.
I have also created a Makefile which has two main targets: build-image and build-push. The first one is to build the image and the second is to push it.

Note: I am keeping this blog concise as it is difficult to cover all the things. The code in the repo that I have shared in the beginning covers most of the important concepts around services. I am still working and continue committing improvements and features.

Let’s see how we can deploy:

We will see how to deploy all these services in the containerized orchestration tools (ex: Kubernetes). Assuming you have worked on Kubernetes with at least a beginner’s understanding before.

In deploy dir, create a sample deployment having three containers: auth, watermark and database. Since for each container, the entry point commands are already defined in the dockerfiles, we don't need to send any args or cmd in the deployment.

We will also need the service which will be used to route the external traffic of request from another load balancer service or nodeport type service. To make it work, we might have to create a nodeport type of service to expose the watermark-service to make it running for now.

Another important and very interesting part is to deploy the API Gateway. It is required to have at least some knowledge of any cloud provider stack to deploy the API Gateway. I have used Azure stack to deploy an API Gateway using the resource called as “API-Management” in the Azure plane. Refer the rules config files for the Azure APIM api-gateway:

Further, only a proper CI/CD setup is remaining which is one of the most essential parts of a project after development.
I would definitely like to discuss all the above deployment-related stuff in more detail but that is not in the scope of my current blog. Maybe I will post another blog for the same.

Wrapping up:

We have learned how to build a complete project with three microservices in Golang using one of the best-distributed system development frameworks: Go kit. We have also used the database PostgreSQL using the GORM used heavily in the Go community.
We did not stop just at the development but also we tried to theoretically cover the development lifecycle of the project by understanding what, how and where to deploy.

We created one microservice completely from scratch. Go kit makes it very simple to write the relationship between endpoints, service implementations and the communication/transport mechanisms. Now, go and try to create other services from the problem statement.