public/subjects/devops/crud-master/README.md

## CRUD Master

### Instructions

APIs are a very common and convenient way to deploy services in a modular way.
In this exercise we will create a simple microservices infrastructure, having an API Gateway connected to two other services.
Those two services will in turn get data from two distinct databases.
The communication between services will be done by using HTTP and message queuing systems.
All those services will be in turn encapsulated in different virtual machines.

#### General overview

![CRUD Master architecture diagram](resources/crud-master-diagram.png)

We will setup a movie streaming platform, where one API (`inventory`) will have information on the movies available and another one (`billing`) will process the payments.

The API gateway will communicate in HTTP with `inventory` and using RabbitMQ for `billing`.

In this exercise you will need to install Node.js (with Express, Sequelize and other packages), PostgreSQL, RabbitMQ, Postman, VirtualBox and Vagrant.

While it may seems overwhelming at first there is a lot of resources available both on official website and on community blogs about setting up those tools.
Also the specific configuration details may change from platform to platform so don't hesitate to play around with it and be sure everything is installed correctly before to move on.

#### API 1: Inventory

##### Definition of the Inventory API

This API will be a CRUD (Create, Read, Update, Delete) RESTful API.
It will use a PostgreSQL database.
It will provide information about the movies present in the inventory and allow users to do basic operations on it.

A common way to do so is to use Express which is a popular Node.js web framework.
We will couple it with Sequelize, an ORM which will abstract and simplify the interactions between our API and the database.

Here are the endpoints with the possible HTTP requests:

- `/api/movies`: GET, POST, DELETE
- `/api/movies/:id`: GET, PUT, DELETE

Some details about each one of them:

- `GET /api/movies` retrieve all the movies.
- `GET /api/movies?title=[name]` retrieve all the movies with `name` in the title.
- `POST /api/movies` create a new product entry.
- `DELETE /api/movies` delete all movies in the database.

- `GET /api/movies/:id` retrieve a single movie by `id`.
- `PUT /api/movies/:id` update a single movie by `id`.
- `DELETE /api/movies/:id` delete a single movie by `id`.

The API should work on `http://localhost:8080/`.

##### Defining the Database

For the database we will use PostgreSQL.
The database will be called `movies`.

The `movies` table will contain the following columns:

- `id`: autogenerated unique identifier.
- `title`: the title of the movie.
- `description`: the description of the movie.

##### Testing the Inventory API

In order to test the correctness of your API you should use Postman. You could create one or more tests for every endpoint and then export the configuration so you will be able to reproduce the tests on different machines easily.

#### API 2: Billing

##### Definition of the Inventory API

This API will only receive messages through RabbitMQ.
The message it receives should be a stringified JSON object as in this example:

```json
{
    "user_id": "3",
    "number_of_items": "5",
    "total_amount": "180"
}

```

It will parse the message and create a new entry in the `orders` database.
It will also acknowledge the RabbitMQ queue that the message has been processed.
When the API is started it will take and process all messages present in the queue.

> Take a look into `amqplib` npm package for an easy way to interface with RabbitMQ.

##### Defining the Database

For the database we will use PostgreSQL here as well.
The database will be called `orders`.

The `orders` table will contain the following columns:

- `id`: autogenerated unique identifier.
- `user_id`: the id of the user making the order.
- `number_of_items`: the number of items included in the order.
- `total_amount`: the total cost of the order.

##### Testing the Billing API

To test this API here are some steps:

- Send POST orders to the API Gateway (you can use Postman for that).
- When the Billing API is running the orders should appear instantaneously in the `orders` database.
- When the Billing API is not running the queries to the API Gateway should still return success but the `orders` database won't be updated.
- When the Billing API is started again the unfulfilled messages should be processed and the `orders` database should be updated.

#### The API Gateway

The Gateway will take care of routing the requests to the appropriate service using the right protocol (it could be HTTP for the Inventory API or RabbitMQ for for the Billing API).

##### Interfacing with Inventory API

The gateway will route all requests to `/api/movies` at the API1, without any need to check the information passed through it.
It will return the exact response received by the API1.

In order to achieve this goal it will be necessary to setup a proxy system.
You could check `http-proxy-middleware` npm package in order to achieve such goal.

##### Interfacing with Billing API

The gateway will receive POST requests from `api/billing` and send a message using RabbitMQ in a queue called `billing_queue`.
The content of the message will be the POST request body stringified with `JSON.stringify`.
The Gateway should be able to send messages to the API2 even if that API is not running. When the API2 will be started it should be able to process that message and send an acknowledgement back.

An example of POST request to `http://[API_GATEWAY_URL]:[API_GATEWAY_PORT]/api/billing/`:

```json
{
    "user_id": "3",
    "number_of_items": "5",
    "total_amount": "180"
}
```

> Remember to setup `Content-Type: application/json` for the body of the request.

##### Documenting the API

Good documentation is a very critical feature of every API. By design the APIs are meant for others to use, so there have been very good efforts to create standard and easy to implement ways to document it.

As an introduction to the art of great documentation you must create an OpenAPI documentation file for the API Gateway. There is many different ways to do so, a good start could be using SwaggerHub with at least a meaningful description for each endpoint. Feel free to implement any extra feature as you see fit.

> You must also create a `README.md` file at the root of your project with detailed instructions on how to build and run your infrastructure and which design choices you made to structure it.

#### Virtual Machines

##### General overview

You will use VirtualBox and Vagrant to setup three different VMs in order to test the interactions and correctness of responses between your APIs infrastructure.

Vagrant is an open-source software that helps you create and manage virtual machines. With Vagrant, you can create a development environment that is identical to your production environment, which makes it easier to develop, test, and deploy your applications.

Your VMs will be structured as follow:

- `gateway-vm`: This VM will only contain the `api-gateway`.
- `inventory-vm`: This VM will contain the `inventory-app` API and the database `movies`.
- `billing-vm`: This VM will contain the `billing-app` API, database `orders` and RabbitMQ.

> Vagrant is designed for development and should not be used in production environments.

##### Environment variables

To simplify the building process it is a common practice to store useful variables in a `.env` file. It will make very easy to modify/update information like URLs, passwords, users and so on.

Those variables will then be used by Vagrant and passed through the different microservices in order to centralize all the credentials.

Your `.env` file should contain all the necessary credentials and none of the microservices should have any credential hardcoded in the source code.

> For the purpose of this exercise the `.env` file must be included in your repository, though usually in public and production projects the `.env` file is never included in repos to avoid sensitive data leaks.

##### Configuration of the VMs

- You will have a `Vagrantfile` which will create and start the three VMs. It will import the environment variables and pass them through each API.
- You will have a `script/` directory which will store all the scripts you may want to run in order to install the necessary tools on each VM. Those scripts may also be very useful for setting up the databases.

Your configuration will work properly for the following commands (executed from the root of the project):

- `vagrant up --provider virtualbox`: Starts all the VMs.
- `vagrant status`: Shows the status for all the VMs.
- `vagrant ssh <vm-name>`: Will let you access the VM through SSH.

#### Manage Your Node.js applications with PM2

PM2 is a process manager for Node.js applications that makes it easy to manage and scale your application.
It is designed to keep your application running continuously, even in the event of an unexpected failure.

PM2 can be used to start, stop, and list Node.js applications, as well as monitor their resource usage and log output.

Additionally, PM2 provides a number of features for managing multiple applications, such as load balancing and automatic restarts.

In our situation we will use it mainly to test resilience for messages sent to the Billing API when the API is not up and running. 

After entering in your VM via SSH you may run the following commands:

- `sudo pm2 list`: List all running applications.
- `sudo pm2 stop <app_name>`: Stop a specific application.
- `sudo pm2 start <app_name>`: Start a specific application.

#### Overall file structure

You can organize your internal file structure as you prefer. That said here is a common way to structure this kind of projects that may help you:

```console
.
├── config.yaml
├── .env
├── scripts
│   └── [...]
├── srcs
│   ├── api-gateway
│   │   ├── package.json
│   │   ├── proxy.js
│   │   ├── routes.js
│   │   └── server.js
│   ├── billing-app
│   │   ├── app
│   │   │   ├── config        // This is a directory with some .js files
│   │   │   ├── controllers   // This is a directory with some .js files
│   │   │   └── models        // This is a directory with some .js files
│   │   ├── package.json
│   │   └── server.js
│   └── inventory-app
│       ├── app
│       │   ├── config        // This is a directory with some .js files
│       │   ├── controllers   // This is a directory with some .js files
│       │   ├── models        // This is a directory with some .js files
│       │   └── routes        // This is a directory with some .js files
│       ├── package.json
│       └── server.js
└── Vagrantfile
```

When testing and before to automate it through the VM build you should be able to start the API Gateway and the two APIs by using the command `node server.js` inside their respective directories.

If you decide to use a different structure for your project remember you should be able to explain and justify your decision during the audit.

For clarity autogenerated files and directories like `node_modules` and `package-lock.json` have not been included in that tree.

> As a best practice it is strongly advised to add `node_modules` to your `.gitignore` in order not to upload useless files into your git repository (they will be autogenerated during the build process).