As a cloud consulting company, we at DNX work directly with many Fintechs and SaaS companies using AWS.

Many of these companies need to meet high levels of compliance  providing their software in a single-tenant architecture, where each of their customers has their own AWS account to guarantee full isolation between customers.

This raises a challenge for managing and monitoring these individual accounts, which can reach the hundreds.

The Business Challenge

The challenge is to build a centralised observability solution that would aggregate metrics, logs, and alarms (known as the three pillars of observability) with the data flowing privately within AWS.

In this example, we will consider a client where each tenant is using a stack with ECS Fargate with AppMesh and wishes to centralise the observability of these stacks in a cost-effective way. 

Objectives

The SaaS Observability proposal includes the following main objectives:

• Reduced Operation hours (having a centralised panel across all customers)
• Reduced Cost (having a observability backend stored in a centralised account)
• Quick response to alarms (automation to trigger alarms based on metrics across customers)
• Plugable (ability to add the observability strategy in the current stack)

The Solution

We started breaking the problem into 3 parts.

Tracing

We decided to try Jaeger, which is an open source, end-to-end distributed tracing. All we needed was to deploy a sidecar container with Jaeger running along with AppMesh envoy and configured 3 parameters for Envoy container configuration to send the tracing data to Jaeger. This works out of the box.

ENABLE_ENVOY_JAEGER_TRACING = 1
JAEGER_TRACER_PORT = 9411
JAEGER_TRACER_ADDRESS = 127.0.0.1

But another problem was coming; Jaeger supports different storage backends including memory, Cassandra, Elasticsearch, Kafka, and more. The recommended backend is Elasticsearch and we were also always looking to use the most managed services from AWS. Meaning that an Amazon Elasticsearch Service cluster should be deployed in the main account to store all tracing data from each tenant, but because we are trying to send requests between two different accounts, we ended up with a question; how do we make this link happen, given that there is no VPC endpoint support for ES into two accounts?

The answer was to create a proxy using a private API Gateway and a Lambda on the border of the main account. The lambda will take the request coming from Jaeger and just proxy to Elaticsearch adding AWS credentials headers to the request. On the tenant account we configured a VPC Endpoint to this API Gateway and limited access to specific VPCs (only tenant VPC can make requests to API Gateway).

Moving to the metrics we decided to go with AWS Distro for OpenTelemetry Collector (AWS OTel Collector), which is an AWS supported version of the upstream OpenTelemetry Collector and is distributed by Amazon. It supports the selected components from the OpenTelemetry community. It is fully compatible with AWS computing platforms including EC2, ECS, and EKS. It enables users to send telemetry data to Amazon Managed Service for Prometheus as well as the other supported backends.

Metrics

Like we did with Jaeger, we deployed aws-otel-collector as a sidecar container along with ECS services. The configuration was stored inside a SSM parameter folling the documentation https://aws-otel.github.io/docs/setup/ecs/config-through-ssm.

aws-otel-collector configuration

Usually a basic configuration is required to set up receivers, processors, service, and exporters.

For receivers, we used awsecscontainermetrics (memory, network, storage and CPU usage metrics), otlp, and statsd. Remember to open ports on the container definition or otherwise services will not be able to send metrics to the collector.

```yaml
receivers:
  awsecscontainermetrics:
    collection_interval: 20s
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:55681"
  statsd:
    endpoint: "0.0.0.0:8125"
    aggregation_interval: 60s
```

The processors specification section enables us to apply filters and tell how the metrics should be treated.

```yaml
processors:
  batch/metrics:
    timeout: 60s
  filter:
    metrics:
      include:
        match_type: strict
        metric_names:
          - ecs.task.memory.utilized
          - ecs.task.memory.reserved
          - ecs.task.cpu.utilized
          - ecs.task.cpu.reserved
          - ecs.task.network.rate.rx
          - ecs.task.network.rate.tx
          - ecs.task.storage.read_bytes
          - ecs.task.storage.write_bytes
```

The exporters is the most important configuration because is where we will use awsprometheusremotewrite to send the metrics to the main account.

```yaml
exporters:
  awsprometheusremotewrite:
    endpoint: "https://aps-workspaces.{{region}}.amazonaws.com/workspaces/{{workspace_id}}/api/v1/remote_write"
    namespace: "tenant_name"
    aws_auth:
      service: aps
      region: "{{region}}"
      role_arn: "arn:aws:iam::{{main_account_id}}:role/{{amp_role}}"
  logging:
    loglevel: debug
```

Notice that the role_arn we are pointing to a role on the main account, and when aws-otel-collector starts, it will assume the role that gives permissions to remote write to the Amazon Managed Prometheus. To do that, we need to:

1. 1. Assume role permissions to the trust relationship for the Task Role to the tenant ecs.

      ```json
      {
        "Version": "2008-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Principal": {
              "Service": "ecs-tasks.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
          }
        ]
      }
      ```
      

   2. The Task Role policy should point to the role that aws-otel-collector will try to assume inside the main account.

      ```json
      {
          "Version": "2012-10-17",
          "Statement": {
              "Effect": "Allow",
              "Action": "sts:AssumeRole",
              "Resource": "arn:aws:iam::{{main_account_id}}:role/{{amp_role}}"
          }
      }
      ```
      

   3. {{amp_role}} should contain permissions to reach the AMP cluster.

      ```json
      {
          "Statement": [
              {
                  "Action": [
                      "aps:RemoteWrite",
                      "aps:GetSeries",
                      "aps:GetLabels",
                      "aps:GetMetricMetadata"
                  ],
                  "Resource": "*",
                  "Effect": "Allow",
                  "Sid": "Prometheus"
              }
          ]
      }
      ```
      

   The last Otel configuration is the service that makes it possible to route what each receiver, processor, and exporter will do.

   ```yaml
   service:
     extensions: [health_check]
     pipelines:
       metrics/otlp:
         receivers: [otlp]
         processors: [batch/metrics]
         exporters: [logging, awsprometheusremotewrite]
       metrics/statsd:
         receivers: [statsd]
         processors: [batch/metrics]
         exporters: [logging, awsprometheusremotewrite]
       metrics/ecs:
         receivers: [awsecscontainermetrics]
         processors: [filter]
         exporters: [logging, awsprometheusremotewrite]
   ```
   

The diagram for the metrics solution:

Logs

For the logs we did a simple trick inside AWS that is enabling cross-account cross-Region CloudWatch logs.
Checkout https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Cross-Account-Cross-Region.html for more information.

Grafana

Once the otel-collector and jaeger were running, the last step was to deploy and set up Amazon Managed Service for Grafana inside the main account.

The deployment is simple but requires AWS SSO configured in the root account.

To setup you can go to:

1. AWS DataSources
2. Amazon Manager Service for Prometheus
3. Select region and click Add data source
   Then the visualisation we have in the dashboard is:

Conclusion

With the new observability stack built by DNX, they can now achieve a centralised storage and dashboard for metrics, tracing, and logs in an elastic and highly available way on AWS. The uncoupled solution enables hybrid configurations. This can accelerate development by finding improvements and bugs with alarms. We calculate that this architecture would bring a considerable cost reduction, reducing the usage of CloudWatch and XRay up to 70%!

Well, this is the magic that the CDK provides us. As there is a library behind all the methods and functions, it sees all the dependencies and automatically creates the missing resources for us, connecting them so that everything has a connection with as little access as possible, leaving what is necessary for the correct function between the resources. For example, the instance’s Security Group. As we marked that the EC2 instance listens on port 80, only port 80 will be added to the Security Group as an ingress value.

Photo by John Schnoon on Unsplash

Amazon MWAA (Managed Workflow for Apache Airflow) was released by AWS at the end of 2020. This brand new service provides a managed solution to deploy Apache Airflow in the cloud, making it easy to build and manage data processing workflows in AWS.

MWAA enables automatic deployment of all the infrastructure and configuration for Airflow Web Server, Scheduler, Workers, Metadata Database and also the Celery executor combined with SQS, to manage jobs dispatching. Users just need to setup an S3 bucket for DAGs, plugins and Python dependencies (via requirements.txt) and associate its content with the MWAA environment.

Authentication is also managed by AWS — native integration with IAM and resources can be deployed inside a private VPC for additional security.

Example of an Amazon MWAA architecture deployed inside a VPC

Since all the resources are deployed by AWS, developers don’t have access to the underlying infrastructure. Consequently, the main interface used by Data Engineers is the Airflow UI, which is available via public URL or VPC endpoint, depending on the deployment type selected (public or private network).

Example of DAG managed via Airflow UI

However, Airflow UI is not the only option for interacting with your environment; MWAA also provides support to the Airflow CLI. This is a useful option if you want to automate operations to monitor or trigger your DAGs, and in this post I explain how you can best make use of Airflow CLI from an MWAA environment.

The following content is suitable for those already familiar with the benefits and functionality of Apache Airflow. If you are new to Airflow, or you’re searching for more insights about the advantages of using Amazon MWAA compared to hosting your own environment, I recommend you explore earlier posts on this subject before reading on.

The Apache Airflow CLI and its use with Amazon MWAA

Airflow has a very rich command-line interface that allows for many types of operation on DAGs, starting services, and support for development and testing.

Airflow CLI is an interesting maintenance alternative within MWAA, since it allows Data Engineers to create scripts to automate otherwise manual/ repetitive tasks.

In MWAA, not all commands are supported because as developers we cannot perform operations that might impact server resources or user management (e.g. webserver, scheduler, worker, etc), but all commands related to monitoring, processing and testing DAGs are supported in the current version.

To check the full list of supported and unsupported commands, refer to the official User Guide. At the time of writing, this is the status of different commands:

List of supported commands

• backfill
• clear
• dag_state
• delete_dag
• list_dag_runs
• list_dags
• list_tasks
• next_execution
• pause
• pool
• render
• run
• show_dag
• task_failed_deps
• task_state
• test
• trigger_dag
• unpause
• variables
• version

List of unsupported commands

• checkdb
• connections
• create_user
• delete_user
• flower
• initdb
• kerberos
• list_users
• resetdb
• rotate_fernet_key
• scheduler
• serve_logs
• shell
• sync_perm
• upgradedb
• webserver
• worker

How to trigger a CLI command from Amazon MWAA

To access the Airflow CLI from MWAA, there are four basic steps:

1. Authenticate your AWS account via AWS CLI;
2. Get a CLI token and the MWAA web server hostname via AWS CLI;
3. Send a post request to your MWAA web server forwarding the CLI token and Airflow CLI command;
4. Check the response, parse the results and decode the output.

This sounds complicated but is actually a fairly straightforward process. Let’s deep dive and investigate each step in detail.

1. Authenticate to your AWS account

To access your MWAA cluster, you must install and configure AWS CLI, granting access to the account where your environment is deployed.

If you are not used to this process, read the AWS CLI User Guide, which explains how you can configure a profile in your AWS CLI and grant access to your accounts.

2. Get CLI token and MWAA web server hostname via AWS CLI

The next step is to collect a CLI Token which is a Bearer token used for authentication in your MWAA environment. When first authenticating in the AWS account we can also authenticate to the MWAA environment and collect the token which grants access to perform Airflow CLI commands, by entering the following command :

aws mwaa create-cli-token --name $MWAA_ENVIRONMENT

Remember to assign the name of your MWAA environment by exporting the environment variable $MWAA_ENVIRONMENT.

export MWAA_ENVIRONMENT=my_environment_name

If the command is successfully executed, you should receive a JSON response with two attributes:

{
  "CliToken" : "",
  "WebServerHostname" : ""
}

Parse the results and store them in other environment variables for later use. I suggest the following names:

• $CLI_TOKEN
• $WEB_SERVER_HOSTNAME

3. Send a post request to your MWAA web server forwarding the CLI token and Airflow CLI command

Finally, using the CLI token and the web server hostname, you can trigger your Airflow CLI command via curl request by following the example below:

curl \
  --request POST "https://$WEB_SERVER_HOSTNAME/aws_mwaa/cli" \
  --header "Authorization: Bearer $CLI_TOKEN" \
  --header "Content-Type: text/plain" \
  --data-raw "$AIRFLOW_CLI_COMMAND"

Notice we assigned the environment variables acquired from the previous step ($CLI_TOKEN and $WEB_SERVER_HOSTNAME) and also published a third variable with the name $AIRFLOW_CLI_COMMAND. In this variable we send the Airflow command to be performed by the CLI, for example, if you want to execute the following command:

airflow list_dags

The variable $AIRFLOW_CLI_COMMAND should be filled with:

list_dags

Important note: if your MWAA environment is published in a private network you can’t perform the curl request via public internet; a VPN must be used to establish the connection between your local machine and the VPC endpoint, or you may need to execute this command from another computing resource placed inside of the same VPC.

4. Check the response, parse the results and decode the output

Finally, the last step is to parse and decode the output of the curl request. If everything went well you should have received a JSON response with the following attributes:

{
  "stderr" : "",
  "stdout" : ""
}

Notice both attribute values are encoded in Base64. Remember to decode the results to collect the final output from Airflow CLI. A simple way to achieve that is by using the command:

base64 -d

Automating all the steps with a single script

In the previous sections we discussed all the steps needed to run a CLI command in MWAA, but now I’ll describe how to combine everything in a single script which enables you to quickly and easily perform CLI calls.

This is a shell script created for Unix based operational systems (e.g. Linux, MacOS); if you are running Windows you may need to adapt this content or run the script through Windows WSL (Windows Subsystem for Linux).

Notice I am using jq to parse the JSON responses from AWS CLI and the curl request to MWAA, but feel free to adapt the code if you prefer another approach.