Logs support now available in AWS Distro for OpenTelemetry

/ Tutorial

Introduction

In recent years, containerization has become a popular choice for deploying applications in the cloud. AWS, being one of the leading cloud service providers, offers two major container orchestration services – Amazon Elastic Kubernetes Service (EKS) and Amazon Elastic Container Service (ECS). These services provide a scalable and reliable environment for running containerized workloads.

However, gaining visibility into the logs generated by these containerized workloads has always been a challenge. As applications become more distributed and complex, monitoring and understanding the logs becomes crucial for troubleshooting issues, optimizing performance, and ensuring compliance.

To address these challenges, AWS has introduced Logs support in AWS Distro for OpenTelemetry (ADOT). This new feature enables customers to collect all telemetry data, including application logs, from their containerized AWS workloads running in EKS and ECS. In this guide, we will explore the new capabilities of ADOT and discuss how it can empower you to gain deep insights into your applications’ logs.

Table of Contents

  1. Overview of AWS Distro for OpenTelemetry (ADOT)
  2. Introduction to Logs support in ADOT
  3. Architecture of Logs collection with ADOT
  4. Configuring ADOT in an EKS cluster
  5. Configuring ADOT in an ECS cluster
  6. Leveraging ADOT with Syslog and Log4j
  7. Enriching logs with metadata using OpenTelemetry’s semantic conventions
  8. Integrating ADOT with observability backends
  9. Securing logs collection with ADOT
  10. Best practices for using Logs support in ADOT
  11. Troubleshooting common issues with ADOT
  12. Conclusion

1. Overview of AWS Distro for OpenTelemetry (ADOT)

AWS Distro for OpenTelemetry (ADOT) is a distribution of the OpenTelemetry project, which is a widely adopted observability framework for collecting, processing, and exporting telemetry data from applications. ADOT provides a unified and consistent way to instrument and collect telemetry data from various sources such as metrics, traces, and logs.

ADOT includes a collector component, responsible for collecting telemetry data, and an exporter component, responsible for exporting the collected data to various observability backends. With the introduction of Logs support, ADOT now enables customers to collect application logs seamlessly.

2. Introduction to Logs support in ADOT

Logs support in ADOT allows customers to collect application logs from their containerized AWS workloads running in EKS and ECS. This support brings several benefits, including:

  • Centralized log collection: ADOT provides a standardized and centralized way to collect logs from multiple sources within your containers.

  • Compatibility with popular log formats: ADOT supports popular log formats, such as Syslog and Log4j, making it easy to integrate with your existing logging infrastructure.

  • Metadata enrichment: ADOT leverages OpenTelemetry’s semantic conventions to enrich logs with contextual metadata. This enables better correlation between logs, metrics, and traces, improving your ability to troubleshoot and analyze application behavior.

  • Seamless integration with observability backends: ADOT supports exporting logs to various observability backends, allowing you to choose the tooling that best suits your needs.

3. Architecture of Logs collection with ADOT

To understand how logs collection with ADOT works, let’s dive into its architecture. The process involves three main components:

  • Application: This is your containerized application running in EKS or ECS, generating logs that need to be collected.

  • ADOT Collector: The ADOT collector runs as a daemon alongside your application. It intercepts all log outputs from the application and sends them to the configured backend.

  • Backend: The observability backend is where the logs are received, stored, and made accessible for analysis, monitoring, and troubleshooting. Examples of popular observability backends include AWS CloudWatch Logs, Grafana Loki, and Elastic Stack (also known as the Elasticsearch-Logstash-Kibana stack).

When logs support is enabled in ADOT, the collector component is enhanced to include the Filelog receiver and the AWS CloudWatch Logs exporter. The Filelog receiver ensures all logs generated by your application are captured, while the AWS CloudWatch Logs exporter provides a seamless integration with AWS CloudWatch Logs, a highly scalable and managed log storage and analytics service.

4. Configuring ADOT in an EKS cluster

ADOT can be easily configured in an EKS cluster using the EKS add-on. The EKS add-on simplifies the deployment and management of essential observability components, including ADOT, for collecting logs.

To configure ADOT in an EKS cluster, follow these steps:

  1. Install and configure the EKS add-on for observability.

  2. Enable logs support in ADOT by updating the configuration to include the Filelog receiver and the AWS CloudWatch Logs exporter.

  3. Deploy the ADOT collector as a DaemonSet in your EKS cluster, ensuring it runs alongside your applications in each node.

  4. Verify logs collection is functioning correctly by checking the logs in AWS CloudWatch Logs or your preferred observability backend.

5. Configuring ADOT in an ECS cluster

If you are running your containerized workloads in ECS, you can also configure ADOT to collect logs effectively. The process involves the following steps:

  1. Set up an ECS task definition with ADOT configured as a sidecar container.

  2. Deploy your task definition to an ECS cluster, ensuring the ADOT container runs alongside your application containers.

  3. Configure the ADOT container to capture logs by utilizing the Filelog receiver and exporting them to the desired observability backend.

  4. Monitor the logs in your observability backend to validate the successful collection.

6. Leveraging ADOT with Syslog and Log4j

ADOT’s versatility shines when it comes to working with various log formats, such as Syslog and Log4j. These formats are commonly used in many applications and integrations.

To leverage ADOT with Syslog:

  1. Configure your application to generate logs in Syslog format.

  2. Enable the Syslog receiver in the ADOT collector configuration.

  3. Configure the exporter to send logs to your observability backend of choice.

  4. Verify that logs generated in Syslog format are seamlessly collected and available for analysis.

To leverage ADOT with Log4j:

  1. Ensure your application is using Log4j for logging.

  2. Configure Log4j to output logs to the standard output.

  3. Ensure the ADOT collector is capturing logs from the standard output.

  4. Export the collected logs to your observability backend.

  5. Validate the successful ingestion of Log4j logs by reviewing them in your selected observability tool.

7. Enriching logs with metadata using OpenTelemetry’s semantic conventions

OpenTelemetry’s semantic conventions provide a set of guidelines and attributes for enriching telemetry data, including logs, with contextual metadata. By adhering to these conventions, you can improve the correlation between logs, metrics, and traces, enabling more effective analysis and troubleshooting.

When using ADOT, you can benefit from the out-of-the-box metadata enrichment based on OpenTelemetry’s semantic conventions. This means that ADOT will automatically add relevant metadata to your logs, such as the application name, host information, and request identifiers.

For example, if your application includes an HTTP request, ADOT will automatically append attributes like “http.method” and “http.url” to the captured log entries. This enrichment facilitates searching and filtering logs based on specific criteria, ultimately enhancing your ability to gain insights into your application’s behavior.

8. Integrating ADOT with observability backends

ADOT provides flexibility when it comes to exporting logs to observability backends. While AWS CloudWatch Logs is the default choice due to its tight integration with the AWS ecosystem, you can export logs to other popular tools such as Grafana Loki and Elastic Stack.

To integrate ADOT with AWS CloudWatch Logs:

  1. Update the ADOT collector configuration to include the AWS CloudWatch Logs exporter.

  2. Ensure the necessary IAM permissions are granted to the ADOT collector to write logs to CloudWatch Logs.

  3. Verify the successful export of logs by checking the logs in AWS CloudWatch Logs.

To integrate ADOT with Grafana Loki:

  1. Configure the ADOT collector to use the Loki exporter.

  2. Update the configuration to point to your Grafana Loki instance.

  3. Validate the logs are successfully exported to Grafana Loki.

To integrate ADOT with Elastic Stack:

  1. Configure the ADOT collector to use the Elasticsearch exporter.

  2. Provide the necessary connection details for your Elasticsearch cluster.

  3. Verify the successful export of logs to Elasticsearch using Kibana.

By leveraging ADOT’s flexibility in exporting logs, you can choose the observability backend that aligns best with your existing tooling and requirements.

9. Securing logs collection with ADOT

Security is a critical aspect of any log collection solution. ADOT provides several mechanisms to ensure secure log collection from your containerized workloads.

You can enhance the security of logs collection with the following practices:

  • Enable encryption: ADOT supports encrypting data in transit using secure transport protocols such as HTTPS. By enabling encryption, you ensure that logs travel securely from your application to the collector and then to the observability backend.

  • Restrict access: Implement fine-grained access controls using IAM policies to control who can access and manage the ADOT collector. This helps prevent unauthorized access to sensitive logs.

  • Monitor audit logs: Enable monitoring and alerting on ADOT’s audit logs to detect any suspicious activities or potential security breaches. Regularly reviewing these logs can help you proactively identify and mitigate security risks.

By following these security practices, you can ensure the confidentiality, integrity, and availability of your logs throughout the collection process.

10. Best practices for using Logs support in ADOT

To make the most out of Logs support in ADOT, consider the following best practices:

  • Define a log retention policy: Determine the necessary retention duration for your logs based on compliance requirements, troubleshooting needs, and available storage. Regularly review and update this policy to optimize costs and ensure compliance.

  • Implement log rotation: Configure log rotation mechanisms within your application to prevent logs from consuming excessive storage. This ensures efficient storage usage and improves the performance of log analysis.

  • Leverage log aggregation: Use ADOT’s ability to aggregate logs from multiple sources to gain a holistic view of your application’s behavior. Aggregation enables you to detect patterns and anomalies across different components and environments.

  • Apply log filtering: Fine-tune log filtering mechanisms to reduce noise and focus on relevant log entries. Filtering enables you to focus on critical information, making troubleshooting faster and more efficient.

  • Monitor log ingestion rate: Implement monitoring and alerting on the ingestion rate of logs to ensure the collector can handle the expected load. This helps detect any potential bottlenecks and ensures uninterrupted log collection.

By following these best practices, you can optimize your logs collection process and effectively utilize the capabilities of ADOT.

11. Troubleshooting common issues with ADOT

Despite its robustness, ADOT may encounter issues that can impact log collection. Being prepared to troubleshoot common issues is essential to maintain an efficient logs collection pipeline.

Some common issues you may encounter include:

  • Connectivity issues: Ensure that the ADOT collector can reach the observability backend and that the necessary network configurations are in place.

  • Configuration errors: Double-check the configuration of the ADOT collector, including the receiver and exporter settings, to rule out any misconfigurations.

  • Permissions: Verify that the IAM roles and permissions are correctly set up to grant the necessary access for the ADOT collector and the observability backend.

  • Performance problems: Monitor the resource utilization of the ADOT collector and scale it up if necessary to handle the log ingestion rate.

  • Log format mismatches: Confirm that the log formats generated by your application match the configuration of the ADOT collector, avoiding any parsing or ingestion issues.

By systematically troubleshooting these common issues, you can ensure a smooth logs collection experience with ADOT.

12. Conclusion

AWS Distro for OpenTelemetry’s new Logs support adds a powerful capability to collect application logs from containerized AWS workloads running in EKS and ECS. By leveraging this feature, you can gain deep insights into your application’s behavior, troubleshoot issues effectively, and optimize performance.

In this guide, we explored the architecture of logs collection with ADOT, learned how to configure ADOT in both EKS and ECS clusters, and discussed how to leverage popular log formats such as Syslog and Log4j. We also explored the benefits of metadata enrichment using OpenTelemetry’s semantic conventions and discussed the flexibility of integrating ADOT with various observability backends.

To ensure the security and reliability of your logs collection process, we discussed best practices, security considerations, and troubleshooting strategies.

With the Logs support in ADOT, AWS continues to provide comprehensive tools for observability, enabling you to gain deep insights into your applications running in a containerized environment. By following the techniques and recommendations outlined in this guide, you can unlock the full potential of ADOT for log collection and analysis.