Skip to main content

One post tagged with "temporal"

View All Tags

Build a robust logging system with Temporal and Parseable

· 6 min read
Shantanu Vishwanadha
Developer Relations

Picture this: You've built a workflow application using Temporal, and it's humming along nicely. But when things go sideways (as they inevitably do), you find yourself drowning in a sea of logs, desperately searching for that needle in the haystack. Sound familiar?

Today, we're diving into workflow logging, combining the power of Temporal with the flexibility of Parseable. By the end of this guide, you'll have a robust logging system that will make debugging feel less like archeology and more like time travel.

Let's get started.

Integrating Temporal and Parseable

Integrating Temporal with Parseable creates a robust system for workflow logging. Temporal handles the workflow orchestration, ensuring that tasks are executed in the right order and can recover from failures. Parseable captures and stores the logs generated by these workflows, providing a platform for analyzing and monitoring the logs.

With the integration of Temporal and Parseable, debugging workflows becomes more straightforward. Temporal's ability to manage state and retries, combined with Parseable's efficient log storage and querying capabilities, allows developers to quickly identify and resolve issues.

Setting up your environment

Before we dive into the nitty-gritty, let's make sure we have everything set up:

  1. Ensure you have Temporal installed and running (either locally or in a cluster, or you can use the Cloud version).
  2. Install Parseable. We'll use the Quick Start version for simplicity.
  3. Have your Temporal workflow application ready. This application will use Temporal to run tasks and workflows, and we'll set it up to send logs to Parseable.

Starting the Parseable server

Let's get Parseable up and running:

docker run -p 8000:8000 \
-v /tmp/parseable/data:/parseable/data \
-v /tmp/parseable/staging:/parseable/staging \
-e P_FS_DIR=/parseable/data \
-e P_STAGING_DIR=/parseable/staging \
containers.parseable.com/parseable/parseable:latest \
parseable local-store

This command pulls the latest Parseable image and runs it. It exposes port 8000 and creates a local volume for data persistence. This setup ensures that Parseable will store its data on your local machine and be accessible through port 8000 on your web browser.

Starting the Temporal server

Assuming your temporal server is installed as per previous steps, let's start the temporal server to create our first workflow

temporal server start-dev

The Temporal server is served on the URL localhost:7233 and you can access the Web UI at http://localhost:8233. The Temporal Web UI provides a comprehensive interface for monitoring and managing workflows, making it easier to visualize workflow states, debug issues, and understand the overall health of your system.

Next step, Let's get our Sample application cloned and running!

Setting up the Demo Workflow application

In this example, we'll be using the Typescript implementation. Feel free to choose any other favourable stack. Let's first clone the repository locally. Open your terminal and type the command:

git clone https://github.com/parseablehq/blog-samples.git
cd blog-samples/temporal-workflow-blog

You can choose any other stack that you are comfortable with. Cloning the repository gives us a pre-configured setup for a Temporal application integrated with Parseable. The example provided includes sample workflows and logging configurations, which can serve as a reference for your own applications. This step is crucial for getting hands-on experience with the integration process.

Installing the packages

Install the required node packages using the command.

npm install

Integrating Parseable-winston with our custom Logger

We'll be using the parseable-winston package to send forward the logs generated and captured by winston to the parseable instance.

You'll find these configuration in the logging.ts file in the repository. Winston is a versatile logging library for Node.js, offering various transports for log data, including files, databases, and third-party services like Parseable. By integrating parseable-winston, we ensure that logs are seamlessly forwarded to Parseable, allowing for centralized log management and analysis.

Inside the parseable function, you can change the following values as per your requirements in .env file:

  • PARSEABLE_LOGS_URL: Replace this with your Parseable URL. Example: https://demo.parseable.com/api/v1/logstream.
  • PARSEABLE_LOGS_USERNAME: Replace this with your username.
  • PARSEABLE_LOGS_PASSWORD: Replace this with your password.
  • PARSEABLE_LOGS_LOGSTREAM: Enter the logstream in which you want to ingest the workflow logs.

The environment variables allow for flexible configuration of the logging system.

Starting the Worker

The Worker Process is where Workflow Functions and Activity Functions are executed. To start the worker, type the command

npm run start.watch

This will initiate the worker and it will start generating logs and will interact with the workflow. Now, to run the workflow, open a separate terminal and type:

npm run workflow

Once you run the workflow, you should have the worker log output in your terminal. This two-step process ensures that the worker process and the workflow execution are handled independently, allowing for better management and debugging. The start.watch command starts the worker in watch mode, meaning it will automatically reload if there are changes in the code, which is particularly useful during development.

Log ingestion in Parseable

We have setup everything required to have the ingested logs in Parseable. Let's login to the parseable dashboard and check if the logs are being ingested or not. You can see the logs being ingested into your logstream in the parseable dashboard. The Parseable dashboard provides a user-friendly interface to search, filter, and analyze logs. By confirming that logs are being ingested correctly, you ensure that your logging setup is functioning as expected, enabling effective debugging and monitoring.

Additional tips

  1. Configuring Alerts: Set up alerts in Parseable to notify you of specific events or errors in your workflows.
  2. Custom Log Streams: Create custom log streams for different parts of your application to better organize your logs.
  3. Advanced Queries: Use Parseable's query language to filter and search logs based on various criteria, making troubleshooting easier.

Conclusion

Congratulations! You've just leveled up your Temporal workflow observability game. By integrating Parseable, you've created a powerful logging system that allows for easy querying, analysis, and alerting. Remember, this is just the tip of the iceberg. As you become more familiar with Parseable, you can create more application based ingestion and fine-tune your alerts to match your specific needs.

Effective logging and observability are crucial for maintaining the health and performance of your workflow applications. By following the steps outlined in this guide, you’ve taken a significant step towards achieving robust observability. With Parseable’s advanced features and Temporal’s powerful workflow management capabilities, you can ensure that your applications run smoothly and efficiently.

Get Updates from Parseable

Subscribe to keep up with latest news, updates and new features on Parseable