Skip to main content

Build Clickstream Data Infrastructure with Winston and Parseable

· 7 min read
Abhishek Sinha
Solution Architect

Clickstream data are a series of individual events. These individual events are sequenced to do analytics and draw a variety of insights. To build a dependable analytics, you need to build reliable infrastructure and pipeline to record each event. You can build a robust clickstream data infrastructure using React, Nodejs, Winston, and Parseable.

Server side logging

Clickstream data can be logged either from the client-side or server side. Is there a preferred mode for sending logs? Client side logging sends the data directly from the browser or from within the app. But companies have reported loss up to 30% events when sent from the frontend. This happens because different browsers, ad blockers, manufacturer settings, connectivity issues etc hinders in sending event logs consistently.

Server side logging is reliable. Server can comfortably handle a large number of requests per second and log them all to the storage. Every time a user interacts on the web page or on the app screen it triggers an API to the backend. We use each API call to log user interaction to the clickstream database. Hence, you get comprehensive, reliable, and accurate clickstream and user behavior analytics when we log from the backend server.

But there is a category of events that can only be captured from the frontend. For example, mouse hover, scroll, and other user interactions. So, like everything else in life, there is a trade-off. Refer to our other article in this series to see how to log events from the frontend using React and Parseable.

In this article, we will focus on logging events from the backend. We will use Nodejs, Winston, and Parseable to create a robust clickstream data infrastructure.

Auto capture clickstream

First we need to capture user interaction. Auto capture script or library can capture all user events on the frontend. We will capture every event, fetch data, and create a log object to send it to the server.

Along with event data, we can also collect metadata. For example if you have more than one signup button, one in the header and another in the footer. In our example we add custom attribute “data-component-name=“HeaderSignUpButton” to the button. This hook will also fetch custom-attributes, and send it to the logger service along with event data.

Our logger service send the log object to the backend. To keep it simple, we will send logs through the HTTP request. You can use HTTP request safely, but at high volume, prefer to use bi-directional API.

Getting started

As explained, we'll use Nodejs, Winston, and Parseable to create a customer data infrastructure. Winston is the most popular nodejs logging library. It’s highly adaptable and offers flexible log formatting and transportation options. We use Winston to collect logs and store it in Parseable.

Parseable is developer friendly log storage and analytics tool. Like Winston, Parseable is adaptable and user friendly. It supports dynamic schema so that you can send log objects without any restriction. It comes with a powerful SQL based query engine, live tail, table data with filter, alerting and a lot more features.

At a high level, these are the steps to be followed to capture clicks and send the log to the backend:

  1. Create globalEventListener Hook to auto capture all events
  2. Add global event listener at the root
  3. Create a button with custom attributes
  4. Send logs to the backend

Auto capture all events

Create globalEventListener Hook to auto capture all events


//useGlobalEventTracker.js

import { useEffect } from "react";
import { useLocation } from "react-router-dom";

import packageJson from "../../package.json";

const useGlobalEventListener = (loggerService) => {
const logger = new loggerService();

const location = useLocation();
useEffect(() => {
const captureClickEvent = (event) => {
let element = event.target;
// Traverse up the DOM tree until we find an element with the data-component-name attribute
while (element && !element.dataset.componentName) {
element = element.parentElement;
}
let componentName = "NA";
if (element) {
componentName = element.dataset.componentName;
}

const { innerText, className } = event.target;

const metadata = {
event: "click",
component: componentName,
class: className,
content: innerText,
appVersion: packageJson.version,
url: location.pathname,
};

logger.trackEvent("click", metadata);
};

document.addEventListener("click", captureClickEvent);

return () => {
document.removeEventListener("click", captureClickEvent);
};
}, []);
};

export default useGlobalEventListener;

Add global event listener at the root

import Home from "../src/pages/Home";
import { LoggerService } from "./logger/logger-service";
import useGlobalEventListener from "./hooks/useGlobalEventTracker";

function App() {
useGlobalEventListener(LoggerService);
return <Home />;
}

export default App;

Create a button

Set custom attribute on button to pass additional metadata. For example we have a send event button in the header. We are setting component name for example to the HeaderSendButton

<button data-component-name="“HeaderSignUpButton”">Click to Send Event</button>

Send logs to the backend

We are using Axios to make http request


// logger-service.js

import axios from "axios";

const backendURL = "Your Backend URL"; // Replace with your backend

export class LoggerService {
trackEvent(eventType, payload) {
axios({
method: "POST",
baseURL: backendURL,
url: `/logger`,
headers: {
"x-source": "ReactApp", // Your custom data (optional)
"x-event-type": eventType, // Custom header params (optional)
},
data: payload,
});
}
}

Events from NodeJs backend service to Parseable

Now that the backend server has the events, let's see how to send these event streams to Parseable. We will use Winston to log events to Parseable.

Create a Parseable transport service

As a pre-requisite, you'll need to have Parseable installed and running. Refer the documentation here to install Parseable. Once done, keep these details of Parseable ready to configure the transport:

  • URL of your Parseable instance
  • Username & Password to access Parseable
  • Stream name where you want to send these events

// parseable-transport.js

const axios = require("axios");
const Transport = require("winston-transport");

const BASE_URL = "https://demo.parseable.com"; //your parseable instance

const auth = Buffer.from("username:password").toString("base64"); //
const streamName = "clickstream";

class WinstonParseableTransport extends Transport {
constructor(opts) {
super(opts);
this.name = opts.name || "customTransport";
}

log(info, callback) {
console.log("info", info);
const config = {
method: "POST",
baseURL: BASE_URL,
url: `/api/v1/logstream/${streamName}`,
headers: {
Authorization: `Basic ${auth}`,
"Content-Type": "application/json",
},
data: {},
};

axios(config)
.then(function (response) {
console.log(
`Custom Transport:sent with status with code ${response.status}`
);
})
.catch(function (error) {
console.log("axios error", error);
});
callback();
}
}

module.exports = WinstonParseableTransport;

Create a Winston logger instance to handle events & logs

// packages/logger.js

const { createLogger, format, transports } = require("winston");
const WinstonParseableTransport = require("./parseable-transport");

const logger = createLogger({
level: "info",
format: format.combine(
format.timestamp({
format: "YYYY-MM-DD HH:mm:ss",
}),
format.errors({ stack: true }),
format.splat(),
format.json()
),
transports: [
new transports.File({ filename: "error.log", level: "error" }),
new transports.File({ filename: "combined.log" }),
new WinstonParseableTransport({ name: "customTransport" }),
],
});

if (process.env.NODE_ENV !== "production") {
logger.add(
new transports.Console({
format: format.combine(format.colorize(), format.simple()),
})
);
}

module.exports = logger;

Add API Endpoint

This is to process frontend events to parseable backend.

// Example: Logging events using Winston

const express = require("express");
const path = require("path");
const logger = require("./logger-utils/logger");

const app = express();
const port = 4444;

// Middleware
app.use(express.json());
app.use((req, res, next) => {
// To handle global events on request (Optional)
// logger.info(`${req.method} ${req.url}`);
next();
});

// Handle React Frontend App rendering
app.use(express.static(path.join(__dirname, "../frontend/build")));

// API to handle log process
app.post("/logger", (req, res) => {
const { body = {} } = req;
const ip = req.ip;
const userAgent = req.get("User-Agent");
logger.info({
...body,
userAgent,
host: ip,
});
res.json({
message: "Log Triggered successfully!",
});
});

app.listen(port, () => {
console.log(`Express app listening at http://localhost:${port}`);
});

Summary

We have implemented on frontend a react hook to auto-capture all user interaction and a logger service to send all these events to the server. At the server we collect, and store all data using winston, and Parseable. At last, parseable gives you a console to query data, and visualize any report.

The code is available on Github.

Get Updates from Parseable

Subscribe to keep up with latest news, updates and new features on Parseable