Introducing Epoch, a Real-Time Visualization Library

Running a high performance website requires insight through metrics, which means collecting a large amount of data. In a perfect world, this data is highly organized, easily queryable, and gives insight into the system as a whole. Unfortunately, data in the real world is often untidy, slow to access, and requires analysis before meaning can be derived.

While data organization can be solved by process, and query speed can be improved by optimization, deriving meaning via analysis and computation still remains an art. At Fastly, we’ve dabbled in this realm twice: first, with creating our real-time analytics dashboard, and again with the addition of the historical stats tool. Our goal with both features is to provide visualization tools that allow our customers to extract meaningful information from the large number of statistics that we collect on our network.

Today, we’re excited to announce that we’ve taken Fastly's stats tools one step further and distilled our visualization code into a stand-alone open source library. Introducing Epoch: a general purpose, real-time visualization library.

Motivations

We built Epoch to address our two primary stats-related needs: historical reporting and real-time monitoring. While superficially similar, each tool requires a reasonably different approach. For historical reports, we work primarily with large data sets that only change given user interaction:

Data Set AData Set B

For real-time monitoring, we start with much smaller data sets and receive new data points every second. The charts must smoothly transition to insert data into the visualization:

This dualism is fundamental to the way we designed Epoch and presents itself in code via our separation of basic and real-time charts. For large and static datasets, we provide a set of reporting-centric basic charts written atop d3: area, grouped-bar, line, pie, and scatter. We address real-time monitoring via HTML5 Canvas-based charts: area, bar, gauge, heat map, and line.

In addition to being conceptually clean, this also made it easy for us to focus on specific optimizations and features for each of general categories.

Using Epoch

We did our best to make Epoch easy to use when building large and complex stats interfaces. Adding charts to a page follows a basic three-step recipe.

Step 1: Add a sized container element, such as a div, to the page:

<div id="chart" class="epoch" style="height: 180px"></div>

Step 2: Format the incoming data to the format expected by the chart:

var sinLayer = {label: 'sin', values: []},
cosLayer = {label: 'cos', values: []}

for (var x = 0; x <= 2*Math.PI; x += Math.PI / 64) {
  sinLayer.values.push({ x: x, y: Math.sin(x) + 1 });
  cosLayer.values.push({ x: x, y: Math.cos(x) + 1 });
}

Step 3: Use jQuery to select the container element and invoke the .epoch method:

var chart = $('#chart').epoch({
  type: 'area',
  data: [sinLayer, cosLayer],
  axes: ['left', 'right']
});

The chart will then be instantiated, rendered, placed into the page, and automatically sized to fill its container, like this:

At this point you can modify the styles of any individual chart using pure CSS. These styles even work with the Canvas-based charts via Epoch’s built-in CSS Querying Engine.

Looking Forward

Epoch is currently in open beta (0.5.1 beta, to be exact). You can visit the newly launched project page to download the library, read the documentation, and learn how to contribute.

Over the next couple of months, we’ll be writing a few more posts detailing CSS querying engine and how to leverage Epoch’s class hierarchy to create new types of high performance visualizations. For a sneak peek on the series, check out my talk from this year’s ScotlandJS conference:

We hope you find the library useful. If you have any feedback, ideas for features, or find bugs, please let us know by opening up a Github issue.

Thanks for reading, and keep an eye out for the next installment.