Modules¶

livemetrics¶

This module implements a utility to collect (efficiently) business metrics showing the activity of an application.

For testing, we will reduce the tick interval

>>> import livemetrics.metrics
>>> livemetrics.metrics.Meter.TICK_INTERVAL = 0.1

First you need to create an object and to send an event:

>>> lm = LiveMetrics("version","about",lambda: True,lambda: True)
>>> lm.mark('start','ok')
>>> lm.mark('start','error')

This is enough to get a first set of metrics:

>>> import pprint
>>> pprint.pprint(lm.get_metrics()) 
{'start': {'error': {'count': 1,
...
                     'rate1': 0.0,
                     'rate15': 0.0,
                     'rate5': 0.0},
           'ok': {'count': 1,
...
                  'rate1': 0.0,
                  'rate15': 0.0,
                  'rate5': 0.0}}}

We can now emit more events:

>>> import time
>>> time.sleep(0.02)
>>> lm.mark('start','ok')
>>> time.sleep(0.03)
>>> lm.mark('start','ok')
>>> lm.mark('start','error')
>>> time.sleep(0.2)
>>> round(lm.get_metrics()['start']['ok']['rate1'],2)
29.95
>>> round(lm.get_metrics()['start']['error']['rate1'],2)
19.97
>>> m = lm.get_metrics()
>>> m['start']['error']['rate1']<m['start']['error']['rate5']
True
>>> m['start']['error']['rate5']<m['start']['error']['rate15']
True
>>> m['start']['error']['rate1']<m['start']['ok']['rate1']
True
>>> m['start']['ok']['rate1']>29.0
True
>>> m['start']['error']['rate1']>19.0
True

For gauges, here is an example:

>>> lm = LiveMetrics("version","about", lambda: True, memory_and_cpu=False)
>>> lm.gauge('G',10)
>>> lm.gauge('G',15)
>>> lm.gauge('G',20)
>>> lm.gauge('G',12)

>>> print(lm.get_gauges())
{'G': {'min': 10, 'max': 20, 'count': 12}}

>>> print(lm.get_gauges('G'))
{'min': 10, 'max': 20, 'count': 12}

>>> print(lm.get_gauges('G','max'))
20

This module also provides a decorator to encapsulate a meter and a histogram. Example:

>>> lm = LiveMetrics("version","about",lambda: True)
>>> @lm.timer('decorated','OK','EXC')
... def for_test(value,exc=False):
...     if exc: raise Exception("Exception")
...     time.sleep(0.1)
...     return value

Then, every time the function is called, new metrics are collected:

>>> for_test(1)
1
>>> for_test(2)
2
>>> for_test(3,True)
Traceback (most recent call last):
...
Exception: Exception

Metrics can be accessed using LiveMetrics.get_metrics():

>>> pprint.pprint(lm.get_metrics()) 
{'decorated': {'EXC': {'count': 1,
...
                       'rate5': 0.0},
               'OK': {'count': 2,
...
                      'rate5': 10.0}}}

>>> pprint.pprint(lm.get_metrics('decorated')) 
{'EXC': {'count': 1, 'mean': 0.0, 'rate1': 0.0, 'rate15': 0.0, 'rate5': 0.0},
 'OK': {'count': 2,
...
        'rate5': 10.0}}

>>> M = lm.get_metrics('decorated','OK')
>>> M['count']
2
>>> round(M['rate5'],1)
10.0

>>> lm.get_metrics('decorated','OK','count')
2

Event must be specified if result and metric are defined: >>> lm.get_metrics(None,’OK’,’count’) Traceback (most recent call last): … SyntaxError: if metric/result is specified, event must also be specified >>> lm.get_metrics(‘decorated’,None,’count’) Traceback (most recent call last): … SyntaxError: if metric is specified, result must also be specified

Or LiveMetrics.get_histograms():

>>> H = lm.get_histograms()
>>> H['decorated']['count']
3
>>> round(H['decorated']['quantiles'][0.05],1)
0.1
>>> round(H['decorated']['stddev'],2)
0.05

>>> H = lm.get_histograms('decorated')
>>> H['count']
3
>>> round(H['quantiles'][0.05],1)
0.1
>>> round(H['stddev'],2)
0.05

>>> pprint.pprint(lm.get_histograms('decorated','count')) 
3

>>> math.isclose(lm.get_histograms('decorated','quantiles')[0.05],0.05,rel_tol=0.01)
True

>>> pprint.pprint(lm.get_histograms('decorated','distribution',scale=1)) 
[3]

class livemetrics.LiveMetrics(version, about, is_healthy, is_ready=None, memory_and_cpu=True)¶

Bases: object

An object to be used as a singleton to aggregate all live metrics of an application.

version: a string giving the version of the application

about: a string describing the application

is_healthy: a function called to know if the application is healthy or not. Function must have no parameter and return a boolean.

is_ready: a function called to know if the application is ready to process requests or not. Function must have no parameter and return a boolean. If None is provided, is_healthy is used.

memory_and_cpu: a flag to activate gauges to report the memory (memory), number of threads (num_threads) and CPU usage (cpu) on Linux only. Default is True.

Added in version 0.7: is_healthy and is_ready can be coroutine if the publisher is aiohttp.

gauge(name, value)¶

Register a new gauge for this name. If value is a callable, it will be called everytime the gauge value is accessed.

Return the Gauge object.

Added in version 0.7: The Gauge object is returned.

get_gauges(name=None, metric=None)¶

Return a structure of dictionaries with the gauge metrics.

name: the name of the gauge

metric: the name of the metric, one of min, max, count

If metric is not provided, all metrics are returned.

If name is not provided, all gauges are returned.

get_histograms(event=None, metric=None, percentiles=[0.05, 0.25, 0.5, 0.75, 0.95], scale=10)¶

Return a structure of dictionaries with the histograms metrics.

event: the name of the event

metric: the name of the metric, one of count, min, max, mean, stddev, quantiles, distribution

percentiles: the list of percentiles for which the quantiles are returned

scale: the resolution (number of values) of the returned distribution

If metric is not provided, all metrics are returned.

If event is not provided, all histograms are returned.

get_metrics(event=None, result=None, metric=None)¶

Return a structure of dictionaries with the requested metrics.

event: the name of the event

result: the result of the event, as used in the call to mark()

metric: the name of the metric, one of mean, count, rate1, rate5, rate15.

If metric is not provided, all metrics are returned.

If result is not provided, all results are returned.

If event is not provided, all events are returned.

histogram(name, value)¶: Register a new value in a histogram and return the Histogram object.

Added in version 0.7: The Histogram object is returned.

mark(event, result)¶: Mark the execution of an event event with the result.

timer(event, ok, error)¶

Decorator to automate meter and histogram on one event.

event: the name of the event

ok: value when the function decorated terminates successfully

error: value when the function decorated terminates with an exception

For each call of the decorated function, a call to mark() is done for the event and result (ok or error), and a call to histogram() is made to measure the processing time.

ok and error can be callables that return a string compatible with json dictionary key

Warning

If applied on an async function decorated with aiohttp web.RouteTableDef(), it must be placed between the aiohttp annotation and the async function. Otherwise the aiohttp route will be decorated and not the actual function.

livemetrics.metrics¶

class livemetrics.metrics.EWMA(nb_minutes=1, interval=5)¶

Bases: object

An exponentially-weighted moving average. See http://en.wikipedia.org/wiki/Moving_average#Exponential_moving_average

Usage:

>>> e = EWMA(1)
>>> e.update()
>>> import time
>>> time.sleep(1)
>>> e.update()
>>> e._tick()
>>> e.rate  
0.39999...
>>> e._tick()
>>> e.rate  
0.368017...

update(n=1)¶: Indicate an event has happened. n specifies the number of events.

class livemetrics.metrics.Gauge(value=None)¶

Bases: object

A simple counter, registering a value and statistics (min and max) on this value.

>>> g = Gauge(10)
>>> g.mark(12)
>>> g.min,g.count,g.max
(10, 12, 12)

>>> g.mark(8)
>>> g.min,g.count,g.max
(8, 8, 12)

>>> g = Gauge()
>>> g.mark(12)
>>> g.min,g.count,g.max
(12, 12, 12)

>>> g.mark(8)
>>> g.min,g.count,g.max
(8, 8, 12)

>>> g.mark(20)
>>> g.min,g.count,g.max
(8, 20, 20)

>>> g.mark(15)
>>> g.min,g.count,g.max
(8, 15, 20)

The alias value can also be used:

>>> g.min,g.value,g.max
(8, 15, 20)

Any time, the value can be provided as a callable:

>>> import random
>>> g.mark(lambda: random.randrange(21,50))
>>> g.count>20
True

The value can also be a callable from the start

>>> g = Gauge(lambda: random.randrange(1,20))
>>> g.mark(10)
>>> g.min>=0,g.max<20
(True, True)

property count¶: Return the current value of the gauge.

mark(value)¶: Register a new value in the gauge and update the statistics.

property max¶: Return the maximum value ever registered in this gauge.

property min¶: Return the minimum value ever registered in this gauge.

property value¶: Alias on count().

class livemetrics.metrics.Histogram¶

Bases: object

An histogram represents the distribution of a set of values. Values are exponentially decayed so that more recent values have more weight than old values.

For testing, we will reduce the tick interval managing the decaying of values.

>>> Reservoir.TICK_INTERVAL = 0.1

Usage:

>>> his = Histogram()

A new object has no value yet:

>>> his.count
0

For the test, we Let’s add 100 sequential values:

>>> for i in range(100):
...    his.update(i+1)

The snapshot object gives access to some statistics:

>>> round(his.snapshot.mean,1)
50.5
>>> round(his.snapshot.stddev,1)
28.9

And to the value for a percentile:

>>> round(his.snapshot.get_value(0.25),1)
26.5

>>> his.snapshot.get_distribution()
[10, 10, 10, 10, 10, 10, 10, 10, 10, 10]

>>> his.snapshot.get_distribution(20)
[5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5, 5]

If we wait a little and inject new values:

>>> import time
>>> time.sleep(1.0)
>>> for i in range(100):
...    his.update(i*2)

>>> his.snapshot.get_distribution(20)
[14, 15, 15, 15, 15, 15, 15, 15, 15, 15, 6, 5, 5, 5, 5, 5, 5, 5, 5, 5]
>>> round(his.snapshot.get_value(0.25),1)
34.0

>>> time.sleep(1.0)
>>> for i in range(100):
...    his.update(i*2)
>>> his.snapshot.get_distribution(20)
[19, 20, 20, 20, 20, 20, 20, 20, 20, 20, 11, 10, 10, 10, 10, 10, 10, 10, 10, 10]
>>> round(his.snapshot.get_value(0.25),1)
38.0

The size of the reservoir is by default 1024. When the number of values is over this limit, values are randomly removed, older values first.

>>> his50 = Histogram()
>>> his50._reservoir._size = 50
>>> for i in range(100):
...    his50.update(i+1)

>>> his.snapshot.mean!=his50.snapshot.mean
True
>>> his.snapshot.stddev!=his50.snapshot.stddev
True

property count¶: Get the number of values recorded by this histogram.

property snapshot¶: Get a WeightedSnapshot for this histogram. The snapshot will then give access to quantiles and to the distribution of values.

update(value)¶: Register a new value in the histogram.

class livemetrics.metrics.Meter¶

Bases: object

An object measuring the rates of occurence of an event. Each event is reported with a call to mark(). The rates are decayed as time goes by.

For testing, we will reduce the tick interval managing the decaying of values.

>>> Meter.TICK_INTERVAL = 0.1

Then we create a new object:

>>> mtr = Meter()

At start, it is empty:

>>> mtr.mean
0.0

Then if we mark an event:

>>> mtr.mark()

Mean is still 0 because there is no enough time difference to calculate it.

>>> mtr.mean
0.0

Let’s move forward in time:

>>> import time
>>> time.sleep(0.1)

and report a second event:

>>> mtr.mark()
>>> mtr.mean>17 and mtr.mean<25
True
>>> mtr.count
2
>>> mtr.mark()
>>> time.sleep(0.5)
>>> 10.0-mtr.rate1<0.1,10.0-mtr.rate5<0.1,10.0-mtr.rate15<0.1
(True, True, True)

Rates decreases as time goes:

>>> time.sleep(1.0)
>>> 9.8-mtr.rate1<0.1,9.9-mtr.rate5<0.1,10.0-mtr.rate15<0.1
(True, True, True)

We marked 3 times in 2.1 seconds

>>> round(mtr.mean,1)
1.9

property count¶: Return the number of events reported to this object since it was created.

mark()¶: Indicate an event has just occurred.

property mean¶: The number of events divided by the time the application is running. This gives a very rough estimation of the average number of events per seconds.

property rate1¶: Return the 1-minute moving average rate.

property rate15¶: Return the 15-minute moving average rate.

property rate5¶: Return the 5-minute moving average rate.

class livemetrics.metrics.Reservoir¶

Bases: object

A reservoir holds the data of an Histogram. It records values and decays them as time passes. It keeps only a statistical representative set of values.

class livemetrics.metrics.WeightedSnapshot(values)¶

Bases: object

A (weighted) snapshot, i.e. a copy of a set of values that have a weight. The weight is used to calculate the quantiles.

This class is used by Histogram and returned by Histogram.snapshot

Example:

>>> snapshot = WeightedSnapshot( {i:WeightedSample(i,i) for i in range(10)})

Access the statistics of the values:

>>> print((snapshot.min,snapshot.max,round(snapshot.mean,1),round(snapshot.stddev,1)))
(0, 9, 6.3, 2.2)

>>> print(snapshot.get_value(0.5))
7.5

>>> print(snapshot.get_value(-1))
Traceback (most recent call last):
...
Exception: argument -1 is not in [0..1]

>>> print(snapshot.get_value(2))
Traceback (most recent call last):
...
Exception: argument 2 is not in [0..1]

>>> print(snapshot.get_distribution(5))
[2, 2, 2, 2, 2]

>>> print(snapshot.size)
10

The snapshot can also be empty:

>>> snapshot = WeightedSnapshot( {})
>>> print((snapshot.min,snapshot.max,round(snapshot.mean,1),round(snapshot.stddev,1)))
(0, 0, 0, 0)

>>> print(snapshot.get_value(0.5))
0.0

get_distribution(range_value=10)¶: Return the distribution with the request resolution (i.e. number of values)

get_value(quantile)¶

Access the value (the quantile) for a given percentile, i.e. the value that defines the boundary between the two segment [O,percentile] and [percentile,1].

The percentile must be between 0 and 1.

property max¶: Maximum value in this snapshot.

property mean¶: Weighted average of the values in this snapshot.

property min¶: Minimum value in this snapshot.

property size¶: Number of values in this snapshot.

property stddev¶: Standard deviation of the weighted values in this snapshot.

livemetrics.dashboard¶

This module provides a (very) minimal application to demonstrate the usage of metrics.

Install with:

pip install livemetrics[dashboard]

Run with:

livemetrics-dashboard -i myconf.yaml

And then access http://localhost:9000/ to visualize the dashboard.

The configuration file describes rows, and for each row the cells to display. Each cell is associated to one value (or to the statistics of one value for histograms).

The configuration file is a Jinja2 template used to generate a YAML file. All macros and filters of Jinja2 are available, making possible to define template or macros for complex cases.

A simple example:

{% set server = "http://localhost:7070/monitoring/v1" %}
---
title: Sample
rows:
- label: First Line
  cells:
  - label: Line 1
    type: label
  - label: Health
    server: "{{server}}/is_healthy"
    type: status
  - label: CPU
    server: "{{server}}/metrics/gauges/cpu/count"
    type: value
    unit: "%"
    factor: "*100"
    color: green
  - label: Memory
    server: "{{server}}/metrics/gauges/memory/count"
    type: gauge
    unit: MB
    factor: "/1048576"
    max: 256
    gauge_color: blue
    color: "#C0D0F0"
  - type: empty
  - label: Time
    server: "{{server}}/metrics/histograms/time"
    type: histogram
    unit: ms
    factor: "*1000"
    color: "#ff000080"
    median_color: green
  - label: Test
    server: "{{server}}/metrics/meters/test/ok/rate1"
    type: gauge
    unit: "/h"
    factor: "*3600"
    gauge_color: green
    color: "#C0F0C0"
    max: 25000

The cells can be of the following types:

Type	Description	Options
`empty`	Display an empty cell	N/A
`label`	Display a static text	`label`: The text to display
`status`	Display a boolean value as a colored icon	`label`: A text displayed below the icon `server`: The full URL used to get the boolean value
`value`	Display a numeric value as a text	`label`: A text displayed below the value `server`: The full URL used to get the value or a list of URL `unit`: A small text added after the value `factor`: A factor to apply to the value. It must start with `*` or `/` to indicate the operation to execute `precision`: the number of digit after the decimal separator in the output. Ex: `1`, `2`, etc. `color`: The text color `operator`: in case a list of URL is provided, the operation to perform on the list of values. One of `sum`, `average`, `max` or `min`.
`gauge`	Display a numeric value as a gauge and a sparkline showing the recent evolution of the value	`label`: A text displayed below the value `server`: The full URL used to get the value or a list of URL `unit`: A small text added after the label `factor`: A factor to apply to the value. It must start with `*` or `/` to indicate the operation to execute `color`: The fill color of the sparkline `gauge_color`: The color of the gauge `max`: The maximum value of the gauge `operator`: in case a list of URL is provided, the operation to perform on the list of values. One of `sum`, `average`, `max` or `min`.
`histogram`	Display statistics about a value as a whisker box	`label`: A text displayed below the value `server`: The full URL used to get the statistics. It must return a json dictionary with values for the keys [“0.05”, “0.25”, “0.75”, “0.95”, “0.5”] `unit`: A small text added as the legend of the diagram `factor`: A factor to apply to the value. It must start with `*` or `/` to indicate the operation to execute `color`: The fill color of the box `median_color`: The color of the line showing the median value

Publishers¶

Publishers are classes that expose the live metrics through a REST interface.

livemetrics.publishers.http¶

This module provides a http.server.BaseHTTPRequestHandler subclass exposing a livemetrics.LiveMetrics object.

A minimal application would be:

import livemetrics
import livemetrics.publishers.http

import http.server

LM = livemetrics.LiveMetrics(version,about,is_healthy,is_ready)

class Sample(livemetrics.publishers.http.HTTPRequestHandler):

    @LM.timer("sample","ok","error")
    def do_GET(self):
        if self.path=='/sample':
            self.send_response(200)
            self.end_headers()
        else:
            return super().do_GET()

httpd = http.server.HTTPServer((host,port), lambda r,a,s: Sample(LM,r,a,s))
httpd.serve_forever()

class livemetrics.publishers.http.HTTPRequestHandler(LM, request, client_address, server)¶

Bases: BaseHTTPRequestHandler

Subclass this class in your application to inherit the handling of GET requests exposing a livemetrics.LiveMetrics object.

livemetrics.publishers.aiohttp¶

This module provides aiohttp routes exposing a livemetrics.LiveMetrics object.

A minimal application would be:

import livemetrics
import livemetrics.publishers.aiohttp

from aiohttp import web
routes = web.RouteTableDef()

LM = livemetrics.LiveMetrics(version,about,is_healthy,is_ready)

@routes.get("/sample")
@LM.timer('sample','ok','error')
async def sample(request):
    return web.Response(status=200)

app = web.Application()
app.add_routes(livemetrics.publishers.aiohttp.routes(LM))   # register the routes to publish the metrics
app.add_routes(routes)                                      # register the routes of the application
web.run_app(ap,host,port)                                   # run the application

livemetrics.publishers.aiohttp.routes(LM)¶

Return a list of routes to be registered in the aiohttp application.

LM: a livemetrics.LiveMetrics object

livemetrics.publishers.flask¶

This module provides a flask blueprint exposing a livemetrics.LiveMetrics object.

A minimal application would be:

import livemetrics
import livemetrics.publishers.flask

from flask import Flask
app = Flask(__name__)

LM = livemetrics.LiveMetrics(version,about,is_healthy,is_ready)

@app.route("/sample")
@LM.timer('sample','ok','error')
def sample():
    return "Sample!"

app.register_blueprint(livemetrics.publishers.flask.blueprint(LM))

livemetrics.publishers.flask.blueprint(LM)¶

Return a blueprint with the routes and view_func for the publication of the metrics.

LM: a livemetrics.LiveMetrics object

livemetrics.publishers.django¶

This module provides Django views exposing a livemetrics.LiveMetrics object.

A minimal application would be:

import livemetrics
import livemetrics.publishers.django

import django
from django.http import HttpResponse
from django.urls import path

# Declare a global LiveMetrics object in one of your modules
LM = livemetrics.LiveMetrics(version,about,is_healthy,is_ready)

@LM.timer('sample','ok','error')
def sample_view(request):
    return HttpResponse("sample")

In your urls.py, register your views with:

urlpatterns = [
    path('sample', sample_view, name='sample'),
]

urlpatterns += livemetrics.publishers.django.urlpatterns(LM)

livemetrics.publishers.django.urlpatterns(LM)¶

Return a list of Django paths to be registered in the application.

LM: a livemetrics.LiveMetrics object

Modules¶

livemetrics¶

livemetrics.metrics¶

livemetrics.dashboard¶

Publishers¶

livemetrics.publishers.http¶

livemetrics.publishers.aiohttp¶

livemetrics.publishers.flask¶

livemetrics.publishers.django¶

livemetrics

Navigation

Related Topics