事物序列化_大规模测量每件事物m3时间序列简介

2022-03-14 00:00:00 序列 数据 文件 节点 时间

事物序列化

Today, it’s easy to say that almost everything we do, everything we use, and even everything around us is capable of producing data. But what is even more true, is that this data is produced in real-time to describe something that is happening.

如今,可以很容易地说,我们所做的几乎所有事情,我们使用的所有东西,甚至我们周围的一切都能够产生数据。 但是更真实的是,这些数据实时生成的用于描述正在发生的事情。

Therefore, it’s logical to think that data must be also harnessed in real-time to be able to extract the most value from it. In addition, and perhaps most importantly, data must be stored and processed with a temporal context to retain its full significance. This is actually the condition necessary to fully understand the context in which something exists or occurred.

因此,逻辑上认为还必须实时利用数据才能从中获取大价值。 另外,也许是重要的是,必须在时间上下文中存储和处理数据,以保持其全部意义。 实际上,这是充分理解存在或发生某事的上下文所必需的条件。

So, let’s take some real-life examples where the temporal context (i.e., time) is an essential part of the meaning of your data:

因此,让我们举一些真实的例子,其中时间上下文(即时间)是数据含义的重要组成部分:

  • Recording sports performance metrics (i.e. speed, position, heart rate) during a sporting activity through a connected watch.

    通过连接的手表记录体育活动过程中的运动表现指标(即速度,位置,心律)。
  • Measuring atmospheric conditions to provide data for weather forecasts (wind speed, temperature, atmospheric pressure, etc).

    测量大气条件以提供天气预报数据(风速,温度,大气压力等)。
  • Monitoring of a server’s system resource usage.

    监视服务器的系统资源使用情况。
  • Monitoring of a home’s energy consumption.

    监视房屋的能源消耗。
  • Monitoring stock prices, etc.

    监视股票价格等

All of these examples have one thing in common: they are all about data that we want to measure over time to monitor their evolution, to detect or predict trends (maybe in correlation with other events), or to alert on thresholds. We more commonly refer to these data as time-series.

所有这些示例有一个共同点:它们都是关于我们要随时间测量以监视其演变,检测或预测趋势(可能与其他事件相关)或警告阈值的数据。 我们通常将这些数据称为时间序列

The explosion of the IoT (Internet of Things) in recent years has greatly accelerated the need to be able to efficiently store and analyze this data, which most often means millions of new metrics produced every second.

物联网(爆炸物联网)近年来大大加快了需要能够有效地存储和分析数据,其中大部分往往意味着数以百万计的新的标准生产的每一秒。

什么是时间序列,什么是时间序列数据库(TSDB)? (What is a time-series and what is a time-series database (TSDB) ?)

Time-series are sequences of numeric data points that are generated in successive order. Each data point represents a measure (also called a metric). Each metric has a name, a timestamp, and usually one or more labels that describe the actual object being measured.

时间序列是按连续顺序生成的数字数据点的序列。 每个数据点代表一个度量(也称为度量)。 每个度量标准都有一个名称,一个时间戳记,通常还有一个或多个描述实际测量对象的标签。

To store such data we could perfectly use a traditional relational database (such as PostgreSQL) and create a simple SQL table like this :

为了存储此类数据,我们可以完美地使用传统的关系数据库(例如PostgreSQL)并创建一个简单SQL表,如下所示:

CREATE TABLE timeseries ( metric_name TEXT NOT NULL, metric_ts timestamptz NOT NULL DEFAULT CURRENT_TIMESTAMP, value double precision NOT NULL, labels json, PRIMARY KEY(metric_name, metric_ts)  );

And, for example, to query and aggregate every point from now to the last 10 minutes we could use a SQL query similar to :

并且,例如,要查询和汇总从现在到后10分钟的每个点,我们可以使用类似于以下内容SQL查询:

SELECT avg(value) FROM timeseries WHERE metric_name = ‘heart_rate_bpm’ AND metric_ts >= NOW() — INTERVAL10 minutes’;

However, this solution would not be really effective for>但是,此解决方案对于数据密集型应用程序和长期使用而言并不会真正有效。 迟早我们可能会受到以下限制:

  • The horizontal scalability capabilities, whether for long-term storage, resiliency, or multi-region deployment needs.

    水平可伸缩性功能,无论是针对长期存储,弹性还是多区域部署需求。
  • The ability to massively insert millions of metrics per second (most relational databases are based on B-TREE index structures).

    每秒大量插入数百万个指标的能力(大多数关系数据库基于B-TREE索引结构)。
  • The ability to automatically roll-up data over time. For example, to aggregate all metrics from the previous month into 5-minute points).

    随着时间的推移自动汇总数据的能力。 例如,将上个月的所有指标汇总为5分钟)。

Also, there are likely hotspots when inserting very high throughput measurements. This can lead to poor performance, depending on the type of index used by the database, due to concurrent accesses.

此外,插入非常高的吞吐量测量值时可能会出现热点。 由于并发访问,这可能导致性能下降,具体取决于数据库使用的索引类型。

For all of these reasons, it‘s usually preferable to use solutions that are specifically designed to enable efficient storage and querying of this kind of data. These solutions are called time-series databases (TSDB).

由于所有这些原因,通常好使用专门设计的解决方案,以实现对此类数据的有效存储和查询。 这些解决方案称为时间序列数据库(TSDB)。

Below are some of the most known TSDB :

以下是一些的TSDB:

  • InfluxDB

    InfluxDB

  • TimescaleDB

    时标数据库

  • OpenTSDB

    OpenTSDB

Finally, there are also other very popular solutions such as Prometheus and Graphite which are sometimes (perhaps wrongly) compared to TSBDs because of their ability to store time series. But they are actually monitoring systems that use features similar to those of TSDBs for storing metrics.

后,还有其他非常流行的解决方案,例如Prometheus 石墨 由于它们可以存储时间序列,因此与TSBD相比有时(可能是错误的)。 但是它们实际上是监视系统,其使用与TSDB类似的功能来存储度量。

In this article, we will focus on a more recent solution: M3DB a distributed time series platform.

在本文中,我们将重点介绍新的解决方案: M3DB 分布式时间序列平台。

M3,一个分布式时间序列数据库 (M3, A distributed time-series database)

M3 is a distributed time-series platform that was developed by Uber to meet its growing storage and access needs for the trillions of metrics that the platform generates every day around the world.

M3 是一个分布式时间序列平台,由Uber开发,以满足其平台每天在全球产生的数万亿个指标的不断增长的存储和访问需求。

The M3 platform is available in open-source under the Apache v2.0 license since 2018 on GitHub. It is developed entirely in Go and has been designed to be able to scale horizontally in order to support both high throughput writes and low-latency queries.

M3的平台是在Apache许可证2.0版开放源代码,因为2018上可用GitHub上。 它完全在Go中开发,并且已设计为能够水平扩展,以支持高吞吐量写入低延迟查询

The M3 platform provides key features that make it a complete and robust solution for storing and processing time-series data:

M3平台提供的关键功能使其成为用于存储和处理时间序列数据的完整而强大的解决方案:

  • Cluster Management: M3 is built on top of etcd to provides support for handling multiple clusters out of the box.

    集群管理:M3建立在etcd之上 提供对开箱即用处理多个群集的支持。

  • Built-in replication: Time-series data points are replicated across nodes with tunable configuration to achieve the desired balance between performance, availability, durability, and consistency.

    内置复制:时间序列数据点在具有可调配置的节点之间复制,以在性能,可用性,耐用性和一致性之间实现所需的平衡。

  • Highly Compressed: M3 provides an efficient compression algorithm inspired by Gorilla TSZ.

    高度压缩:M3提供了一种受大猩猩TSZ启发的高效压缩算法。

  • Configurable Consistency: M3 supports different consistency levels for both write and read requests (i.e: One, Quorum, All).

    可配置的一致性:M3支持写入和读取请求的不同一致性级别(即:一个,法定人数,全部)。

  • Out of order writes: M3 can seamlessly handle out-of-order writes for a configurable period.

    序写入:M3可以在可配置的时间内无缝处理乱序写入。

  • Seamless Prometheus Integration: M3 has built-in supports PromQL and can be used as a Prometheus Long-term Storage

    无缝Prometheus集成:M3内置支持PromQL,可以用作Prometheus长期存储

All these features are offered by different components that make up the M3 platform: M3 DBM3 Coordinator, M3 Queries and M3 Aggregator.

所有这些功能都是由组成M3平台的不同组件提供的: M3 DB , M3协调器, M3查询M3聚合器

Now, let us now take a closer look at these four components.

现在,让我们仔细看一下这四个组成部分。

M3组件概述 (M3 Components Overview)

M3数据库(M3 DB)

M3DB is the actual distributed time-series database that provides durable and scalable storage as well as reverse indexes for time-series.

M3DB是实际的分布式时间序列数据库,可提供持久且可扩展的存储以及时间序列的反向索引。

M3 DB relies on etcd for clustering-management and provides synchronous replication with configurable durability and read consistency (one, majority, all, etc).

M3 DB依赖etcd 用于集群管理,并提供具有可配置的持久性和读取一致性(一个,多数,全部等)的同步复制。

M3协调员 (M3 Coordinator)

M3 Coordinator is the service, part of the M3 platform, dedicated to the coordination of reads and writes in M3DB between upstream systems. For example, it can act as a bridge with Prometheus (or other systems such as Graphite). In addition, M3 Coordinator is used as a global service to configure other components of the platform.

M3协调器是M3平台的一部分,该服务专用于上游系统之间的M3DB读写协调。 例如,它可以作为与Prometheus(或其他系统,如Graphite)的桥梁。 此外,M3协调器还用作全局服务来配置平台的其他组件。

相关文章