Date: Thu, 28 Mar 2024 15:08:45 +0000 (UTC)
Message-ID: <734554283.15.1711638525553@ip-172-31-8-223.us-west-2.compute.internal>
Subject: Exported From Confluence
MIME-Version: 1.0
Content-Type: multipart/related;
boundary="----=_Part_14_535800147.1711638525548"
------=_Part_14_535800147.1711638525548
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
Content-Location: file:///C:/exported.html
This is a planning / work breakdown page for driving the develop=
ment of a well defined efficient data serialization format for VSS data.
Background:
- VISS v2 has a comprehensive feature set, and therefore=
also defines a JSON message transfer format that is full =
featured.
- JSON is the default and popular way to encode data in many web (and oth=
er) environments. Whenever JSON is applicable then reusing the VISS d=
efinition is recommended.
- JSON is however text-based and somewhat verbose. Memory and bandwidth e=
fficiency is important in certain situations.
- Unlike VISS, many other protocols do not define the payload format.&nbs=
p; These require reuse of a defined format.
- Reuse VISS message definition in JSON
- ...or a new data serialization format.
- VISS v2 development have occasionally also discussed adding an alternat=
ive or "compressed" format in addition to VISS. The analysis/developm=
ent here might potentially influence also that -> leading to a VISS+bina=
ry encoding variation
- Either way, there is a clear driver towards developing a common standar=
d, and supporting tools, for serializing VSS data.
- Since this is not a new problem, a definition is likely to leverage exi=
sting binary serialization formats and simply document the message definiti=
ons that are applicable for data following the VSS data model.
- Additional rationale/background could be found in: Data serialization / value formats=
a>
Initial design thoughts:
- To complement the JSON approach, this is aiming for a highly space effi=
cient (binary) representation of data.
- It should likely reuse some framework to define the serialization. =
; Protobuf and AV=
RO are two popular choices:
- GRPC is a popular choice of RPC framework, and that may influence in th=
e direction of using protobuf also for data
- Protobuf is used in SENSORIS definitions
- AVRO is used in eSync Alliance and therefore =
a natural choice for our liaison / collaboration with eSync Alliance.
- AVRO is part of Apache technology stack, and thus fits=
well into Kafka, Spark, NiFi... that are also popula=
r data processing technologies.
Propos=
al: We should develop VSS data message descriptions in both Protobu=
f and AVRO.
Please note the difference between:
- Describing a VSS-formatted catalog of data in a new format, e.=
g. VSS(YAML)=E2=86=92JSON conversion that exists in vss-tools, or the =
VSS(YAML) =E2=86=92 Protobuf converter.
- Describing the payload format to transfer values that have bee=
n measured, for VSS-defined data items.
We are here talking about 2), not 1).
VSS has a limited set of data types (numbers represented as integers of =
different bit size, floating point types, strings, etc.) so the serializati=
on format of the data itself is likely trivial. But since the scope o=
f this work is to define useful payloads in protocol transfer, it =
includes more information than only the data value itself. Prominently, of =
course, value carrying messages almost always have a time stamp. Agre=
eing on and implementing the message format is required, even if leveraging=
encoding technologies built into protobuf, AVRO, and similar.
A full background and the suggested definition of several message types is =
here: Data=
serialization / value formats (please provide input/improvements=
there, so that it can be the blueprint for the development described on th=
is page.)
------=_Part_14_535800147.1711638525548
Content-Type: image/svg+xml
Content-Transfer-Encoding: 7bit
Content-Location: file:///C:/04def10815e0bd4ac50539c3c364c1c9