Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Update introduction text

...

Data formats sometimes overlap protocol definitions because some protocols (but not all) define the data format in the sense that a protocol, its specification.  VISS / W3C Gen2 is an example of a protocol definition that defines both the protocol interactions between client and server, and the data exchange format.  Ultimately, any chosen protocol or stack of protocols , must at some point clarify the used transferred data formats, or no understandable exchange can be had.
However, based on the protocols we are discussing, more is required. 
Looking at a widet set of protocols it is clear that we have some more work remaining.
The data exchange protocols we discuss fall into different categories, each requiring some more work on defining value exchange data formats:

  1. Protocol A protocol does not (yet) cover all variations of data exchange.
    1.  VISS / W3C Gen2 covers a lot of usage, but supports primarily fetching a single "latest value" measurement, or transferring instant updates according to subscriptions. But the bigger picture should include also exchanging sets of previously measured data between systems.  We can benefit from adding transfer of historically measured data, or derived statistics (e.g. get the average value over time instead of transferring all values), and more.  This work could help protocols like W3C "Gen 2" to cover these bases.
  2. Protocol A protocol defines only a "transport"
    1. We often discuss protocols that define some mechanisms of data transfer, such as pub/sub semantics, but they are designed to be generic and assume that any type of information can be transferred by the protocol.  Specifically, they do not strictly define the format of the content of the data container (payload).  This makes the protocols widely applicable, but is ultimately not enough without also defining the payload format.  Examples of some such protocols would be MQTT or WAMP
  3. Protocol A protocol defines transport, query semantics, and even some rules expectations for the exchanged data formatsformat, but is still too genericgeneric and requires additional definitions to become unambiguous for a particular case.
    1. Example:  GraphQL is a generic technology that is complemented by requires a schema that defines to be defined indicating the underlying data model and what types of queries can be made and the using the GraphQL language, including details such as the datatypes that are expected to be returned.   For that reason it is required to agree on the definition of schemas and types in a particular usage.  This  A schema must be defined for GraphQL, and for other similar situations, and that schema might also be derived from this generic analysis.
      Update: As a similar example, to consume and process data in Apache Spark, Kafka and presumably for many other generic data-handling frameworks,  we also need to define schemas that define the format and content of the transferred data, in a similar fashion.


...

Definitions

(proposal, open for discussion)

...