What is it and why?

The VSS Taxonomy defines what data "entities" (Signals and Attributes) we can deal with, and are used in the protocol(s) defined by W3C Automotive Working group, as well as other initiatives inside and outside the vehicle.

But in addition to VSS itself, we need to define the data-exchange formats for measured values of those Signals.  This starts by defining terms, but quickly develops into defining one or several variants of the actual message content format, whether in JSON or other.

Relationship to Protocols

Data formats sometimes overlap protocol definitions in the sense that a protocol, or stack of protocols, must at some point clarify the used data formats, or no understandable exchange can be had.

However, based on the protocols we are discussing, more is required. 
The data exchange protocols we discuss fall into different categories, each requiring some more work on defining value exchange data formats:

1. Protocol does not (yet) cover all variations of data exchange.
   1.  VISS / W3C Gen2 covers a lot of usage, but supports primarily fetching a single "latest value" measurement, or transferring instant updates according to subscriptions. But the bigger picture should include also exchanging sets of previously measured data between systems.  We can benefit from adding transfer of historically measured data, or derived statistics (e.g. get the average value over time instead of transferring all values), and more.  This work could help protocols like W3C "Gen 2" to cover these bases.
2. Protocol defines only a "transport"
   1. We often discuss protocols that define some mechanisms of data transfer, such as pub/sub semantics, but they are designed to be generic and assume that any type of information can be transferred by the protocol.  Specifically, they do not strictly define the format of the content of the data container (payload).  This makes the protocols widely applicable, but is ultimately not enough without also defining the payload format.  Examples of some such protocols would be MQTT or WAMP
3. Protocol defines transport, query semantics, and some rules for data formats, but are still too generic.
   1. GraphQL is a generic technology that is complemented by a schema that defines what types of queries can be made and the datatypes that are expected to be returned.    For that reason it is required to agree on the definition of schemas and types in a particular usage.  This could also be derived from this generic analysis.

Definitions

(proposal, open for discussion)

Signal:

• An data entity in a Data Taxonomy, defined by its unique identifier
• In practice we here mean the absolute path of the node in a VSS taxonomy

Request:

• A request to deliver a measurement or measurements, as according to the chosen communication protocol.
• For example an instance of GET in VISS/W3C Gen2, or a Query in GraphQL, or ...
• This is assumed to request a set of data that has already been measured, (or if it is an instantaneous value, a value that is measured and delivered instantly).

Job:  

• A request to make measurements, typically some time in the future.
• Unlike Request, which asks for Data Delivery, a Job definition is used to instruct a system to perform a measurement or several measurements over time, at some time in the future.
• A Job does not deliver data until it is requested.
• A Job may include conditions such as a time-period, but in advanced systems also other logical conditions that should be fulfilled for the job to execute.
• TODO: Compare SENSORIS work on this.

Record:  (or Measurement, or Observation)

• Format to represent one single measured data value
• Subtypes (Record Types) indicate which Signal has been measured.
• Records are used to represent data with associated metadata.  Different metadata depending on the Record Type, as needed for different cases
• Example of possible record types:
  • Just the value.
  • The value plus a timestamp
  • The value plus a timestamp plus a timestamp accuracy information
  • The value plus additional qualitative information
• All of the above may also specify the signal name, or not:  Some record types may need to specify the signal it is referring to but most do not, because record is delivered in a context where it is known which signal is being measured.

Why not just use a single record type (superset of all functionality)?  

The reason would be to optimize the performance and bandwidth.  In other words, don't transfer what is not needed for a certain case.  If a timestamp is not needed, we should make sure we support transferring data without providing a timestamp, for example.

Record Subtypes: 
PlainRecord, TimeStampedRecord, ToleranceTimeStampedRecord, ... *

+ Record types which specify the signal name inside:  
SpecifiedPlainRecord, SpecifiedTimeStampedRecord, etc.

Measurement, "Container" – naming of this? 

(i.e. the message that is in response to a Request)

=   A delivery of data sent at a particular time.

Might need to include some metadata regarding the request:

• Job ID?
• Sequence number (if partial delivery of a Job
• The values container (type Snapshot, Bundle, or single Record)
   

Discussion: What is the general metadata needed when measuring data?

• Vehicle identity – should this be special or is just a measurement on a VSS attribute.  For example VIN number is already defined in VSS.  Presumably it could be just a VSS defined data item?
• <Benjamin to add what more metadata is needed>

Bundle:

• Several measured values, of the same signal, taken over a period of time.
• A Bundle contains time stamp for each value
• A Bundle is a collection of Records

Snapshot:

• Several measured values, of different signals, that have a relationship to each other because they were measured "at the same time"
  (which, due to potential time sync limitations or timestamp inaccuracy this "same time" could be defined as a time range of a chosen length)
• A snapshot is built up of several Records, and additional information
• A Job would likely define beforehand, which different signals shall be grouped into the Snapshot instances.
• Clarify if snapshot should have only one measurement per signal.
• Explain timestamp inaccuracy...

Note that values in a Snapshot need a record type that specifies the signal, since different signals are included in the same message.

Stream:

• Continuous delivery of data points according to a predefined agreement
• Not only a different container type.  It deals more with the delivery method (constant stream compared to atomic message)
• Does not have a fixed start/end time
• Delivers a stream of Records
• + could also deliver side-band information (e.g. Job information)
• Is analogous to the delivery of a Subscription for protocols that support subscriptions.

Examples, in JSON

PlainRecord:

{
   "value" : " 100.54"
}

TimestampedRecord:

{
   "ts" : "20200319123245Z123456" # Zulu time ISO std + microsecond - FIXME:format is not checked
   "value" : "42 "
}

SpecifiedRecord:

{
   "signal" : "vehicle.Chassis.Axle2.WheelCount"
   "value" : "2"
}

SpecifiedTimestampedRecord:

{
   "signal" : "vehicle.Body.ExteriorMirrors.Heating.Status"
   "ts" : "20200319123245Z123456"
   "value" : "false"
}

Bundle:

{ 
    "signal"  :  "vehicle.body.cabin.temperature"
    "count"   : "132"  # Might be redundant information, optional.
    "values" : {
        { 
            "ts" : "20200319123245Z123456"
            "value" : "42 "
        },
        { 
            "ts" : "20200320120101Z100000"
            "value" : "43"
        },
        ... 130 more records
    }
}

Snapshot:

{ 
    "timeperiod" { 
        "start" : "xxx",
        "end" : "xxx"
    },
    "values" : {
     {
        "signal" : "vehicle.body.cabin.temperature",
        "value" : "22",
        "ts" : "20200320120101Z100000"
     },
     {
        "signal" : "vehicle.drivetrain.engine.rpm.average",
        "value" : "3200",
        "ts" : "20200320120101Z100000"
     }
}