What is it and why?

The VSS Taxonomy defines what data "entities" (Signals and Attributes) we can deal with, and are used in the protocol(s) defined by W3C Automotive Working group, as well as other initiatives inside and outside the vehicle.

But in addition to VSS itself, we need to define the data-exchange formats for measured values of those Signals.  This starts by defining terms, but quickly develops into defining one or several variants of the actual message content format, whether in JSON or other.

Relationship to Protocols

Data formats sometimes overlap protocol definitions because some protocols (but not all) define the data format in its specification.  VISS / W3C Gen2 is an example of a protocol definition that defines both the protocol interactions between client and server, and the data exchange format that fits the VSS model.  Ultimately, any chosen (stack of) protocols must at some point define the transferred data formats, otherwise no understandable exchange can be had, and this page is intended to support the development of such a definition.

Looking at a wider set of protocols it is clear that we have some more work remaining.
The data exchange protocols we discuss fall into different categories, each requiring some more work on defining value exchange data formats:

1. A protocol does not (yet) cover all variations of data exchange.
   1.  When this page was written, W3C VISS protocol (v2) supported subscription to updates and on-demand fetching of the current value of one or several, specified signals in one go.  In its latest form it has a significant number of query parameters and filters, and supports the fetching of a series of historical recorded values (i.e. TimeSeries according to the definitions below).  VISS v2 now specifies features that mirror most of the types of messages listed here, with the exception of Snapshots.
2. A protocol defines only a "transport"
   1. We often discuss protocols that define some behavior of data transfer, such as pub/sub semantics, but they are designed to be generic and therefore support any type of information to be transferred by the protocol.  This means they do not (can not) define the format of the content of the data container (payload), and frequently are set up to transfer just any arbitrary sequence of bytes.   This makes those technologies widely applicable, but choosing them is not enough without also defining the payload format.  Examples of some such protocols would be MQTT or WAMP, but the principle extends to many "generic" protocols or frameworks.
3. A protocol defines transport, query semantics, and even a few expectations for the exchanged data format, but is still generic and requires additional definitions to become unambiguous for a particular case.
   1. Example:  GraphQL is a generic technology that clarifies a bit more about expected data semantics and formats but still requires a schema to be defined to indicate the exact underlying data model, what types of queries can be made using the GraphQL language, and other details such as the datatypes that are expected to be returned.   A schema must be defined for GraphQL, and for other similar situations, and that schema might also be derived from this generic analysis.
   2. Example 2:  To consume and process data in Apache Spark, Kafka and presumably for many other generic data-handling frameworks we also need to define schemas that define the format and content of the transferred data, in a similar fashion.  These protocols might also match category 2/3.

Related references

• MF4/MD4 data format
  • A C++ implementation

Definitions

(proposal, open for discussion)

Signal:

• An data entity in a Data Taxonomy, defined by its unique identifier
• In practice we here mean the absolute path of the node in a VSS taxonomy

Request:

• A request to deliver a measurement or measurements, as according to the chosen communication protocol.
• For example an instance of GET in VISS/W3C Gen2, or a Query in GraphQL, or ...
• This is assumed to request a set of data that has already been measured, (or if it is an instantaneous value, a value that is measured and delivered instantly).

Job:  

• A request to make measurements, typically some time in the future.
• Unlike Request, which asks for Data Delivery, a Job definition is used to instruct a system to perform a measurement or several measurements over time, at some time in the future.
• A Job does not deliver data until it is requested.
• A Job may include conditions such as a time-period, but in advanced systems also other logical conditions that should be fulfilled for the job to execute.
• TODO: Compare SENSORIS work on this.

Observation:

• The act of making a measurement, a.k.a. to record one or several values for a particular named data item.

Data Package:

=   A delivery of data sent at a particular time.

(think of it as the whole Message that is in response to a Request)

This will likely need to include some metadata regarding the request:

• Vehicle identity – should this be special or is just a measurement on a VSS attribute.  For example VIN number is already defined in VSS.  Presumably it could be just a VSS defined data item?
  • Depending on the specific implementation of this concept this might be an "anonymous" ID instead of one that can identify a vehicle or person.
• Job ID (when applicable)
• Sequence number (if partial delivery of a Job)
• The values container itself (type Snapshot, Bundle, or single Record) 

Additional Metadata

Following input given in the W3C data TF:

• Signal metadata
  • Quality
  • Sampling/Compression methods
  • Transmission method
• Usage metadata
  • Data sensitivity
  • Retention time
  • Consent requirements
    

Record:

• A format to represent one single measured data value
• Subtypes (Record Types) indicate which Signal has been measured.
• Records are used to represent data with associated metadata.  Different metadata depending on the Record Type, as needed for different cases
• Example of possible record types:
  • Just the value.
  • The value plus a timestamp
  • The value plus a timestamp plus a timestamp accuracy information
  • The value plus additional qualitative information
• All of the above may also specify the signal name, or not:  Some record types may need to specify the signal it is referring to but most do not, because record is delivered in a context where it is known which signal is being measured.

Why not just use a single record type (superset of all functionality)?  

A: The reason would be to optimize the performance and bandwidth.  In other words, don't transfer what is not needed for a certain case.  If a timestamp is not needed, we should make sure we support transferring data without providing a timestamp, for example.

Record Subtypes:

• PlainRecord ← there is no need to differentiate it from Record unless we want the top parent type (Record) to be "abstract" and only allow subtypes be concrete types.  
  As far as I can see there is nothing to gain from that.  So we can consider Record to be a concrete parent type and  Record == Plain Record
• TimeStampedRecord
• ToleranceTimeStampedRecord, ... ?

+ Record types which specify the signal name inside: 

• SpecifiedPlainRecord
• SpecifiedTimeStampedRecord, etc.

Record types which specify the geospatial position in addition to the time value:

• GeospatialRecord
• SpecifiedGeospatialRecord  

(N.B. Geospatial records also always include time stamp, because it seems to be the overwhelmingly dominant usage)

DerivedRecord and StatisticsRecord

• This is a record type that does not deliver the original data but something that is calculated from it.
• These are  needed only if the  "derived" signal is not already listed in the VSS signal database.  In other words, it is perfectly possible to use VSS to define some derived or statistical value already.  A normal request for that VSS signal would then deliver the value in a normal record, and there is no need to define in the value-measurement protocols how this  value is calculated since it is declared in the VSS description or simply decided by the system that delivers it.   
  Here is one example of this already listed in VSS today:   Vehicle.AverageSpeed.   It has the description "Average speed for the current trip".  You could imagine defining another signal named Vehicle.Speed.MonthlyAverage, for example and just fetch it as a normal Record.
• But as a complement to predefined "hard coded" statistical signals inside VSS ,  a DerivedRecord type such as statistical record is useful since it can have modifiable parameters, such as over which time period is the measurement done.
• Subtypes of DerivedRecord are StatisticsRecord and maybe some others (e.g. mathematical function / curve matching?)
• Further Subtypes of StatisticsRecord:  Average, Median, Max, Min, Histogram, and so on.  

TimeStamp

NOTE:  The exact format of all of these may differ when these concepts are translated to different protocols or languages.
As a starting point however, let's propose to use ISO 8601 standard format, with fractional seconds (e.g. microseconds) and always UTC (Zulu) time zone, if a TimeStamp is given in text format.  For other purposes such as binary formats, more efficient encodings should be considered. 

Real, "Wall clock time"

•    "ts" : "2020-01-10T02:59:43.492750Z # Zulu time, ISO std with microseconds

Relative to a previously predefined time stamp reference:

• "rts" : "T02:59:43.100044"    # Similar to ISO 8610.  Years/month/dates can be omitted if zero
• "rts" : "02:59:43.100044"      # Alternative, also OK
• "rts2", "rts3", ...                      # If more than one relative time stamp reference had been previously agreed

Bundle

• A collection of several records, transferred together.
• Subtypes are TimeSeries and Snapshot.

TimeSeries

• Several measured values, of the same signal, taken over a period of time.
• A TimeSeries contains time stamp for each value
• A TimeSeries is a collection of Records

Snapshot:

• Several measured values, of different signals, that have a relationship to each other because they were measured "at the same time"
  (which, due to potential time sync limitations or timestamp inaccuracy this "same time" could be defined as a time range of a chosen length)
• A snapshot is built up of several Records, and additional information
• A Job would likely define beforehand, which different signals shall be grouped into the Snapshot instances
• A Snapshot can be a generalisation of the "Freeze Frame" concept used in automotive diagnostics.
  • Side note: Freeze Frames are sometimes delivered as an opaque data dump that can only be interpreted by those who know the internal structure but we are here proposing an understandable and readable format for Snapshot.    (With that said, these are so far general concepts and could be mapped to any data representation, only a proposal/example given here)

    • (and it is also implied by the example, that the snapshot only contains values from signals that exist in the VSS database, because each is identified by path)

• Clarify if snapshot should have only one measurement per signal.
• Explain timestamp inaccuracy...

Note that values in a Snapshot need a record type that specifies the signal, since different signals are included in the same message.

Stream:

• Continuous delivery of data points according to a predefined agreement
• Not only a different container type.  It deals more with the delivery method (constant stream compared to atomic message)
• Does not have a fixed start/end time
• Delivers a stream of Records (or Snapshots)
• + could also deliver side-band information (e.g. Job information)
• Is analogous to the delivery of a Subscription for protocols that support subscriptions.

Examples, in JSON

(Plain) Record:

{
   "value" : " 100.54"
}

TimestampedRecord:

{
   "ts" : "2020-01-10T02:59:43.491751Z   # Zulu time, ISO std with microseconds
   "value" : "42"
}

GeospatialRecord:

{
   "pos" : "[format tbd]"
   "ts" : "2020-01-10T02:59:43.491751Z   # Zulu time, ISO std with microseconds
   "value" : "42"
}

SpecifiedRecord:

{
   "signal" : "vehicle.Chassis.Axle2.WheelCount"
   "value" : "2"
}

SpecifiedTimestampedRecord:

{
   "signal" : "vehicle.Body.ExteriorMirrors.Heating.Status"
   "ts" : "2020-01-10T02:59:43.491751"
   "value" : "false"
}

TimeSeries:

{ 
    "signal"  :  "vehicle.body.cabin.temperature"
    "count"   : "132"  # Might be redundant information, optional.
    "values" : {
        { 
            "ts" : "2020-01-10T02:59:43.491751"
            "value" : "42.5"
        },
        { 
            "ts" : "2020-01-10T02:59:43.491751"
            "value" : "43.0"
        },
        ... 130 more records
    }
}

AVRO-schema example (not complete, and of course the "#comments" are not allowed to be there)

# Simple SpecifiedTimestampedRecord (note, no array support in this one yet)
{
  "type": "record",
  "name": "SpecifiedTimeStampedRecord",
  "fields" : [
    { "name" : "signal_identifier", "type" : "string" },
    { "name" : "ts", "type" : "long" },
    { "name" : "value", "type" : [ "int", "long", "float", "double", "string", "boolean" ] }
  ]
}

More AVRO example work here: vss-tools (serializations branch)

Snapshot:

{ 
    "timeperiod" { 
        "start" : "2020-01-10T02:00:00Z",
        "end" : "2020-01-11T01:59:59Z"
    },
    "values" : {
     {
        "signal" : "vehicle.body.cabin.temperature",
        "value" : "22.0",
        "ts" : "2020-01-10T02:59:43.491751"
     },
     {
        "signal" : "vehicle.drivetrain.engine.rpm.average",
        "value" : "3200",
        "ts" : "2020-01-10T02:59:44.100403"
     }
}

todo: Examples of Derived/Statistics Records