The minutes of the meetings for the Data Architecture and Infrastructure pillar of the Data Expert Group.
Historical note: the minutes for the previous project, the CVII Tech Stack, can be found here: CVII Technology Stack meeting notes
3rd May 2023
- People report successful AMM
- Combining VSS with other data models
- Logistically neither Ulf or Christian get to each others presentations on this subject
- Positive discussion to look at the proposals and find commonalities for way forward
- Logical Components / Terminology
- Discussion of the usefulness of logical component diagrams as found in the CVII Tech Stack/CCS diagrams and the Terminology dictionary.
- Avoids the competitive aspects of talking about products, falling into specific system architectures and gives a common approach for discussion.
- Useful for design patterns, deployment scenarios and architecture documents.
- How to move towards a common view?
- Review terminology list.
- Christian will see how the components fit the i7 showcase demo.
- Collectively iterate both the terminology and diagrams.
- Data Server / State Storage PoC basic building block
- Steve summarised the discussion Thursday at the AMM to create a basic building block PoC consisting of a Data Server and State Storage.
- The aim is a reusable component that could be configured to extend it for use in various deployment scenarios, e.g. connecting Vehicle API gateway.
- At the same time the basic block could also be used to investigate connections between server and storage, e.g. low latency data.
26th April 2023
19th April 2023
- AMM planning
- Steve added an AMM sub-folder to the group scratch Google Drive:
- Tues Intro
- Steve has merged the prior outline into the template and uploaded a new working draft to the above folder.
- Needs completing by Steve/Christian
- Thur Data Arch Workshop
- Arrivals
- Christian arriving Sunday, but working on car Monday. Steve Sunday. Felix Monday.
12th April 2023
- AMM planning
- Tues Intro
- Steve/Christian discussed the outline.
- Steve will redo the draft outline in the AMM template towards end of the week.
- Christian will work on his Data Centric slides in parallel.
- Aim to merge and finalise next week
- Wed Deployment Scenarios
- Thur Data Arch Workshop
- No outlines yet from Christian or Felix (see last week) for their proposals, or Felix's workshop.
- Christian: I am meeting with Felix tomorrow in Munich and should be able to finalise my session afterwords.
- Discussion of Deployment Scenarios proposal. See workshop planning page for notes.
- Piotr: Might be idea to put Deployment first in the running order as somewhat of an introduction.
- Piotr: One topic for discussion could be the management of version compatibility between components
- Piotr describes some examples. Such as Vehicle API supporting different VSS versions
- Steve: Interesting questions. Can you create something like a summary of the problem scope and what needs to be tackled? That could then be used in various ways at the AMM, e.g. Vehicle API workshop Friday and afterwords.
- Piotr: yes.
5th April 2023
- AMM planning
- Intro
- Steve: As the Intro contains slides from Christian on his views on Data Centric Arch would he like to present that part?
- Christian: OK
- Deployment Scenarios
- Steve expresses concern that we (Data EG) advance the Deployment Scenarios whilst at the AMM. 1.5h session is not enough..
- Data Arch Workshop
- Share updates and proposals:
- Christian thinks he can do a Different Data Domains (approx 20-30 mins)
- Presentation on BMW experience.
- Starting from using personal data as example of different data domain and moving to data middleware.
- Steve: m/w may be handy bridge towards follow on discussion of persistence
- Felix: Persistence
- Presentation on classifying different persistence options
- Steve: Persistence
- Steve: My usual topics:
- Data Server to Data Store connection, including high volume data
- How Data Store can support Data Centric Arch, e.g. Data analytics using TS DB features
- etc.
- AI: Add outlines for each proposed session to the planning page so ppl have an idea of what is to be discussed and what the hoped for outcome is.
- Thurs PM/Friday
- Return of experience: Felix proposed a workshop using tech from the work between BMW and Mongo.
- AI (Felix): add outline to the planning page as above.
- From discussion this could be scheduled Thurs PM
22nd March 2023
Agenda:
- Roadmap creation
- Any updates on internal discussions / Action Items
- AMM Sessions: Intro, Deployment Scenarios, Workshop
- Roadmap creation
- Updates on internal discussions / Action Items?
- Christian summarised the result of BMW discussions about the AMM. It's not finalised by some update on results from the internal PoC work should be possible. Unclear at the moment if its a longer session, video of the PoC, or some slides.
- Organisation
- AMM
- Steve summarised the three main Data Arch sessions:
- Tues PM: "Data Arch Intro (currently called "Data Centric Arch" in schedule)" (Steve/Adnan)
- Weds PM : "Deployment Scenarios" diagrams (Adnan/Erik/Steve)
- Thurs AM: 9-11am Data Arch Workshop (all)
- In addition there is a Data Expert Group general update that will mention the Data Arch pillar (Steve) and Friday an all day Data EG Workshop which of course will include Data Arch topics
- Data Arch Intro
- Steve showed a rough outline of the ppt discussed previously
- Group agrees with the approach
- Workshop
- Steve: I would like to have a topic skeleton for us to follow to keep things on track and avoid too much unproductive tak
- Felix suggests 3-4 topics if we have two hours
- Felix and Steve both interested in Data Storage/Persistance.
- Christian suggests requirements of "Different Data Domains (which means also models)"
- Discussion of possibilities
- Felix offers to do 10 mins on OSS version (minus sync) of Realm
- Steve shows the diagram hacked on last week which facilitates discussion of relationship between storage, server and feeder (Felix away last week). Interested in possibilities that functional storage brings, e.g. alternator health example. Felix makes good suggestion to have cleaned-up version which includes high level topics to tackle down (saves jumping around slides in discussion)
- Group agrees this is a way forward. Needs further work.
- AI (Steve): Provide a placeholder abstract for the Thursday workshop. Website currently shows the one from last year.
- Timings
- Steve: Thursday PM is not packed with sessions so we may be able to expand if needed.
- Felix: I do not leave until Sunday.
- Steve: I am the same.
- Steve: Friday is more a general Data Expert Group workshop but we can of course bring Data Arch topics.
15th March 2023
- Welcomed new participant from Aicas.
- AMM
- Friday workshop
- Paul: who can make it?
- Christian: BMW (Christian/Adnan/Andre) has an appointment Friday. Need to have internal discussion about what we can do that day
- Piotr: Flying later, should be available in the morning.
- Ulf: Need to leave in the afternoon for my flight
- Steve: available all day
- Steve: That's useful as it shows that the Thursday morning workshop 9-11am will be important moment when we all there. Need to plan to use that time and Friday AM wisely and have an outline of what we are discussing.
- In-vehicle Data Stores
- Steve reports that he's restarted work adding further storage options as backend to WAII VISS server. Shows rough diagram to illustrate.
- Initial discussion of some possibilities that could enable
- Christian: the diagram shows data store holding VSS data. Wouldn't it be useful for other data models also to be stored?
- Steve: Completely agree. The box is a hold over from the Covesa Tech Stack in-vehicle diagram and should be amended. This is definitely about the arch supporting the various data models required.
- Piotr brings up some possibilities using a message queue as a data feeder northbound and southbound of data server
8th March 2023
Agenda:
- Roadmap creation
- AMM organisers need to finalise schedule
Minutes:
- AMM schedule
- Weds
- Paul: There will me a mgt level VSSo session (as opposed to down in the weeds of VSSo syntax) to be delivered by VSSo team
- Explain use case and ask for more..
- Thurs technical track
- Currently Data Architecture scheduled 9:00-11:00
- Technical intro
- ... <tbd>
- Steve: We need to decide an abstract for this session ASAP
- Friday workshop
- Discussion / updates
- Christian summarized some internal discussion. Defining other data domains. Personal data one example. Lessons learned from a few years ago (explained in earlier AMM presentation) starting with using VSS which moved to need for VSSo. He will be talking to Daniel Alverez. BMW may present something at AMM but not yet clear.
- Felix: Interest in data persistence in-vehicle and data models. They are interested in performant needs for query of automotive data, e.g. GPS data as an area filter.
- Steve: we have just released v2 of our OSS Whitebox SDK for our R-Car S4 SoC. Included is a Yocto recipe for the WAII VISS Server. The OSS nature of the SDK means it will be a natural integration point for me for our work.
1st March 2023
- Ulf outlines his plan to present forest of data model idea at AMM
- Christian: Internally we have been looking at Ontologies and generation of data models. So interested to discuss Ulf's forest of data model ideas.
- Piotr: Interested in interfacing southbound native data via messaging northbound to VSS.
- Moving forward..
- Steve: Need to keep eye on mid-term and the roadmap. What we do together. If there is not an appetite to document an architecture as a first step what are we doing instead? Work on viewpoints/touchpoints to find commonalities? Perhaps document some fragments or illustrations - I feel we there are some interesting discussions but we are not capturing.
- Ulf will document his "VISS" viewpoint over next couple of weeks.
- Christian will sketch some areas he is interested in. He will also discuss internally?
- AMM
- Intro presentation
- What
- Scope diagram
- Goals/motivations and some questions
- How
- Documenting Viewpoints/touchpoints
- Call to action on input/participation
- What sessions for Thursday? Want to encourage broard input but also need to keep eye on the roadmap
22nd February 2023
Agenda:
- Workshop debrief
- Do we have agreement on some high level points?
- Acceptance of the zone arch diagram from Miro
- Creation of Data Arch / Pattern
- Outline:
Document Goals, Motivations, Questions - Stephen, Andre, Christian, Adnan
Illustrate Design Patterns - Stephen, Andre, Christian, Adnan
POC Definition [ongoing] - Stephen, Andre, Christian, Adnan
- Arch / Pattern outline Qs
Minutes:
- Debrief
- General agreement around the high level arch diagram and results from the workshop.
- Ulf: Detail - Revisit arched line from computing to HU in top level diagram
- Ulf: More than one arch may be possible, e.g. data centric, interface centric (e.g. CCS centricity around VISS)
- Roadmap
- Steve outlined some possible approaches to describing a data architecture and associated patterns electronically.
- As discussed in the workshop moving from the general (Goals, Motivations, Questions) to specifics such as Design Patterns and Touchpoint illustrations.
- Andre will have internal discussion about this
- Goal remains to build out a work plan
15th February 2023
8th February 2023
Two themes:
- The general data centric data architecture pattern
- Motivations/goals
- Key questions to be answered
- Data domains:
- Autosar, Telematic, IVI, etc
- Data characterisation:
- More descriptive/prescriptive orientations of the pattern for specific use cases or touchpoints
- e.g. Android touch-point:
- Methods of signal abstraction via VHAL or VSS Data Server
- App connection via cloud and in-vehicle
- Workshop
- Agreement building
- Opening
- Motivations/goals
- Key questions to be answered
- Data domains:
- Roadmap
1st February 2023
- Useful long discussion on finding common ground. Overran by 30 mins.
- Recognition that a workshop soon would be useful to facilitate longer discussion.
- Working towards this 'central text' Design Pattern
- Data model and technology agnostic
- Capable of supporting multiple touch points from same arch (robust, flexible etc)
- Capable of multiple data domains sharing same data m/w
- Explain requirements but avoid deep heavy specification setting
- Vehicle is not a single IT blob. Need to engage with reality of requirements from different domains in outlining use cases and touch points: Hard RT vs needs of Analytics, signal read vs storage for processing.
25th January 2023
- Steve: Let's continue discussion of BMW presentation with goal of moving towards a roadmap
- Christian is waiting for permission to make presentation public
- Christian: Requirements are definitely an interesting collaboration point for us
- Felix: Alternative approaches to meet the requirements
- Ulf: I've forwarded the presentation within Ford waiting for feedback
- Christian: Slide 7
- Important point is the architecture supporting other data models and personal data (arch not be VSS specific)
- VSSo not just for data integration but enabling knowledge transfer
- Steve: to my mind we seem to be moving towards a central design pattern from which others and implementations may be spun off
- Proposal sketch: Design Pattern: Data Centric Arch including requirements
- Patterns for Touchpoints
- Implementations/PoCs proving such
- e.g Realm showing VSS sync via middleware
- General agreement this makes sense.
- Ulf: The implementation section matches what I am working on in Ford
- Next steps:
- Start to plan the central Design Pattern
18th January 2023
Agenda:
- BMW presenting input for the roadmap
- Adnan/Christian Muehlbauer from BMW presented their input into the early 2023 roadmap
- Presentation: 2023_01_18_COVESA_Data_Middleware_V2.0.pptx
- Initial discussion during the remaining time shows overlap in interests between participants. Needs expanding upon. Participants commit to returning next week to continue discussions.
- Shows in part the usefulness of the proposed longer workshops which might take some of the questions and requirements raised and provide the time for longer discussion.
11th January 2023
(Paul's notes from the meeting)
- Adnan - will bring in arch and use cases for next meeting including use cases
- Goal is to settle on piece that is useful and focus on a piece of the architecture
- Setting up direction for data architecture
- Setup high level arch
- Requirements - like objectbox.io
- List use cases
- In workshop provide hands on experience
- What is the initial purpose? What are we trying to
get?- Exploring solutions to data architecture
- Learning how things are built and Thought
leadership! - What solutions look like when provided requirements
- Learn how to shape with requirements then look at what solutions
- Community requirements
- Let vendors show their solutions
- Christian will provide first proposal
- Ted
- Be able to receive data from OEM - $0.60 a month - Curve logging
- Want to work on TCU’s directly
- Willing to work with us
- Adnan - How to improve data handling between car and cloud.
- Stephen - use 80% vss but need to be able to handle other data in addition
4th January 2023
Agenda:
- Continue roadmap discussion including possibility of virtual workshop
- Thoughts over xmas?
- Ulf has been considering his thin API proposal over xmas and will update in the next Autosar sync call. Has proposal to extend beyond data into procedure calls for services.
- Steve repeated his intention to resume work on extending state storage to support time series DBs including Apache IoTDB
- Discussion of connection between state storage, VSS servers and the proposed vehicle API
7th December 2022
- EG project announcements
- Update on Autosar Collaboration
- Xmas break
- Steve can only remotely phone in next week, which would be the last meeting before Xmas. Nic can't make it.
- Decision taken to break for Xmas early. This is the last meeting of the year. Thanks to all who contributed.
- Request: think about the Design Pattern goals over xmas
- Data Arch Theme discussion
- Proposal to hold a virtual workshop for multiple hours early next year to accelerate the design pattern discussion.
30th November 2022
- Project status
- Majority of VSS call taken up with VSS struct support discussion. Best practices good/bad one of the focus points next.
- Ted summarised VISS/VSSo updates
- Data Arch Theme discussion
- GraphQL
- Andre: Nice for query. Problematic to have generalised set method. Have to do heavy development in schema for setter. Complicates cross vehicle model support.
- GraphQL still of interest in the tech landscape but VISS is the more obvious first priority
- Discussion of existing high level in-vehicle data architecture diagram. What it was created for and its limitations.
- Ulf showed how his V2C proposal could address data needs
- (Data) Design Patterns
- Discussion of in-vehicle architecture leads back to the existing topic of local data reduction at the edge/within function and smart exchange between functions
- Local processing generating higher logic data possibly at lower frequency, e.g. engine management does local monitoring and then externally reports decisions
- Reduces in-vehicle network traffic
- Reduces costs associated with data transmission to the cloud
- This is an obvious candidate for a design pattern: benefits, how it may be done, VSS/VISS role in that.
23rd November 2022
- Project status
- VSS struct support heads up
- VSSo meetings have restarted
- Status of the collaboration between Covesa and Autosar and work on Autosar Gateway/Vehicle API
- Data Arch Themes
- Given prior input on peoples interest Steve outlined an early roadmap for work that could create both PoCs and data patterns and which is easily adopted into related Covesa projects such as the Autosar Gateway and Android.
- VSS Data Server
- GraphQL (Covesa C++)
- VISS Data Server
- Aim: demonstrating two possibilities
- VSS Data Store
- Connection to Data Server
- Embedded app DB (e.g. Realm), 'full' DB (e.g. IoTDB)
- Aim: demonstrating two possibilities
- Target: x86 (e.g container) => embedded h/w
- Discussion:
- Piotr: suggests adding VM to x86 targets as it allows investigation of concepts like VirtIO that may be harder in containers
- Ulf: It makes sense to follow a pluggable architecture
- Piotr makes the good point that the data architecture diagrams will need to evolve to cover higher function needs such as the proposed Vehicle API as well as the current coverage for data.
- AoB
- Ulf presented his proposal for a thin API approach to the Vehicle API requirements for the Autosar collaboration.
- This is not yet public. He will work on enabling that with goal to present it to the Autosar working group once it gets going.
16th November 2022
- Brief recap of Covesa project status
- Continuing discussion of future work
- Paul: what would people like to see?
- Felix: interested in GraphQL and connection of databases as data back-end. Along with what benefits data store may enable, e.g. processing at the edge.
- Nick: ability for apps devs to discover what signals are available and on what terms, e.g. frequency.
- Answered Felix's question about what work had gone on before.
- Future is very much open to be set by the group. Cloud needs picking up again - with VSS/VISS now established the original work on CCS could move to other aspects. In-vehicle the component landscape is sketched but much work to be done selecting patterns based on data needs.
- Steve showed a draft high level data arch diagram showing the proposed Autosar gateway communicating with QM data architecture.
- Agreement that the interest around data architecture presented and discussed at the AMM is a useful starting point both for PoC investigation and likely creation of data patterns. Let's continue the discussions with a view to creating a strategy.
9th November 2022
- Introductions as we have newcomers Felix Reichenbach (MongoDB) and Francois Ozog (Shokubai.tech)
- Project updates
- Steve summarises VSS, VSC, VSSo status and EG Leadership discussions around governance and project reporting
- Roadmap
- Discussion of possible data patterns. What are people interested in?
- Felix: One area is GraphQL data server shown in the architecture. Would be interesting to have embedded implementation backending to data store.
- Steve outlines some of the work to date on that including the GraphQL libraries in the Covesa github.
- Francois outlines his interests around function abstractions.
- Steve summarises work in that area in Covesa and suggests background reading.