Versions Compared

Old Version 5

changes.mady.by.user Swen Schisler

Saved on

compared with

New Version 6

changes.mady.by.user Swen Schisler

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

The following table gives an overview which services are required.

List of Essential Services for the Distributed Data Market Ecosystem

Service NameDescription
User Management (with Authentication Federation & Identity Mapping)Manages user authentication, registration, and profile management across distributed instances. Uses Azure AD B2C to provide federated login (Microsoft, Google, corporate SSO). Each user’s identity is mapped to a unique internal ID, ensuring they can be recognized across different ecosystem services without requiring a new login.
Role & Rights ManagementDefines and enforces access control, user roles, and permissions across instances for data usage and monetization.
Consent & Privacy ManagementAllows users to manage data-sharing permissions, revoke consent, and ensure compliance with GDPR, EU Data Act, and similar regulations.
Data MarketplaceEnables data providers (OEMs, fleet managers) to list, sell, and manage datasets while allowing buyers to search, preview, and purchase data.
API Gateway & ManagementProvides a secure API layer for communication between distributed instances. Ensures authentication and standardization using Azure API Management.
Data Storage & ProcessingA scalable, globally distributed database for storing metadata, logs, and access control settings. Uses Azure Cosmos DB and Azure Functions for event-driven processing.
Search & Discovery (Global Registry)Provides a federated registry for discovering datasets, services, and API endpoints across distributed instances. Supports metadata tagging for easy filtering.
Monetization & BillingHandles subscription models, pay-per-use pricing, and revenue-sharing among participants. Uses Azure Payment Connector for transactions.
Security & AuthenticationEnsures secure access using OAuth 2.0, OpenID Connect, and multi-factor authentication (MFA). Protects against unauthorized access and fraud.
Data Anonymization & ComplianceAutomatically anonymizes sensitive user data before sharing, ensuring compliance with data privacy laws. Uses Azure Purview for governance.
Service OrchestrationManages workflows for data ingestion, user onboarding, and API integrations using Azure Logic Apps or Azure Event Grid.
Logging & MonitoringProvides real-time monitoring and logging of ecosystem activities, including API requests, user transactions, and data access logs. Uses Azure Monitor & Log Analytics.
Incident & Violation HandlingDetects anomalies, data breaches, or regulatory violations and alerts operators. Provides remediation workflows.
Regional Compliance ManagementEnsures that different jurisdictions comply with data residency laws (e.g., GDPR, CCPA) by routing requests to appropriate data centers.
Data Aggregation & InsightsAggregates data from multiple sources and provides insights using Azure Synapse Analytics and Power BI for visualization.
Edge ProcessingHandles real-time data processing closer to the source for low-latency applications (e.g., connected vehicles, fleet monitoring). Uses Azure IoT Edge.
Developer PortalA hub for developers to access API documentation, sample datasets, and testing environments for building applications.
Customer Support & Help DeskProvides technical assistance, user guides, and ticketing support for all ecosystem participants.
Distributed Node ManagementManages the lifecycle of distributed instances, allowing operators (OEMs, data collectors) to onboard, configure, and maintain their nodes.
Capability Registration & Node DiscoveryA global registry where distributed instances register their capabilities (e.g., data types available, APIs, supported regions).
Instance Health & Load BalancingEnsures each instance is healthy, load-balanced, and available, using Azure Front Door to route traffic efficiently across distributed nodes.
Service namedescriptionimplementation statususer service

this is a crud-service whicht maintains the

user base for the whole eco-system. It allows

user to onboard, change data and be deleted.

It requires a database to store the data permanently

right management

this right management deals only with the right of the 

users of the system. Not the cloud management.

identity federation

this is used to enable login with a

foreign identity (like gmail, facebook). The user identity

is matched to an internal id.

data store service

is a crud service that allows the management

of a data source.  

data source

management

This is a  crud service to onboard a  data source. 

This can be a car or another IoT device. These sources

needs to be linked with an individual (the owner)

and the corresponding data store

usually the manufacturer owns. 

service management

crud service for managing services and corresponding

consent declarations.

consent management

crud service for managing consent declarations of the user

registry

is used to register components of the system

(nodes and services) and is used for

runtime resolution of requests

audit trail

legal compliance needs proof. This service writes

an audit trail of events that can be used in trials

as a proof

usage service

this service simply counts the usage of resources

as a metrik. This can be used for billing later.

billing

based on usage and negotiated rates, a monthly

bill can be generated.

monitor

this service monitors the system and based

on thresholds will trigger alarms or countermeasures

logging

this services receives log messages from all

services (per node) and will store them within the node

As for all µ-service based architectures it is important to cut the services in a way that on one hand the

size of the service stays maintainable but on the other hand that they are not too simple (too many services).

Workflows

A concept called orchestration makes out of the services a workflow. For example, if a user from the type of a 

data collector is onboarding the orchestrator will ask to onboard a data source as well. These workflows

will be described in the following sectionshere.