Spiega documentation

Article data_quality

What spiega is about

data quality

Report on the data quality of the features

feature extraction

Code: etl_feature .

Data are stored in Athena.

redash_query

redash query on telemetry table

We have 4 relevant tables ['telemetry','network_log','session','incident'] which log all relevant events connected to the drive. Tables are partioned down to the single hour and vehicle.

We have 3 environments ['prod','stg','dev'] where only in prod data are complete.

telemetry

Telemetry contains the most useful information, most of the sensor data concerning vehicle dynamics and board usage. Data are ingested in an unregular way, every sensor calls the backend with different timing. Each 200ms we see a main ingestion of 2 to 5 sensors around 30ms. Information is scattered and uncomplete.

telemetry_table

telemetry table, on record per topic

We call the ['mean_km_per_hour', 'lateral_force_m_per_sec_squared','longitudinal_force_m_per_sec_squared'] the sensor_features and the ['v_cpu_usage_percent', 'v_ram_usage_percent' the board features and the ['e2e_latency', 'camera_latency', 'joystick_latency'] the predictions .

data ingestion

We have different sources populating the telemetry table, some coming from the vehicle , others from the control_room . Data are collected and ingested with a different pace and sent to the backend and follow this a protobuffer schema.

telemetry_meta

telemetry meta information

Each ROS topic collects the data at a different publishing rate, data are punctual and not averaged on the device. Some data don't get sent or they don't get collected, additional logs are stored on the vehicle but not sent to the backend.

Each topic has a publisher and a subscriber, the publisher set the timestamp and sends data over the network.

At first we see that data ingestion pretty irregular.

We than group the different sources based on their firing behaviour

ingestion_group

grouping sources per frequency range

We see that even within the same source group we have diffente firing behaviour and syncing the different sources is not trivial.

We create a heatmap to visualize the firing behaviour

telemetry_matrix

firing behaviour for telemetry features

Despite the visualization the features are not ingested regularly and some frequencies are pretty low

telemetry_frequency

telemetry frequency

The data is than sent to the backend and arrives at stream_ingestion_timestamp_ms and get processed at processing_timestamp_ms by kinesis.

ingestion delay

We see a clear delay between the time the topic publisher sends the data and kinesis process it

delay_processing

delay in processing the data, around 5 seconds as median

While the data arriving at the backend has at least 200ms delay till 1s. Streaming data has a curious cycle pattern to be further investigated.

delay_streaming

delay in streaming the data

We finally check that the latency is equal for all cameras

latency_per_camera

all 4 cameras have the same mean latency

data stream

Source: proc_telemetry

Data is coming irregularly and values fluctuates artificially because of time buckets. We need to re write the data flow to have consistent data for the predictions.

We take cut the timestamp and calculate the deci seconds to create more consistent time bins

spark_resample

resampling time bins

We still have many empty values and we create a rolling window to replace the missing values with the running average

spark_runningAv

running average over the previous and successive 3 records

data extraction

We mainly use athena to download the data but we have a strong preference for spark since it enables a more careful and complete workflow moving from simple queries to a complete software design

Athena limitations:

We than avoid all arithmetic operations in athena to avoid fake zeros.

network_log data

Query network_log.sql
Code stat_network.py

We analyze the network data to understand which features should be joined with the telemetry table.

The series have many empty values

series_network

time series network features

Distributions are pretty narrow and have many outliers, not all cameras have the same network figure.

boxplot_network

boxplot of network features

Some features correlate and can be neglected

correlation_network

correlation across network features

Some features work in particular regimes and are multimodal

joyplot_network

network joyplot

Modem data are clearly multimodal

modem_joy

joyplot for modem data

We join the the telemetry table with the network_log table to explore additional features but the new sources are even more noisy

netlog

joined network_log and telemetry tables

modem data

Source: etl_telemetry_deci
Query: network_log.sql

We analyze the time series for the modem features

modem0_pair

time series of features

and correlate the features

modem0_pair

correlation between modems

We show that interpolation on deci seconds creates many artefacts

modem0_pair

correlation between modems

We display the feature correlation per modem

modem0_pair

feature correlation per modem

We see that upload and download information is redundant

modem_updown

modem upload and download

More interesting is the information in the signal features

we clearly see a correlation between the two D-netz modems (1 and 2) and we keep sinr and rssi

modem_signal

correlation of the modem signal features

sinr and rssi are the most rich features

pair_rsrp

pairplot of the rsrp features

pair_rsrq

pairplot of the rsrp features

pair_rsrp

pairplot of the rsrp features

pair_rsrp

pairplot of the rsrp features

We see that most of the features have to distinguished regimes

modem_joy

joyplot of modem features

We see a different persistance depending on the modem we consider

modem_persistance

modem feature persistance

modem0_pair

boxplot of modem features

modem0_pair

pair plot of modem 0

modem1_pair

pair plot of modem 1

modem2_pair

pair plot of modem 2

modem3_pair

pair plot of modem 3

camera index

We see that during a spike ( camera_latency > 300) all cameras have similar latencies

spike_per_camera

correlation of camera latency during a spike

session time

We see that some quantities are dependent from the session_time which is the time since the starting of the session.

session_correlation

correlation between the session_time and other features

ram is steadly growing over session_time but it's value never gets critical

session_evolution

evolution of ram over session_time

Session time doesn't seem to influence the number of spikes

session_spike

spikes over session_time