data quality

Report on the data quality of the features

feature extraction

Code: etl_feature.

Data are stored in Athena.

redash_query redash query on telemetry table

We have 4 relevant tables ['telemetry','network_log','session','incident'] which log all relevant events connected to the drive. Tables are partioned down to the single hour and vehicle.

We have 3 environments ['prod','stg','dev'] where only in prod data are complete.

telemetry

Telemetry contains the most useful information, most of the sensor data concerning vehicle dynamics and board usage. Data are ingested in an unregular way, every sensor calls the backend with different timing. Each 200ms we see a main ingestion of 2 to 5 sensors around 30ms. Information is scattered and uncomplete.

telemetry_table telemetry table, on record per topic

We call the ['mean_km_per_hour', 'lateral_force_m_per_sec_squared','longitudinal_force_m_per_sec_squared'] the sensor_features and the ['v_cpu_usage_percent', 'v_ram_usage_percent' the board features and the ['e2e_latency', 'camera_latency', 'joystick_latency'] the predictions.

data ingestion

We have different sources populating the telemetry table, some coming from the vehicle, others from the control_room. Data are collected and ingested with a different pace and sent to the backend and follow this a protobuffer schema.

telemetry_meta telemetry meta information

Each ROS topic collects the data at a different publishing rate, data are punctual and not averaged on the device. Some data don’t get sent or they don’t get collected, additional logs are stored on the vehicle but not sent to the backend.

Each topic has a publisher and a subscriber, the publisher set the timestamp and sends data over the network.

At first we see that data ingestion pretty irregular.

We than group the different sources based on their firing behaviour

ingestion_group grouping sources per frequency range

We see that even within the same source group we have diffente firing behaviour and syncing the different sources is not trivial.

We create a heatmap to visualize the firing behaviour

telemetry_matrix firing behaviour for telemetry features

Despite the visualization the features are not ingested regularly and some frequencies are pretty low

telemetry_frequency telemetry frequency

The data is than sent to the backend and arrives at stream_ingestion_timestamp_ms and get processed at processing_timestamp_ms by kinesis.

ingestion delay

We see a clear delay between the time the topic publisher sends the data and kinesis process it

delay_processing delay in processing the data, around 5 seconds as median

While the data arriving at the backend has at least 200ms delay till 1s. Streaming data has a curious cycle pattern to be further investigated.

delay_streaming delay in streaming the data

We finally check that the latency is equal for all cameras

latency_per_camera all 4 cameras have the same mean latency

data stream

Source: proc_telemetry

Data is coming irregularly and values fluctuates artificially because of time buckets. We need to re write the data flow to have consistent data for the predictions.

We take cut the timestamp and calculate the deci seconds to create more consistent time bins

spark_resample resampling time bins

We still have many empty values and we create a rolling window to replace the missing values with the running average

spark_runningAv running average over the previous and successive 3 records

data extraction

We mainly use athena to download the data but we have a strong preference for spark since it enables a more careful and complete workflow moving from simple queries to a complete software design

Athena limitations:

We than avoid all arithmetic operations in athena to avoid fake zeros.

network_log data

Query network_log.sql Code stat_network.py

We analyze the network data to understand which features should be joined with the telemetry table.

The series have many empty values

series_network time series network features

Distributions are pretty narrow and have many outliers, not all cameras have the same network figure.

boxplot_network boxplot of network features

Some features correlate and can be neglected

correlation_network correlation across network features

Some features work in particular regimes and are multimodal

joyplot_network network joyplot

Modem data are clearly multimodal

modem_joy joyplot for modem data

We join the the telemetry table with the network_log table to explore additional features but the new sources are even more noisy

netlog joined network_log and telemetry tables

modem data

Source: etl_telemetry_deci Query: network_log.sql

We analyze the time series for the modem features

modem0_pair time series of features

and correlate the features

modem0_pair correlation between modems

We show that interpolation on deci seconds creates many artefacts

modem0_pair correlation between modems

We display the feature correlation per modem

modem0_pair feature correlation per modem

We see that upload and download information is redundant

modem_updown modem upload and download

More interesting is the information in the signal features

we clearly see a correlation between the two D-netz modems (1 and 2) and we keep sinr and rssi

modem_signal correlation of the modem signal features

sinr and rssi are the most rich features

pair_rsrp pairplot of the rsrp features

pair_rsrq pairplot of the rsrp features

pair_rsrp pairplot of the rsrp features

pair_rsrp pairplot of the rsrp features

We see that most of the features have to distinguished regimes

modem_joy joyplot of modem features

We see a different persistance depending on the modem we consider

modem_persistance modem feature persistance

modem0_pair boxplot of modem features

modem0_pair pair plot of modem 0

modem1_pair pair plot of modem 1

modem2_pair pair plot of modem 2

modem3_pair pair plot of modem 3

camera index

We see that during a spike (camera_latency > 300) all cameras have similar latencies

spike_per_camera correlation of camera latency during a spike ## session time

We see that some quantities are dependent from the session_time which is the time since the starting of the session.

session_correlation correlation between the session_time and other features

ram is steadly growing over session_time but it’s value never gets critical

session_evolution evolution of ram over session_time

Session time doesn’t seem to influence the number of spikes

session_spike spikes over session_time