SDAV — SmartData for Autonomous Vehicles (a.k.a. SmartData on Wheels)

1. Overview

SmartData for Autonomous Vehicles (SDAV) is an internal LISHA project that aims at adapting EPOS and SmartData to support applications in the Autonomous Vehicles scenario, developing additional runtime components that are specific to such applications.

2. Architecture

Fig 1: SDAV Architecture Overview.

3. SmartData: Designing Data-Driven Safety-Critical Systems

Data is at the core of the design of modern Safety-Critical Systems. Data is no longer only sensed and processed in the context of the control loops of such systems. It is also secured, stored, and transmitted for the sake of the decision-making processes required for higher levels of autonomy. The task-centered strategies traditionally used to design critical systems consistently support scheduling analysis and verification of tasks execution times as long as periods, deadlines, and execution time estimates are known, but mostly ignore the flow of data across the various components in the system and often assume that data generation time is constant and can be fully encapsulated in the execution time of tasks. These assumptions, however, are not in phase with the design of modern autonomous systems such as smart factories and autonomous vehicles, which are examples of critical systems that are quickly advancing towards autonomy. A Data-driven approach to the design of such systems can more promptly accommodate requirements such as data freshness, redundant data sources, operational safety, and AI-readiness.

Decomposing the problem domain into SmartData considers the modeling of constructs that will abstract the selected entities and their relationships according to the data they produce and consume. The decomposition of the Problem Domain in SmartData follows the principles of Object Orientation. The Problem domain is decomposed into entities representing the data produced and consumed by the system. They are represented as classes that implement the SmartData interface, tagged with either << Stored >>, << Sensor >>, << Transformer >>, or << Actuator >> stereotypes, and optionally tagged with << Secure >> and << Persistent >>. The decomposition starts with identifying the actuation that will be envisioned for the system, followed by the SmartData the actuators are interested in, up to the sensors. For instance, in an autonomous vehicle, one may need to actuate, at a given rate, over throttle, brake, and steering. Each actuation is associated with a specific data input, which must be provided with a specific freshness constraint to avoid consuming expired data. This data dependency will generate Interest in other SmartData, resulting from a transformation or a sensing process. This Interest relation will then carry the timing and security requirements associated with the actuation. If more than one actuation is interested in a SmartData, this SmartData must adapt its period to supply all its consumers accordingly.

Fig 2: SDAV SmartData Design for a Simple Application.

4. SmartData Framework for Component Integration

Based on the premise that data is the core of the design of modern Safety-Critical Systems, we built upon SmartData to provide a framework able to integrate Hardware and Software-defined components, Hardware- and Software-in-the-Loop solutions, simulations, and virtually any type of process that operates over data on a SmartData-defined vehicle.

Fig 3: SDAV SmartData Integration Framework.

5. The Vehicle and Components

LISHA is currently working on the process of making a Mobilis EV into SDAV.

Fig 4: Mobillis EV model Li.

5.1. EV Components Technology

5.1.1. ECU+GPU

Model: NVIDIA® Jetson AGX Orin™ 64GB
NVIDIA® Jetson AGX Orin™ 64GB Developer Kit, provides a giant leap forward for Robotics and Edge AI. With up to 275 TOPS of AI performance and power configurable between 15W and 60W, you now have more than 8X the performance of NVIDIA® Jetson AGX Xavier™ in the same compact form-factor for developing advanced robots and other autonomous machine products.
With Jetson AGX Orin, developers can now deploy large and complex models to solve problems like natural language understanding, 3D perception, and multi-sensor fusion.

Fig 5: NVIDIA® Jetson AGX Orin™.

5.1.2. Ethernet

Model: NXP LS1021A-TSN-RD
The LS1021A Time-Sensitive Networking (TSN) Reference Design platform allows developers to design with the new IEEE Time-Sensitive Networking (TSN) standard. The board includes the QorIQ Layerscape LS1021A industrial applications processor and SJA1105T TSN switch. The LS1021ATSN is supported by an industrial Linux SDK with Xenomai real time Linux, which provides utilities for configuring TSN on the SJA1105T.

NXP LS1021a Industrial Ethernet.

5.1.3. Câmera

Model: Luxonis OAK D Long Range
Specification:

OAK-SoM-Pro based
USB3.1 and 802.3at PoE
Triple AR0234 2.3MP global shutter color sensor
Camera Spacing {right} <-5cm-> {middle} <-10cm-> {left}
3 stereo baselines: 5cm, 10cm, and 15cm for short, medium, and long-range depth
Swappable M12-mount lenses with M12 lock rings (to keep focus/alignment)
BNO086 IMU single chip 9 axis sensor with embedded sensor fusion
Aluminum Enclosure with Front Gorilla Glass
Rough dimensions: 202 x 43 x 40 mm (without rain cover)
Rain cover dimensions: 50 x 27 x 22 mm
75mm Vesa mount M4 screws
1/4" tripod mount in the center
Recovery button and RGB status LED on the bottom
Comes with the cables rain cover, protecting the device from splashing water

Fig 6: Luxonis OAK D Long Range.

5.1.4. Lidar

Model: Velodyne PUCK VLP-16

Velodyne's new PUCK™ (VLP-16) sensor is the smallest, newest, and most advanced product in Velodyne's 3D LiDAR product range. Vastly more cost-effective than similarly priced sensors, and developed with mass production in mind, it retains the key features of Velodyne's breakthroughs in LiDAR: Real-time, 360°, 3D distance and calibrated reflectivity measurements.

Fig 7: Velodyne PUCK VLP-16.

5.2. INS (GPS + IMU)

Model: LP-RESEARCH LPMS-IG1P
The LP-RESEARCH Motion Sensor LPMS-IG1P ~~RS232~~ CAN is an inertial measurement unit (IMU) / attitude and heading reference system (AHRS) with built-in GPS receiver in an IP67-rated enclosure (waterproof).
For more information on the LP-RESEARCH Motion Sensor LPMS-IG1P RS232 envisioned sensor fusion method please refer to the IMUcore description. Customized algorithms for using LPMS-IG1P as dead reckoning sensor for AGV and automotive applications using a fusion of IMU, GPS and vehicle odomotery data are offered by Zenshin Technology.

Fig 8: LP-RESEARCH LPMS-IG1P.

6. 5G Integration

6.1. Quectel 5G RM520N series

The 5G RM520N series of 5G modules from Quectel key features include a 5G/4G/3G multi-mode module, support for 3GPP Release 16 specification (Standalone — SA — and Non-SA modes), support to 5G NR Sub-6GHz, and M.2 Key B form factor. Moreover, the module comprises a GNSS receiver (Qualcomm IZat location technology Gen9C Lite). Figure 1 presents an image of the module.

Figure 1. Quectel RM520N-GL 5G module. Source: Quectel RM520N series - https://www.quectel.com/product/5g-rm520n-series

6.2. Waveshare M.2 Key-B to USB Adapter

USB to M.2 B Key by Waveshare is an M.2 B-Key USB adapter supporting RM520N. This adapter was acquired and successfully validated in the project. The adapter provides a USB 3.1 Type A interface to the 5G Module, including 4-Ch IPEX 4 to the SMA antenna interface. Finally, it also includes an aluminum alloy panel to improve heat dissipation. Waveshare suggests using a USB double-plug cable to supply sufficient power to the module and avoid energy supply issues.

Figure 3: Waveshare M.2 Key-B to USB Adapter.

7. AV Simulation Tools

Parallel to the EV work, LISHA also works with simulation tools to develop, test, and evaluate software and hardware solutions, ML models, and safety mechanisms.

7.1. Artery Simulator

Artery is a V2X simulation framework for ETSI ITS-G5 protocols like GeoNetworking and BTP. Like many VEINS-based simulators, Artery is a co-simulation of networking (handled by Artery) and a physical representation of the vehicle (handled by SUMO).

Fig 9: Artery main components.

Figure 9 shows the Artery architecture main components. The SUMO simulation is configured by a *.sumocfg file that calls on the roads (*.net.xml), traffic demand (*.rou.xml) and buildings (*.pol.xml) configuration files. Therefore, SUMO runs an independent simulation in parallel to Artery. Artery is composed by a variety of modules. Some handling the ETSI compliant networking layers and others handling data acquisition from sensors and/or vehicular mobility.

The central component of vehicles is the Middleware module. The middleware creates service modules according to an XML configuration file provided by the user. It is possible to equip vehicles with different sets of applications by configuration, i.e. communication capabilities can vary among vehicles.

7.2. CARLA Simulator

CARLA is an urban driving simulator that provides an evaluation scenario for the proposed security protocol in a dynamic urban environment with trafﬁc. CARLA is an open-source simulator implemented using Unreal Engine 4. For the experiments conducted in this paper, we will use a model that is built over a benchmark that includes autonomous driving simulations using two professionally designed towns with buildings, vegetation, and trafﬁc signs, as well as vehicular and pedestrian trafﬁc. The following subsection describes CARLA validation experiments.

Fig 10: CARLA Simulator

8. Related Projects

9. Publications

José Luis Conradi Hoffmann and Leonardo Passig Horstmann and Antônio Augusto Fröhlich, Transparent integration of autonomous vehicles simulation tools with a data-centric middleware, In: Design Automation for Embeddded Systems, -(-), 2024. DOI: https://doi.org/10.1007/s10617-023-09280-w.
José Luis Conradi Hoffmann and Antônio Augusto Fröhlich, Data-Centric Design for Formal Verification of Vehicle Monitoring, In: Proceedings of the SESC '23: Proceedings of the 2023 XIII Brazilian Symposium on Computing Systems Engineering (SBESC), pages 1-6, Porto Alegre, Brazil, November 2023. DOI: 10.1109/SBESC60926.2023.10324280.
José Luis Conradi Hoffmann, Leonardo Passig Horstmann, and Antonio Augusto Frohlich, Using Formal Methods for On-The-Fly Time Series Verification, In: Proceedings of the LADC'23: Proceedings of the 12th Latin-American Symposium on Dependable and Secure Computing, Association for Computing Machinery, pages 21-29, La Paz, Bolivia, October 2023. DOI: 10.1145/3615366.3615427.
Rafael Canal, Felipe Kaminsky Riffel, João Paulo Araujo Bonomo, Rodrigo Santos de Carvalho, and Giovani Gracioli, Misfire Detection in Combustion Engines Using Machine Learning Techniques, In: Proceedings of the SESC '23: Proceedings of the 2023 XIII Brazilian Symposium on Computing Systems Engineering (SBESC), pages 1-6, Porto Alegre, Brazil, November 2023. DOI: 10.1109/SBESC60926.2023.10324046.
José Luis Conradi Hoffmann, Leonardo Passig Horstmann, and Antônio Augusto Fröhlich, Integrating Autonomous Vehicle Simulation Tools using SmartData, In: Proceedings of the 2022 XII Brazilian Symposium on Computing Systems Engineering (SBESC), pages 1-8, Fortaleza/CE, Brazil, November 2022. DOI: 10.1109/SBESC56799.2022.9964834.
José Luis Conradi Hoffmann and Antônio Augusto Fröhlich, SmartData Safety: Online Safety Models for Data-Driven Cyber-Physical Systems, In: Proceedings of the IECON 2022 – 48th Annual Conference of the IEEE Industrial Electronics Society, pages 1-8, Brussels, Belgium, October 2022. DOI: 10.1109/IECON49645.2022.9969074.
José Luis Conradi Hoffmann and Leonardo Passig Horstmann and Matheus Wagner and Felipe Vieira and Mateus Martínez de Lucena and Antônio Augusto Fröhlich, Using Formal Methods to Specify Data-Driven Cyber-Physical Systems, In: Proceedings of the 2022 IEEE 31st International Symposium on Industrial Electronics (ISIE), pages 1-8, Anchorage, AK, USA, June 2022. DOI: 10.1109/ISIE51582.2022.9831686.