Abstract
UAV networks have become a promising approach to provide wireless coverage to regions with limited connectivity. The combination of UAV networks and technologies such as the internet of things (IoT), have resulted in an enhancement in the quality of life of people living in rural areas. Therefore, it is crucial to implement fast, low-complexity, and effective strategies for UAV placement and resource allocation. In this chapter, a deep reinforcement learning (DRL) solution, based on the proximal policy optimization (PPO) algorithm, is proposed to maximize the coverage provided to users requesting microservice-based IoT applications. In order to maximize the coverage and autonomously adapt to the environment in real time, the algorithm aims to find optimal flight paths for the set of UAVs, considering the location of the users and flight restrictions. Simulation results over a realistic scenario show that the proposed solution is able to maximize the percentage of covered users.
Top1. Introduction
In recent years, Unmanned Aerial Vehicles (UAVs) have gained popularity in a wide range of applications, from package delivery to surveillance and infrastructure inspection (So, 2023). These devices have been such a revolution that they are now even being applied to activities in the primary sector, including crop monitoring (Radoglou-Grammatikis et al., 2020) and livestock breeding (Boursianis et al., 2020). All this thanks to the agility, ease and tools offered by these devices, such as measuring lasers, cameras, drums and even robotic arms.
Recently, taking advantage of these benefits, these devices are being applied to the mobile network sector, as it would not be a problem to mount WiFi/LTE antennas on UAVs to create swarms of UAVs that are responsible for providing coverage in the areas where they are deployed (Fotouhi et al., 2019), (Galán-Jiménez et al., 2021). Also, these networks have the capacity to be employed during crucial scenarios like disasters, where they can furnish communication services to terrestrial nodes. These nodes might encompass individuals who are in possession of portable devices and are in need of assistance (Erdelj et al., 2017). Similarly, they can also be used to relieve congestion in densely populated scenarios such as sporting events or concerts. These situations often involve a high number of devices attempting to access the cellular telecommunications infrastructure simultaneously (Zema et al., 2017). These swarms are innovative infrastructures made up of a multitude of interconnected UAVs, that enable the provisioning of connectivity in areas where traditional network architectures do not exist, due to their high cost and Low Return of Investment (ROI) (Cruz & Touchard, 2018). But it is not all advantages. These types of networks introduce a multitude of problems that did not appear in traditional networks and that must be addressed by the research community so that this proposal can be applied to reality.
If a little research is done on each of the techniques that could be applied to this problem, among others such as heuristics or mathematical programming, the characteristics offered by the different Machine Learning techniques stand out, which allow a quick and real-time response to different situations in changing environments, so it can be inferred that it could be a good solution to this problem.
The main goal of this work is to propose a solution that is able to maximize the coverage provided to users requesting IoT applications, as well as to react to their movement along the target area. For this purpose, a Deep Reinforcement Learning (DRL) based algorithm (machine learning sub-area), which exploits the ability of the Proximal Policy Optimization (PPO) algorithm to solve problems in dynamic environments, has been proposed. In order to maximize the coverage and autonomously adapt to the environment in real time, the algorithm aims to find near optimal flight paths for the set of UAVs, considering factors such as the location of points of interest, obstacles in the environment, as well as flight restrictions. Due to the limited capabilities of UAVs in terms of computation and battery, IoT applications requested by users are decomposed into microservices and each UAV is able to deploy and run a specific subset of them. Thus, depending on the type of microservice that users request, the algorithm aims at finding the best network configuration to maximize the coverage and satisfy users requirements. Simulation results over a realistic scenario with an area of 1 km2 and a limited coverage radius of 100 m show that the proposed solution is able to maximize the percentage of users to which it provides coverage reaching values above 80% and an average of 65%, while minimizing the movement of UAVs deployed at 50% of their power.
The rest of the article is organized as follows: first, a review of related works is provided in Section Related Work. The system model is described in Section System Model, where the architecture of the UAV-based network, the UAVs coverage and path loss models are defined. The description of the DRL-based algorithm proposed to maximize the coverage to users requesting IoT applications is provided in Section Deep Reinforcement Learning Model. In particular, the specific DRL technique used in the work, PPO, is discussed, emphasizing the functions that allow achieving good results. Section Experimental Results discusses the experimental results. Finally, Section Conclusion and Future Work reviews the conclusions that are drawn.
Key Terms in this Chapter
PPO (Proximal Policy Optimization): A reinforcement learning algorithm used to train artificial intelligence models in environments with rewards to improve decision-making and achieve better performance in specific tasks. PPO is known for its ability to enhance policies in a more stable and efficient manner compared to other policy optimization algorithms.
Path Planning: The process of determining an optimal or safe route for a vehicle or mobile agent from its current position to a specific destination, taking into account obstacles, constraints, and other relevant environmental factors.
Digital Twin: A real-time virtual representation of a physical object, system, or process. It uses data and models to simulate its behavior and state, enabling monitoring, analysis, and optimization of its performance in the real world.
MEC (Mobile Edge Computing): A distributed computing architecture that allows processing and data storage to be carried out in proximity to the user or device, instead of relying solely on centralized cloud resources. MEC aims to reduce latency, improve network efficiency, and provide faster and more responsive services for mobile applications and devices.
Coverage: This refers to the extent or range of an area that is being monitored, mapped, explored, or inspected by a system, sensor, or vehicle, such as UAVs. It is the measure of how much of a specific region is being covered or explored by a particular operation or device.
LoS (Line of Sight): An imaginary line that connects the viewpoint of an observer with the object or point of interest being observed. In the context of UAVs, maintaining Line of Sight is important to retain a direct visual connection between the pilot or control system and the UAV during its operation, ensuring effective communication and control.
DRL: (Deep Reinforcement Learning): A branch of machine learning that combines deep learning algorithms with the reinforcement learning approach, where an agent learns to make optimal decisions through interaction with an environment and receiving rewards or penalties for its actions.
UAV: (Unmanned Aerial Vehicle): Commonly known as a drone. It is a type of aircraft that can fly autonomously or be remotely controlled without a pilot onboard.