Future Impact Decomposition in Request-level Recommendations

In recommender systems, reinforcement learning solutions have shown promising results in optimizing the interaction sequence between users and the system over the long-term performance. For practical reasons, the policy's actions are typically designed as recommending a list of items to handle users' frequent and continuous browsing requests more efficiently. In this list-wise recommendation scenario, the user state is updated upon every request in the corresponding MDP formulation. However, this request-level formulation is essentially inconsistent with the user's item-level behavior. In this study, we demonstrate that an item-level optimization approach can better utilize item characteristics and optimize the policy's performance even under the request-level MDP. We support this claim by comparing the performance of standard request-level methods with the proposed item-level actor-critic framework in both simulation and online experiments. Furthermore, we show that a reward-based future decomposition strategy can better express the item-wise future impact and improve the recommendation accuracy in the long term. To achieve a more thorough understanding of the decomposition strategy, we propose a model-based re-weighting framework with adversarial learning that further boost the performance and investigate its correlation with the reward-based strategy.

MACC: Cross-Layer Multi-Agent Congestion Control with Deep Reinforcement Learning

Congestion Control (CC), as the core networking task to efficiently utilize network capacity, received great attention and widely used in various Internet communication applications such as 5G, Internet-of-Things, UAN, and more. Various CC algorithms have been proposed both on network and transport layers such as Active Queue Management (AQM) algorithm and Transmission Control Protocol (TCP) congestion control mechanism. But it is hard to model dynamic AQM/TCP system and cooperate two algorithms to obtain excellent performance under different communication scenarios. In this paper, we explore the performance of multi-agent reinforcement learning-based cross-layer congestion control algorithms and present cooperation performance of two agents, known as MACC (Multi-agent Congestion Control). We implement MACC in NS3. The simulation results show that our scheme outperforms other congestion control combination in terms of throughput and delay, etc. Not only does it proves that networking protocols based on multi-agent deep reinforcement learning is efficient for communication managing, but also verifies that networking area can be used as new playground for machine learning algorithms.

Decentralized Circle Formation Control for Fish-like Robots in the Real-world via Reinforcement Learning

In this paper, the circle formation control problem is addressed for a group of cooperative underactuated fish-like robots involving unknown nonlinear dynamics and disturbances. Based on the reinforcement learning and cognitive consistency theory, we propose a decentralized controller without the knowledge of the dynamics of the fish-like robots. The proposed controller can be transferred from simulation to reality. It is only trained in our established simulation environment, and the trained controller can be deployed to real robots without any manual tuning. Simulation results confirm that the proposed model-free robust formation control method is scalable with respect to the group size of the robots and outperforms other representative RL algorithms. Several experiments in the real world verify the effectiveness of our RL-based approach for circle formation control.

An Electrocommunication System Using FSK Modulation and Deep Learning Based Demodulation for Underwater Robots

Underwater communication is extremely challenging for small underwater robots which typically have stringent power and size constraints. In our previous work, we developed an artificial electrocommunication system which could be an alternative for the communication of small underwater robots. This paper further presents a new electrocommunication system that utilizes Binary Frequency Shift Keying (2FSK) modulation and deep-learning-based demodulation for underwater robots. We first derive an underwater electrocommunication model that covers both the near-field area and a large transition area outside of the near-field area. 2FSK modulation is adopted to improve the anti-interference ability of the electric signal. A deep learning algorithm is used to demodulate the electric signal by the receiver. Simulations and experiments show that with the same testing condition, the new communication system outperforms the previous system in both the communication distance and the data transmitting rate. In specific, the newly developed communication system achieves stable communication within the distance of 10 m at a data transfer rate of 5 Kbps with a power consumption of less than 0.1 W. The substantial increase in communication distance further improves the possibility of electrocommunication in underwater robotics.

Motion Planning for Heterogeneous Unmanned Systems under Partial Observation from UAV

For heterogeneous unmanned systems composed of unmanned aerial vehicles (UAVs) and unmanned ground vehicles (UGVs), using UAVs serve as eyes to assist UGVs in motion planning is a promising research direction due to the UAVs' vast view scope. However, due to UAVs flight altitude limitations, it may be impossible to observe the global map, and motion planning in the local map is a POMDP (Partially Observable Markov Decision Process) problem. This paper proposes a motion planning algorithm for heterogeneous unmanned system under partial observation from UAV without reconstruction of global maps, which consists of two parts designed for perception and decision-making, respectively. For the perception part, we propose the Grid Map Generation Network (GMGN), which is used to perceive scenes from UAV's perspective and classify the pathways and obstacles. For the decision-making part, we propose the Motion Command Generation Network (MCGN). Due to the addition of memory mechanism, MCGN has planning and reasoning abilities under partial observation from UAVs. We evaluate our proposed algorithm by comparing with baseline algorithms. The results show that our method effectively plans the motion of heterogeneous unmanned systems and achieves a relatively high success rate.

A Thermoplastic Elastomer Belt Based Robotic Gripper

Novel robotic grippers have captured increasing interests recently because of their abilities to adapt to varieties of circumstances and their powerful functionalities. Differing from traditional gripper with mechanical components-made fingers, novel robotic grippers are typically made of novel structures and materials, using a novel manufacturing process. In this paper, a novel robotic gripper with external frame and internal thermoplastic elastomer belt-made net is proposed. The gripper grasps objects using the friction between the net and objects. It has the ability of adaptive gripping through flexible contact surface. Stress simulation has been used to explore the regularity between the normal stress on the net and the deformation of the net. Experiments are conducted on a variety of objects to measure the force needed to reliably grip and hold the object. Test results show that the gripper can successfully grip objects with varying shape, dimensions, and textures. It is promising that the gripper can be used for grasping fragile objects in the industry or out in the field, and also grasping the marine organisms without hurting them.

Artificial Lateral Line Based Relative State Estimation for Two Adjacent Robotic Fish

The lateral line enables fish to efficiently sense the surrounding environment, thus assisting flow-related fish behaviours. Inspired by this phenomenon, varieties of artificial lateral line systems (ALLSs) have been developed and applied to underwater robots. This article focuses on using the pressure sensor arrays based on ALLS-measured hydrodynamic pressure variations (HPVs) for estimating the relative state between two adjacent robotic fish with leader-follower formation. The relative states include the relative oscillating frequency, amplitude, and offset of the upstream robotic fish to the downstream robotic fish, the relative vertical distance, the relative yaw angle, the relative pitch angle, and the relative roll angle between the two adjacent robotic fish. Regression model between the ALLS-measured and the mentioned relative states is investigated, and regression model-based relative state estimation is conducted. Specifically, two criteria are proposed firstly to investigate not only the sensitivity of each pressure sensor to the variations of relative state but also the insufficiency and redundancy of the pressure sensors. And thus the pressure sensors used for regression analysis are determined. Then four typical regression methods, including random forest algorithm, support vector regression, back propagation neural network, and multiple linear regression method are used for establishing regression models between the ALLS-measured HPVs and the relative states. Then regression effects of the four methods are compared and discussed. Finally, random forest-based method, which has the best regression effect, is used to estimate relative yaw angle and oscillating amplitude using the ALLS-measured HPVs and exhibits excellent estimation performance. This work contributes to local relative estimation for a group of underwater robots, which has always been a challenge.

Three-Dimensional Dynamic Modeling and Motion Analysis for an Active-Tail-Actuated Robotic Fish with Barycentre Regulating Mechanism

Dynamic modeling has been capturing attention for its fundamentality in precise locomotion analyses and control of underwater robots. However, the existing researches have mainly focused on investigating two-dimensional motion of underwater robots, and little attention has been paid to three-dimensional dynamic modeling, which is just what we focus on. In this article, a three-dimensional dynamic model of an active-tail-actuated robotic fish with a barycentre regulating mechanism is built by combining Newton's second law for linear motion and Euler's equation for angular motion. The model parameters are determined by three-dimensional computer-aided design (CAD) software SolidWorks, HyperFlow-based computational fluid dynamics (CFD) simulation, and grey-box model estimation method. Both kinematic experiments with a prototype and numerical simulations are applied to validate the accuracy of the dynamic model mutually. Based on the dynamic model, multiple three-dimensional motions, including rectilinear motion, turning motion, gliding motion, and spiral motion, are analyzed. The experimental and simulation results demonstrate the effectiveness of the proposed model in evaluating the trajectory, attitude, and motion parameters, including the velocity, turning radius, angular velocity, etc., of the robotic fish.

Fish lateral line inspired perception and flow-aided control: A review

Any phenomenon in nature is potential to be an inspiration for us to propose new ideas. Lateral line is a typical example which has attracted more interest in recent years. With the aid of lateral line, fish is capable of acquiring fluid information around, which is of great significance for them to survive, communicate and hunt underwater. In this paper, we briefly introduce the morphology and mechanism of the lateral line first. Then we focus on the development of artificial lateral line which typically consists of an array of sensors and can be installed on underwater robots. A series of sensors inspired by the lateral line with different sensing principles have been summarized. And then the applications of artificial lateral line system in hydrodynamic environment sensing and vortices detection, dipole oscillation source detection, and autonomous control of underwater robots have been surveyed. In addition, the existing problems and future foci in the field have been further discussed in detail. The current works and future foci have demonstrated that artificial lateral line has great potentials of research and contributes to the applications of underwater robots.