Publications

2023

1. Enhancing AUV Autonomy With Model Predictive Path Integral Control

   Pierre Nicolay, Yvan Petillot, Mykhaylo Marfeychuk, and 2 more authors

   In OCEANS 2023, 2023

   Abs arXiv

   Autonomous underwater vehicles (AUVs) play a crucial role in surveying marine environments, carrying out underwater inspection tasks, and ocean exploration. However, in order to ensure that the AUV is able to carry out its mission successfully, a control system capable of adapting to changing environmental conditions is required. Furthermore, to ensure the robotic platform’s safe operation, the onboard controller should be able to operate under certain constraints. In this work, we investigate the feasibility of Model Predictive Path Integral Control (MPPI) for the control of an AUV. We utilise a non-linear model of the AUV to propagate the samples of the MPPI, which allow us to compute the control action in real time. We provide a detailed evaluation of the effect of the main hyperparameters on the performance of the MPPI controller. Furthermore, we compared the performance of the proposed method with a classical PID and Cascade PID approach, demonstrating the superiority of our proposed controller. Finally, we present results where environmental constraints are added and show how MPPI can handle them by simply incorporating those constraints in the cost function.

2. 

   Learning Complex Teamwork Tasks using a Sub-task Curriculum

   Elliot Fosong, Arrasy Rahman, Ignacio Carlucho, and 1 more author

   arXiv preprint arXiv:2302.04944, 2023

   Abs arXiv Code

   Training a team to complete a complex task via multi-agent reinforcement learning can be difficult due to challenges such as policy search in a large policy space, and non-stationarity caused by mutually adapting agents. To facilitate efficient learning of complex multi-agent tasks, we propose an approach which uses an expert-provided curriculum of simpler multi-agent sub-tasks. In each sub-task of the curriculum, a subset of the entire team is trained to acquire sub-task-specific policies. The sub-teams are then merged and transferred to the target task, where their policies are collectively fined tuned to solve the more complex target task. We present MEDoE, a flexible method which identifies situations in the target task where each agent can use its sub-task-specific skills, and uses this information to modulate hyperparameters for learning and exploration during the fine-tuning process. We compare MEDoE to multi-agent reinforcement learning baselines that train from scratch in the full task, and with naïve applications of standard multi-agent reinforcement learning techniques for fine-tuning. We show that MEDoE outperforms baselines which train from scratch or use naïve fine-tuning approaches, requiring significantly fewer total training timesteps to solve a range of complex teamwork tasks.s

3. Multi-Agent Reinforcement Learning for Coalitional Bargaining Games

   Lucia Cipolina-Kun, Stephen Mak, Ignacio Carlucho, and 4 more authors

   ICLR 2023 Tiny Papers, 2023

   Abs

   In recent years, there has been growing attention to the application of MARL to coalition formation problems, in particular, on coalitional bargaining games as a means of negotiation. However, the lack of theoretical principles for using MARL in coalitional bargaining games remain less explored. This paper aims to address this gap by providing an examination of the theoretical link between coalition formation, coalitional bargaining games, and MARL through the link of stochastic games. Through this analysis, the paper seeks to shed light on the underlying principles that support the use of MARL in coalitional bargaining and to explore the contributions of this approach and its limitations in comparison to traditional game theoretical methods.

4. 

   Semi-Parametric Control Architecture for Autonomous Underwater Vehicles Subject to Time Delays

   Ignacio Carlucho, Dylan Stephens, William Ard, and 1 more author

   IEEE Access, 2023

   Abs

   This paper presents a data-driven model-based control system for autonomous underwater vehicles (or AUVs) subject to input delays. This work is motivated by the input time delays that can arise in underwater robotics due to communication restrictions and sensor malfunctions. Such delays can highly degrade the performance of classical control structures resulting in unpredictable system behaviours. The proposed control architecture addresses such limitations. The approach incorporates a linear dynamic representation of the system obtained using the Koopman operator in an observer/state prediction formulation. The proposed control architecture is designed based on discrepancies between the data-driven estimation of the system’s behaviour and the actual AUV performance using chain predictors. The capabilities of the proposed approach are shown through experiments performed with a 4 degrees-of-freedom autonomous underwater vehicle. The results demonstrate stable behaviours without steady-state errors, even in the presence of long delay.

5. 

   A General Learning Framework for Open Ad Hoc Teamwork Using Graph-based Policy Learning

   Arrasy Rahman, Ignacio Carlucho, Niklas Höpner, and 1 more author

   Journal of Machine Learning Research, 2023

   Abs arXiv Code

   Open ad hoc teamwork is the problem of training a single agent to efficiently collaborate with an unknown group of teammates whose composition may change over time. A variable team composition creates challenges for the agent, such as the requirement to adapt to new team dynamics and dealing with changing state vector sizes. These challenges are aggravated in real-world applications where the controlled agent has no access to the full state of the environment. In this work, we develop a class of solutions for open ad hoc teamwork under full and partial observability. We start by developing a solution for the fully observable case that leverages graph neural network architectures to obtain an optimal policy based on reinforcement learning. We then extend this solution to partially observable scenarios by proposing different methodologies that maintain belief estimates over the latent environment states and team composition. These belief estimates are combined with our solution for the fully observable case to compute an agent’s optimal policy under partial observability in open ad hoc teamwork. Empirical results demonstrate that our approach can learn efficient policies in open ad hoc teamwork in full and partially observable cases. Further analysis demonstrates that our methods’ success is a result of effectively learning the effects of teammates’ actions while also inferring the inherent state of the environment under partial observability.

6. 

   Generating Teammates for Training Robust Ad Hoc Teamwork Agents via Best-Response Diversity

   Arrasy Rahman, Elliot Fosong, Ignacio Carlucho, and 1 more author

   Transactions on Machine Learning Research, 2023

   Abs arXiv Bib Code

   Ad hoc teamwork (AHT) is the challenge of designing a robust learner agent that effectively collaborates with unknown teammates without prior coordination mechanisms. Early approaches address the AHT challenge by training the learner with a diverse set of handcrafted teammate policies, usually designed based on an expert’s domain knowledge about the policies the learner may encounter. However, implementing teammate policies for training based on domain knowledge is not always feasible. In such cases, recent approaches attempted to improve the robustness of the learner by training it with teammate policies generated by optimising information-theoretic diversity metrics. The problem with optimising existing information-theoretic diversity metrics for teammate policy generation is the emergence of superficially different teammates. When used for AHT training, superficially different teammate behaviours may not improve a learner’s robustness during collaboration with unknown teammates. In this paper, we present an automated teammate policy generation method optimising the Best-Response Diversity (BRDiv) metric, which measures diversity based on the compatibility of teammate policies in terms of returns. We evaluate our approach in environments with multiple valid coordination strategies, comparing against methods optimising information-theoretic diversity metrics and an ablation not optimising any diversity metric. Our experiments indicate that optimising BRDiv yields a diverse set of training teammate policies that improve the learner’s performance relative to previous teammate generation approaches when collaborating with near-optimal previously unseen teammate policies.

   @article{rahman2023generating,
     title = {Generating Teammates for Training Robust Ad Hoc Teamwork Agents via Best-Response Diversity},
     author = {Rahman, Arrasy and Fosong, Elliot and Carlucho, Ignacio and Albrecht, Stefano V},
     journal = {Transactions on Machine Learning Research},
     issn = {2835-8856},
     year = {2023},
     url = {https://openreview.net/forum?id=l5BzfQhROl},
     note = {},
   }

2022

1. 

   Deep Reinforcement Learning for Multi-Agent Interaction

   Ibrahim H. Ahmed, Cillian Brewitt, Ignacio Carlucho, and 14 more authors

   AI Communications (AIC), May 2022

   arXiv Code

2. 

   Autonomous Underwater Manipulation: Current Trends in Dynamics, Control, Planning, Perception, and Future Directions

   Edward Morgan, Ignacio Carlucho, William Ard, and 1 more author

   Current Robotics Reports, Feb 2022

   Abs

   Research in underwater manipulation has mostly focused on solving individual parts of the manipulation challenge; however, we believe a systemic approach needs to be taken to achieve full autonomy. With this survey, we aim to provide a review of the different dynamic modeling, control, motion planning, and perception methodologies presented in the literature, and, more importantly, we intend to highlight the necessary steps that need to be taken to achieve fully autonomous underwater manipulation.

3. A Survey of Ad Hoc Teamwork: Definitions, Methods, and Open Problems

   Reuth Mirsky, Ignacio Carlucho, Arrasy Rahman, and 5 more authors

   European Conference on Multi-Agent Systems (EUMAS), Sep 2022

   Abs arXiv

   Ad hoc teamwork is the research problem of designing agents that can collaborate with new teammates without prior coordination. This survey makes a two-fold contribution: First, it provides a structured description of the different facets of the ad hoc teamwork problem. Second, it discusses the progress that has been made in the field so far, and identifies the immediate and long-term open problems that need to be addressed in ad hoc teamwork.

4. 

   Robotic Manipulators Performing Smart Sanding Operation: A Vibration Approach

   Joshua Nguyen, Manuel Bailey, Ignacio Carlucho, and 1 more author

   In 2022 International Conference on Robotics and Automation (ICRA), Feb 2022

   Abs

   This paper presents the design of a novel expert system for robotic manipulators performing sanding tasks on work surfaces. The expert system adjusts the velocity of the robotic manipulator based on the observed surface quality. These observation are obtained by an analysis of the raw force data provided by a force-torque sensor at the end-effector level. The expert system consists of two governing control laws that act in parallel, a variable velocity generation law and a pose regulation-based law. The variable velocity law regulates the velocity of the manipulator along a set path, in the tangent direction, based on an analysis of the frequency and amplitude of the force signal generated during the sanding process. The pose regulation-based law drives the manipulator in the bi-normal and rotational direction, ensuring the manipulators remain on the sanding path with the desired orientation. The proposed strategy is experimentally evaluated using the UR5e collaborative robotic manipulator sanding wood and metal panels. The obtained results show that such an approach is beneficial to ensure accurate contact between the sanding tool and the working environment, robust path tracking, and smart sanding.

5. 

   MIDGARD: A Simulation Platform for Autonomous Navigation in Unstructured Environments

   Giuseppe Vecchio, Simone Palazzo, Dario C Guastella, and 4 more authors

   In ICRA Workshop on Releasing Robots into the Wild: Simulations, Benchmarks, and Deployment (ICRA), Feb 2022

   Abs arXiv Website

   We present MIDGARD, an open-source simulation platform for autonomous robot navigation in outdoor unstructured environments. MIDGARD is designed to enable the training of autonomous agents (e.g., unmanned ground vehicles) in photorealistic 3D environments, and to support the generalization skills of learning-based agents through the variability in training scenarios. MIDGARD’s main features include a configurable, extensible, and difficulty-driven procedural landscape generation pipeline, with fast and photorealistic scene rendering based on Unreal Engine. Additionally, MIDGARD has built-in support for OpenAI Gym, a programming interface for feature extension (e.g., integrating new types of sensors, customizing exposing internal simulation variables), and a variety of simulated agent sensors (e.g., RGB, depth and instance/semantic segmentation). We evaluate MIDGARD’s capabilities as a benchmarking tool for robot navigation utilizing a set of state-of-the-art reinforcement learning algorithms. The results demonstrate MIDGARD’s suitability as a simulation and training environment, as well as the effectiveness of our procedural generation approach in controlling scene difficulty, which directly reflects on accuracy metrics.

6. Cooperative Marine Operations Via Ad Hoc Teams

   Ignacio Carlucho, Arrasy Rahman, William Ard, and 3 more authors

   In IJCAI Workshop on Ad Hoc Teamwork, Feb 2022

   Abs arXiv

   While research in ad hoc teamwork has great potential for solving real-world robotic applications, most developments so far have been focusing on environments with simple dynamics. In this article, we discuss how the problem of ad hoc teamwork can be of special interest for marine robotics and how it can aid marine operations. Particularly, we present a set of challenges that need to be addressed for achieving ad hoc teamwork in underwater environments and we discuss possible solutions based on current state-of-the-art developments in the ad hoc teamwork literature.

7. 

   Few-Shot Teamwork

   Elliot Fosong, Arrasy Rahman, Ignacio Carlucho, and 1 more author

   In IJCAI Workshop on Ad Hoc Teamwork, Feb 2022

   Abs

   We propose the novel few-shot teamwork (FST) problem, where skilled agents trained in a team to complete one task are combined with skilled agents from different tasks, and together must learn to adapt to an unseen but related task. We discuss how the FST problem can be seen as addressing two separate problems: one of reducing the experience required to train a team of agents to complete a complex task; and one of collaborating with unfamiliar teammates to complete a new task. Progress towards solving FST could lead to progress in both multi-agent reinforcement learning and ad hoc teamwork.

8. Towards Robust Ad Hoc Teamwork Agents By Creating Diverse Training Teammates

   Arrasy Rahman, Elliot Fosong, Ignacio Carlucho, and 1 more author

   In IJCAI Workshop on Ad Hoc Teamwork, Feb 2022

2021

1. 

   From market-ready ROVs to low-cost AUVs

   Jonatan Scharff Willners, Ignacio Carlucho, Sean Katagiri, and 8 more authors

   In OCEANS 2021: San Diego–Porto, Feb 2021

   arXiv

2. 

   An adaptive data-driven controller for underwater manipulators with variable payload

   Ignacio Carlucho, Dylan W Stephens, and Corina Barbalata

   Applied Ocean Research, Feb 2021
3. Marine Vehicles Localization Using Grid Cells for Path Integration

   Ignacio Carlucho, Manuel F Bailey, Mariano De Paula, and 1 more author

   In OCEANS 2021: San Diego–Porto, Feb 2021

   Abs arXiv

   Autonomous Underwater Vehicles (AUVs) are platforms used for research and exploration of marine environments. However, these types of vehicles face many challenges that hinder their widespread use in the industry. One of the main limitations is obtaining accurate position estimation, due to the lack of GPS signal underwater. This estimation is usually done with Kalman filters. However, new developments in the neuroscience field have shed light on the mechanisms by which mammals are able to obtain a reliable estimation of their current position based on external and internal motion cues. A new type of neuron, called Grid cells, has been shown to be part of path integration system in the brain. In this article, we show how grid cells can be used for obtaining a position estimation of underwater vehicles. The model of grid cells used requires only the linear velocities together with heading orientation and provides a reliable estimation of the vehicle’s position. We provide simulation results for an AUV which show the feasibility of our proposed methodology.

4. Data-driven controllers and the need for perception systems in underwater manipulation

   James P Oubre, Ignacio Carlucho, and Corina Barbalata

   ICRA Workshop on Marine Robotics: Active Perception (ICRA), Feb 2021

2020

1. A reinforcement learning control approach for underwater manipulation under position and torque constraints

   Ignacio Carlucho, Mariano De Paula, Corina Barbalata, and 1 more author

   In Global Oceans 2020: Singapore–US Gulf Coast, Feb 2020
2. Insights into a data driven optimal control for energy efficient manipulation

   Ignacio Carlucho, Dylan W Stephens, and Corina Barbalata

   In Global Oceans 2020: Singapore–US Gulf Coast, Feb 2020
3. DGA: A novel strategy for key gases identification in power transformers

   Matias Meira, Ignacio Carlucho, Raúl Álvarez, and 2 more authors

   In 2020 IEEE Electrical Insulation Conference (EIC), Feb 2020
4. Deep reinforcement learning approach for MPPT control of partially shaded PV systems in Smart Grids

   Luis Avila, Mariano De Paula, Maximiliano Trimboli, and 1 more author

   Applied Soft Computing, Feb 2020

5. 

   An adaptive deep reinforcement learning approach for MIMO PID control of mobile robots

   Ignacio Carlucho, Mariano De Paula, and Gerardo G Acosta

   ISA transactions, Feb 2020

2019

1. A hybrid reinforcement learning perspective for autonomous mobile robot control

   Ignacio Carlucho

   Feb 2019
2. Macábot: Prototipo de vehı́culo autónomo de superficie (asv)

   Roberto C Leegstra, Mariano De Paula, Ignacio Carlucho, and 8 more authors

   Revista Tecnologı́a y Ciencia, Feb 2019
3. MPPT for PV systems using deep reinforcement learning algorithms

   Luis Avila, Mariano De Paula, Ignacio Carlucho, and 1 more author

   IEEE Latin America Transactions, Feb 2019
4. ICTIOBOT-40 a low cost AUV platform for acoustic imaging surveying

   Gerardo G Acosta, Bruno V Menna, Ignacio Carlucho, and 8 more authors

   In OCEANS 2019-Marseille, Feb 2019

5. 

   Double Q-PID algorithm for mobile robot control

   Ignacio Carlucho, Mariano De Paula, and Gerardo G Acosta

   Expert Systems with Applications, Feb 2019

2018

1. 

   Adaptive low-level control of autonomous underwater vehicles using deep reinforcement learning

   Ignacio Carlucho, Mariano De Paula, Sen Wang, and 2 more authors

   Robotics and Autonomous Systems, Feb 2018

2. 

   AUV Position Tracking Control Using End-to-End Deep Reinforcement Learning

   Ignacio Carlucho, Mariano De Paula, Sen Wang, and 3 more authors

   In OCEANS 2018 MTS/IEEE Charleston, Feb 2018
3. An obstacle avoidance system for mobile robotics based on the virtual force field method

   Agustı́n Gianibelli, Ignacio Carlucho, Mariano De Paula, and 1 more author

   In 2018 IEEE Biennial Congress of Argentina (ARGENCON), Feb 2018
4. A Modular Battery Management System for Electric Vehicles

   Ignacio Carlucho, Roberto Vega, Marcelo Spina, and 1 more author

   In 2018 IEEE Biennial Congress of Argentina (ARGENCON), Feb 2018

2017

1. Macabot: prototipo de vehiculo autonomo de superficie (ASV)

   GG Acosta, B Menna, R La Vega, and 8 more authors

   XI Jornadas Argentinas de Robotica, Feb 2017

2. 

   Incremental Q-learning strategy for adaptive PID control of mobile robots

   Ignacio Carlucho, Mariano De Paula, Sebastian A Villar, and 1 more author

   Expert Systems with Applications, Feb 2017

2016

1. Comparison of a PID controller versus a LQG controller for an autonomous underwater vehicle

   Ignacio Carlucho, Bruno Menna, Mariano De Paula, and 1 more author

   In 2016 3rd IEEE/OES South American International Symposium on Oceanic Engineering (SAISOE), Feb 2016