This thesis proposal aims to leverage the exciting direction of integrating Large Language Models with robotic agents for the purpose of understanding and exploiting natural language interactions between humans and robots. During the thesis, the student(s) would integrate available LLM models with ROS interface onboard robots such as NAO. The outcome of the thesis would, for example, be able to enable the robot to execute certain desired tasks such as “Go to the red box” or “Follow the person with the yellow shirt”. Thus, enabling the transformation of commands in natural language to low-level/ high-level control actions through the processing of real-time sensor information such as RGB images. For inspiration on use cases, visit the link here.

The student(s) would be expected to complete during this project are:

• Develop and implement a ROS-interface with LLM models and construct a framework to process RGB image frames to address the query.

• Transform the resulting output from the model to robot actionable states.

• Weekly counselling and guidance meetings with supervisors.

We would like the student(s) to have a working understanding of Python/C++ language and the use of ROS.

Contact:

Vignesh Kottayam Viswanathan (A2563) vigkot@ltu.se  

George Nikolakopoulos (A2556) geonik@ltu.se