Robot plays “Rock, Paper, Scissors” - Part 3/3
The robot in the final game
What began as a spontaneous idea developed into an ambitious project for Sebastian Trella - robotics influencer and blogger: A robot plays “rock, paper, scissors” against a human - live, with an IDS NXT camera and AI-supported gesture recognition.
Part 1 focused on the basic development: the implementation of gesture recognition using intelligent image processing and thus also the training of the neural networks. Image analysis and result communication were handled directly on or via the IDS NXT camera – without the need for an additional PC. Part 2 involved further processing of the recognized gestures using a specially created vision app. Now in Part 3: Ready to play! The system has been completed, tested and the game is running.
Final assembly: When components become a system
With interactive systems like this, the biggest challenge is rarely in the individual components - but almost always in their interaction. In this case too, the camera, logic and robot arm each functioned reliably on their own: The IDS NXT camera precisely recognized the hand signals, the decision logic reacted according to the rules, and the robot implemented the corresponding movement. But bringing all these components together was a challenge - especially when it came to precise timing, signal transmission and synchronization. “What sounds good in theory often looks different in practice,” Sebastian Trella quickly realized. “Thanks to the open architecture and good integration of the IDS NXT platform, however, this challenge was overcome. Targeted testing and iterative refinement turned the prototype into a functioning game.”
Fine adjustment: Gesture recognition becomes robust
How reliably gesture recognition works depends largely on the training data and the environmental conditions - an experience that Sebastian Trella already made in the initial phase: “Initially, I only trained the models with my own hands. However, I had not considered the case of ‘no hand in the picture’. This naturally led to incorrect evaluations.” But with the help of the IDS lighthouse training platform the model could be easily expanded. New images have been added, including other people's hands in front of different backgrounds and under changing lighting conditions. Details such as different skin tones and the wearing of rings were also incorporated into the training. This targeted diversification of the training data significantly improved the recognition performance - the AI now responded stably and reliably, regardless of who was playing or in which environment. At the same time, Trella's understanding of the general handling of neural networks and their practical requirements during training grew with each step.
Implementation: How does the robot play?
The robot's decisions are random - it doesn't bluff and it doesn't learn from previous games. But that is precisely what makes the game so charming: Man against machine, at eye level. The game round takes place in five phases:
- Capture of the human hand by the camera
- AI-based image evaluation of the gesture (scissors, stone, paper)
- Robot communication and movement
- Determining the result (robot wins, human wins, draw)
- Robot communication and movement
The entire game is controlled directly via a vision app on the intelligent IDS NXT camera - without the need for an additional PC. It recognizes the gesture shown by the player, evaluates it using artificial intelligence and then sends a digital IO signal to the robot to trigger its reaction. To keep the game fair, the robot's gesture is not influenced by the player's action, but is determined neutrally and randomly. While the camera analyzes the player's gesture, the robot waits for its start signal. It’s only then that he reveals his gesture too. The camera then evaluates the match result and sends the final decision, which the robot then displays.
The coordination of waiting times and signal transmissions was a key challenge. Although the vision app can analyze the player's gesture within fractions of a second, the robot cannot react at the same speed. Simultaneous display and evaluation of the gestures could therefore not be realized in this way. Nevertheless, the response time was significantly reduced through targeted optimization of processes. “The game now feels more dynamic and fluid. The AI recognizes the player's hand in the camera image and evaluates the gesture directly. This works so reliably that there is no need to redisplay it on a monitor. The robot now completely takes over the display of the game information - this speeds up the entire game flow considerably,” explains Trella.
Outlook: What remains open and what comes next?
The high reliability of gesture recognition opens up exciting prospects. “A possible further development would be touchless machine control, for example through simple hand gestures in industrial environments,” Sebastian Trella reflects and adds: "Of course, questions remain unanswered even after the project has been completed. For example: How can communication between the robot and camera be made even more ‘elegant’ - perhaps via a kind of dialog using interfaces like RS-232, REST or OPC-UA? Wouldn't a movable robot hand be the logical next step for an even more realistic gaming experience?"
Even though the “Rock, Paper, Scissors” project is now coming to an end, Sebastian Trella is already planning new ideas for human-machine interaction with AI support. The reason: If a robot can already play with a human today (using AI) - what else can it do?