Feature-based visual SLAM assumes the scene does not move. A person walking past adds features that move between frames, which corrupts both the map and the robot's estimated position. DynaSLAM handles this with Mask R-CNN, but that is too slow without a GPU. This project asks whether a real-time semantic segmentation model can do the same job on a laptop CPU.
// Overview
Built with Nischal Maharjan, Rashik Shrestha and Sajil Awale (supervisor: Jitendra Kumar Manandhar). A Raspberry Pi on a rebuilt TurtleBot streams webcam frames over Wi-Fi to a PC running ROS. On the PC, ICNet produces a person mask for each frame, and custom ROS nodes pair each frame with its mask and pass both to OpenVSLAM. OpenVSLAM drops ORB features inside the mask before tracking and mapping, and its pose replaces wheel odometry for the ROS navigation stack. The 3D map is projected to a 2D occupancy grid and scaled from measured room dimensions, since a single camera has no metric scale. move_base then plans paths, which an Arduino executes as wheel speeds.
// Gallery
// Features
A rebuilt TurtleBot that carries a Raspberry Pi and a webcam, which streams frames over Wi-Fi to a PC running ROS, plus an Arduino that turns velocity commands into wheel speeds using the differential-drive equations. The robot only senses and moves. Segmentation, SLAM and planning all run on the PC.
Custom ROS nodes use a message_filters time synchronizer to pair each camera frame with its person mask, feed both into OpenVSLAM, and publish the resulting pose as /odom and TF. On the TUM dynamic benchmark, masking cut trajectory RMS error from 23.7 cm to 1.8 cm (walking_xyz) and from 51.5 cm to 4.0 cm (walking_rpy).
Compared ICNet, BiSeNet, DeepLabV3+ and UNet for speed and accuracy on a laptop CPU (Intel i5-8250U, no GPU). ICNet was the only model fast enough for real time. Fine-tuned with a frozen backbone (400K of 11.4M parameters trainable), it reached 83.3% mIoU at 24 FPS on our validation set.
Tried Mask R-CNN / DynaSLAM for handling moving people, found it too slow on a CPU, and helped move to real-time semantic segmentation. Built the dataset for fine-tuning ICNet: filmed people walking from the robot's camera height (focused on legs) in several places and lighting conditions, and auto-labeled the frames with Detectron2 Mask R-CNN converted to binary person masks (1,435 training and 1,350 validation images). This raised accuracy on our validation set from 80.1% to 83.3% mIoU. Also studied OpenVSLAM's tracking and threading code with the team.
A rebuilt TurtleBot with a Raspberry Pi (camera and serial bridge) and an Arduino (motor control using the differential-drive equations). All heavy computation ran on the PC: segmentation, SLAM, map_server and move_base with A* planning.