ROB105 · Robot Perception with Computer Vision
About Course
Give robots the ability to see: understand cameras and calibration, use classical computer vision for color, shape and marker detection, apply pretrained deep-learning detectors and close the loop by steering a simulated robot from what its camera sees.
Who it is for
Learners who can program robot motion and want robots that locate and react to objects using cameras.
Prerequisites
ROB104 (kinematics and frames) and comfortable Python, including lists and functions; NumPy basics help.
What you will be able to do
- Explain the pinhole camera model, intrinsic parameters and why calibration matters for measurement.
- Detect objects by color, contour and fiducial markers with OpenCV and estimate marker pose.
- Run a pretrained object detector and evaluate it with precision, recall and IoU.
- Implement visual servoing so a simulated robot tracks a target using camera feedback.
Lessons
- Lesson 1 · Cameras, images and calibration
- Lesson 2 · Classical vision: color, shape and markers
- Lesson 3 · Deep-learning perception and visual servoing
- Lesson 4 · Lab and assignment
Lab: See, locate and follow: a vision-guided robot
In Webots, you give a mobile robot a camera, convert frames to OpenCV images, detect an orange tote by HSV thresholding and steer toward its centroid with a proportional controller until it is close enough. In Colab, you detect ArUco markers and estimate their pose, then run a pretrained detector on sample warehouse images and compute precision and recall against labeled boxes.
Free tools: Webots (free) with a camera-equipped robot, OpenCV for Python, Google Colab (free); Ultralytics YOLO pretrained model (free, open source)
Assessed assignment
Submit your Webots controller, a short video or screenshots of the robot tracking the tote and your Colab notebook. Report the HSV ranges you chose and show the mask before and after cleaning. Present the detector's precision and recall with two examples of false positives or misses and explain likely causes. Recommend, for a small warehouse, when you would use color thresholding, fiducial markers or a deep-learning detector.
| Criterion | Weight | What excellent work looks like |
|---|---|---|
| Visual servoing | 30% | The robot reliably finds, approaches and stops at the target, with a search behavior when it is not visible. |
| Classical vision | 25% | Color segmentation, contour detection and ArUco pose estimation are correct and clearly shown. |
| Detector evaluation | 25% | Precision, recall and IoU are computed correctly and errors are analyzed thoughtfully. |
| Method recommendation | 20% | The choice between methods is justified by robustness, cost, lighting and setup effort. |
Portfolio piece
A vision-guided robot controller and an evaluated perception notebook that demonstrate practical robot-vision skills.
Next step
Continue with ROB106 · Navigation, Mapping and ROS Basics · 8 hours · $56.00
Course Content
ROB105 · Robot Perception with Computer Vision (8 h)
-
Lesson 1 · Cameras, images and calibration
-
Lesson 2 · Classical vision: color, shape and markers
-
Lesson 3 · Deep-learning perception and visual servoing
-
Lesson 4 · Lab and assignment
-
ROB105 Assessment · Robot Perception with Computer Vision