TCV152 – Mini-project in Computational Vision
Instructor : Prof. Ohad Ben-Shahar TA : Ehud Barnea
Computer Science Department Ben-Gurion University of the Negev Goal
In general
• Exposure to computational vision and robotic • Hands on and practical experience in these two topics
And specifically
• Get to know ROS – the Robot Operating System • Get to know Komodo – a real mobile robotic platform equipped with manipulators and sensors • Get to know OpenCV – and free computer vision library • Apply your knowledge to program a real robot to do some computer vision task
2 TCV152 introduction What to expect
• This is a mini-project, i.e., you will mostly be on your own • You will learn much technical material, all from online sources and by self experimentation. • You will work in groups. Group size depends on number of students enrolled in the course • Cooperation between groups is encouraged
• You will work first without and then with a real robot (more about this in a moment) • ROS programming is done in Linux and C++ (or Python) • Programming the robot is difficult, frustrating, but eventually rewarding
• We have a TA (Ehud Barnea) in a very part time position, i.e., he will help in emergencies only.
3 TCV152 introduction The robot and the lab
• Eventually you will work with a real robot • BUT…do not work with the real robot unless you have to. During large parts of the project you will be able to do without it. • When you do end up needing it, the robot and its ROS computer are located in the AI and robotics lab, room 37/211. • When you end up working in the robot, make sure that most of the time it is placed on a platform that keeps the wheels in the air. This is an important safety requirement. The projects in this mini-project course will not need any movement of the mobile platform. • The lab is the home of graduate students from the AI group. Respect them and their work space. Do not touch anything except the robot and the ROS computer. If you come to work solo, you are welcome any time during the day. If you come as a team (2 students or more), you will be able to access the lab from 17:00 only. • Since the robot is a critical resource and only one team can work at a time, we will use a scheduling system where you reserve your preferred time slot ahead of time. More information on this will follow. • Before you can use the robot, you will need to sign a form in which you verify that you understand the risks and that you will follow safety procedures
4 TCV152 introduction
Course info
• Instructor: Prof. Ohad Ben-Shahar • Office: 37/114 • Phone: (08-64)77868 • Email: [email protected] • Office hours: Tue 10:00-12:00; best to schedule via email. • TA: Mr. Ehud Barnea
5 TCV152 introduction Detailed course plan (no need to copy, you will have on the web + email)
Week 1: No class
Week 2: Class meeting • Introduction to course • ROS overview lecture
Weeks 2-4: Independent work • Read the ROS Getting Started section, including the Introduction, Concepts, and Higher-Level Concepts • Read the ROS Installation guide. Installing ROS on your laptop is optional but desired. • Read and practice the Beginner level ROS Tutorials (tutorials #1.1.1-1.1.20). Some tutorials are similar up to the language used (C++ or Python) and you are free to use your language. • Submit lab assignment #1
6 TCV152 introduction Detailed course plan
Weeks 5-6: Independent work • Read the Komodo user manual • Read the ric_robot package documentation with focus on using the sensors (cameras' depth sensors, laser scanner, etc..) • Read and practice the ric tutorials, and in particular tutorials 1,3,6,8,9 • Submit lab assignment #2
Week 7: Class meeting • Computer vision overview lecture • Introduction to OpenCV
Weeks 7-8: Independent work • Read the fundamental aspects of the OpenCV documentation Try to cover the basic parts of the user guide, and review the essentials from the introductory, core functionality, image processing, 2D features tutorials. • Submit lab assignment #3
7 TCV152 introduction
Detailed course plan
Week 9: Independent work • Review the project topics. • Select a project • Submit a 1-page proposal.
Weeks 10-14 and until project submission date (TBD): • Study any additional OpenCV and ROS resources necessary for your project • Implement your project • Submit report + code + documentation • Demonstrate your project
8 TCV152 introduction Lab assignment #1 (Due April 3)
A 3 node ROS system that does the following • "First" node - Generates messages at random times but no more than 1 second apart. Each message contains a random number in the range [0,100]. Messages are published on the topic First Second "FirstNumber" and logged in a "FirstLog.txt" file with their generation time. • "Second" node - Generates messages at random times but no more than 0.5 second apart. Each message contains a random number in the range [-100,0]. Messages are published on the topic "SecondNumber" and logged in a "SecondLog.txt" file with their generation time.
• "Adder" node - Using callback functions and threads, this node Adder should subscribe to the two topics above, and add and display the line "
9 TCV152 introduction Lab assignment #2 (Due April 26)
Implement and demonstrate a 3 node ROS system that does the following • A "sensor" node that asynchronously publishes the readings of the laser range sensor (this node is already supplied by RoboTiCan as part of ric_robot) • A "brain" node that subscribes to the sensor topic, finds the closest obstacle from the robot, and publishes its distance to the "ObstacleDistance" topic. • A "speed" node that subscribes to the ObstacleDistance topic, computes a speed that is inversely proportional to the present distance, and continuously publishes this speed to the proper topic that actually controls the speed of the wheels. Make sure to rectify the computed and published speed to some reasonable maximal value. If your assignment is successful, you will be able to continuously control the speed of the wheels by simply moving your hand closer and farther away from the robot.
IMPORTANT: never remove the robot from the safety platform so wheels run in the air without any robot motion.
10 TCV152 introduction Lab assignment #3 (Due May 10)
Implement and demonstrate the following system that uses the color and depth images that the Asus RGB-D camera publishes on several topics • A "Face detection" service - Create a ROS service that gets a color image as input, detects faces in the image, and returns the coordinates of boundary box of the detected faces. • A “Control” node that listens to the color and depth images from the Asus camera, detects faces in the color image (using the face detection service above), and moves the wheels forward when a face is detected. This node should also save the original image with colored rectangles over the detected faces, where the brightness of each rectangle depends on the distance between the robot and the (central pixel of the) face. If your assignment is successful, you will be able to control the wheels by simply hiding/exposing your face to the robot.
IMPORTANT: never remove the robot from the safety platform so wheels run in the air without any robot motion.
11 TCV152 introduction Resource (no need to copy, you will have on the web + email)
ROS wiki: Portal http://wiki.ros.org/ Tutorials http://wiki.ros.org/ROS/Tutorials
Komodo roc_robot http://wiki.ros.org/ric_robot roc tutorials http://wiki.ros.org/ric/Tutorials
OpenCV Documentation portal http://opencv.org/documentation.html Tutorials http://docs.opencv.org/master/doc/tutorials/tutorials.html
Optional resources Gazebo http://wiki.ros.org/gazebo PCL http://pointclouds.org/ ActionLib http://wiki.ros.org/actionlib MoveIt http://moveit.ros.org/
12 TCV152 introduction Project examples
Object detection: Detect a cup; Reach Object detection: Detect an elevator button; Reach Object detection: Detect an elevator; Approach Object detection: Detect a closed door; Approach Object detection: Detect an open door; Approach Object detection: Detect a door handle; Reach Face detection: Detect a known face; Orient toward it Text in the wild: Detect room numbers; OCR it View point change: From where should I look next? Reach
Do you have a related idea of your own? Let us know and get our approval
13 TCV152 introduction TCV152 Introduction to ROS
Ehud Barnea
Computer Science Department Ben-Gurion University of the Negev What is ROS
• What is ROS? • “The Robot Operating System (ROS) is a set of software libraries and tools that help you build robot applications.”
• More precisely: • ROS provides a flexible architecture for robot programs. • A simple plug-in system. • ROS is bundled with helpful libraries (e.g., OpenCV, PCL)
15 TCV152 introduction A common robot architecture
Processing
Sensors Robot architectures: Obstacle Object … • Many interacting components detection detection • Asynchronous communication Right eye
Left eye Actions • ROS provides the architecture, Decision making we provide the components Nose
Ears
16 TCV152 introduction ROS basic concepts
Nodes and Topics • Each program is a node • Nodes interact using topics
Images Objects Motion … Camera Object detector … Planner
Services • Provide a function-like mechanism (blocking)
Motion Camera Object detector Planner
…
17 TCV152 introduction ROS features
ROS provides useful tools to make your life easier
• Command line tools – e.g., list all topics, working nodes, etc.
• RQT console – follow topics
• ROS bags – record/play everything in the system
18 TCV152 introduction ROS code example - Publisher
talker.cpp
void main(int argc, char **argv) { ros::init(argc, argv, "talker"); ros::NodeHandle n; ros::Publisher chatter_pub = n.advertise
ros::Rate loop_rate(10); while (ros::ok()) { std::stringstream ss; ss << "hello world " << count;
std_msgs::String msg; msg.data = ss.str(); chatter_pub.publish(msg);
ros::spinOnce(); loop_rate.sleep(); } }
19 TCV152 introduction ROS code example - Subscriber
listener.cpp
void chatterCallback(const std_msgs::String::ConstPtr& msg) { ROS_INFO("I heard: [%s]", msg->data.c_str()); }
int main(int argc, char **argv) { ros::init(argc, argv, "listener"); ros::NodeHandle n; ros::Subscriber sub = n.subscribe("chatter", 1000, chatterCallback); ros::spin();
return 0; }
20 TCV152 introduction ROS code example - CMakeLists
cmake_minimum_required(VERSION 2.8.3) project(beginner_tutorials)
## Find catkin and any catkin packages find_package(catkin REQUIRED COMPONENTS roscpp rospy std_msgs genmsg)
## Generate added messages and services generate_messages(DEPENDENCIES std_msgs)
## Declare a catkin package catkin_package()
## Build talker and listener include_directories(include ${catkin_INCLUDE_DIRS})
add_executable(talker src/talker.cpp) target_link_libraries(talker ${catkin_LIBRARIES}) add_dependencies(talker beginner_tutorials_generate_messages_cpp)
add_executable(listener src/listener.cpp) target_link_libraries(listener ${catkin_LIBRARIES}) add_dependencies(listener beginner_tutorials_generate_messages_cpp)
21 TCV152 introduction Good luck
22 TCV152 introduction