well as the limited potential market Glancey, J.L., W.E. Kee, T.L. Wootten, for specialty harvesters for these minor and D.W. Hofstetter. 1998. Feasibility of Engineering and crops. once-over mechanical harvest of processing Horticultural Clear interactions exist between squash. ASAE Paper No. 98-1093. Amer. the cultivar, cultural practices, a Soc. Agr. Eng., St. Joseph, Mich. Aspects of Robotic mechanical harvester and several Glancey, J.L., W.E. Kee, T.L. Wootten, postharvest requirements. As a result, M.D. Dukes, and B.C. Postles. 1996. Field Fruit Harvesting: a system-level approach is critical losses for mechanically harvested green peas for developing economically viable, for processing. J. Veg. Crop Production Opportunities and highly automated vegetable produc- 2(1):61–81. tion systems. Furthermore, improve- Kahn, B.A., Y. Wu, N.O. Maness, J.B. Constraints ments in plant architectures and yields Solie, and R.W. Whitney. 2003. Densely planted okra for destructive harvest: III. and other modifi cations to crops are 1 2 required before some vegetables can Effects of nitrogen nutrition. HortScience T. Burks , F. Villegas , be machine harvested. Some of the 38(7):1370–1373. M. Hannan3, S. Flood3, attributes requiring further develop- Inman, J.W. 2003. Fresh vegetable harvest- 3 ment include, but are not limited to, ing. Resource: Engineering and Technol- B. Sivaraman , better fruit location within the plant ogy for Sustainable World 10(8):7–8. V. Subramanian3, and J.Sikes3 structure, more uniform fruit sets, in- Palau, E. and A . Torregrosa. 1997. Me- creased mechanical damage resistance, chanical harvesting of paprika peppers in prevention of premature or diffi cult Spain. J. Agr. Eng. Res. 66(3):195–201. ADDITIONAL INDEX WORDS. selective fruit detachment, and more robust harvesting, automated production, postharvest quality and stability. Roberson, G.T. 2000. Precision agriculture machine vision The integration of new tech- technology for horticultural crop produc- tion. HortTechnology 10(3):448–451. SUMMARY. Automated solutions for nologies including DGPS, automatic fresh market fruit and vegetable machine guidance, and computer- Upadhyaya, S.K., U. Rosa, M. Ehsani, M. harvesting have been studied by based vision systems offers signifi cant Koller, M. Josiah, and T. Shikanai. 1999. numerous researchers around the performance benefi ts, and is a substan- Precision farming in a tomato production world during the past several decades. tial component of current vegetable system. ASAE Paper No. 99-1147. Amer. However, very few developments have production and harvesting research in Soc. Agr. Eng., St. Joseph, Mich. been adopted and put into practice. The reasons for this lack of success are the U.S. As costs continue to decrease U.S Dept. Agr. 2004. Vegetables at a due to technical, economic, horti- for these new technologies, commercial glance: Area, production, value, unit value, cultural, and producer acceptance trade and per capita use. 25 June 2004. adoption will increase. issues. The solutions to agricultural
● January–March 2005 15(1) 79
JJan2005HT.indban2005HT.indb 7799 112/6/042/6/04 4:35:154:35:15 PMPM WORKSHOP
tion, vehicle guidance, and so on, each of robotic mechanization for horticul- crops requires major design compo- application will be specialized, due to tural crop harvesting systems. In order nents—machine, variety, and cultural the unique nature of the biological to provide the reader with suffi cient practices. A systems development ap- system. Collaboration and technology breadth of information, this paper is proach must be followed to insure that sharing between commodity groups primarily a literature survey and syn- all three aspects are considered (Sims, offers the benefi t of leveraged research and development dollars and reduced thesis, which tries to identify the key 1969). The major aspects related to overall development time for multiple issues that robotic system developers cultural practices that affect fruit and commodities. This paper presents an and horticultural scientists should vegetable mechanical harvesting in- overview of the major horticultural consider to optimize plant–machine clude fi eld conditions, plant population and engineering aspects of robotic system performance. and spacing, and plant shape and size. mechanization for horticultural crop Effi cient harvest mechanization cannot harvesting systems. Horticultural aspects of be achieved by machine design alone. robotic harvesting Establishing favorable fi eld conditions Robotic solutions for fresh market for the harvesting system under devel- everal horticultural commodity fruit and vegetable harvesting have opment has to be considered before groups around the nation are been studied by numerous research- the harvesting system can be effectively Sfacing growing global market ers around the world during the past developed (Wolf and Alper, 1983). pressures that threaten their long- several decades. However, very few Peterson et al. (1999) developed term viability. For instance, Brazilian developments have been adopted and a robotic bulk harvesting system for orange (Citrus sinensis) growers can put into practice. The reasons for this apples. They trained the apple trees produce, process, and ship juice to lack of success are due to technical, using a Y-trellis system and found them Florida markets cheaper than can economic, horticultural, and pro- to be compatible with the mechanical Florida growers. In the event that tariffs ducer acceptance issues. In industrial robotic harvesting. Fruit was trained to are eliminated, numerous horticultural automation applications, the robots’ grow on the side and lower branches to commodities across the nation will not environment is designed for optimal improve fruit detection and removal. be able to compete in either domestic performance, eliminating as many They further suggested that pruning or international markets with their variables as possible through careful could enhance the harvesting process counterparts in Latin America and systems planning. In agricultural set- by removing unproductive branches Asia. The combination of low com- tings, environmental and horticultural that block effective harvesting. Further modity prices both domestically and control can be a signifi cant hurdle to research was suggested to determine abroad, high labor prices, and low successful automation. Not only must the variety and rootstock combinations labor productivity presents signifi cant the plant system be designed for suc- most compatible with the training and challenges for U.S. agriculture. Several cessful automation, but the cultural and harvesting system. commodity groups, including Florida horticultural practices employed by the The concept of designing a grove tomato (Lycopersicon esculentum) and producers must often be changed to for optimal economic gain is being orange, California citrus (Citrus spp.), provide a plant growth environment in considered for citrus production. In the New York apple (Malus ×domestica), which robotic systems can be success- model grove concept, a grove must be and northwestern U.S. deciduous tree ful. According to Sarig (1993), “The designed for the optimal combination fruits recognize that harvesting labor major problems that must be solved of varieties, rootstocks, grove layout, is one of the most crucial challenges with a robotic picking system include production practices, and harvesting to maintaining economic viability for recognizing and locating the fruit, and methodologies, which will provide U.S. horticultural crops. According to detaching it according to prescribed maximum economic yield. economic studies, harvesting labor rep- criteria, without damaging either the PLANT POPULATION AND SPACING. resents over 40% of citrus production fruit or the tree. In addition, the ro- Harvesting equipment can operate cost and will need to be cut by 50% in botic system needs to be economically at maximum productivity when the order to maintain global competitive- sound to warrant its use as an alterna- workspace has been organized to mini- ness (Brown, 2002). tive method to hand picking.” If the mize ineffi cient obstacles, standardize The potential societal benefi ts plant growth systems can be modifi ed fruit presentation, provide suffi cient from agricultural robotic mechani- to improve harvestability, the robotic alleyways, and maximize fruit density zation are numerous. By sustaining system will have a much better chance on uniform growth planes. crucial commodities, the economic of being successful. Certain tree species and even infrastructure which supports these Modifi cations and improvements certain varieties within species have an industries will be reinvigorated. Rural of cultural practices for mechanization optimal subsistence area for best fruit communities will have new oppor- are continually being made through production, which provides a proper tunities for better jobs that have less research and experience (Sims, 1969). ratio between the number of leaves drudgery than traditional manual In order to have a successful auto- needed to produce carbohydrates and fi eld labor. Opportunities to improve mated/mechanized system, the cul- other organic compounds, and the worker health and safety by automating tural practices must be designed for number of developing fruits (Monselise dangerous operations have signifi cant the machine and the variety (Davis, and Goldschmidt, 1982). The woody potential. 1969). Cultural practices are a critical mass—roots, trunk, scaffolds, and The objective of this paper is factor in mechanization of fruit and branches—supports the tree canopy, to present an overview of the major vegetable production and harvesting. but contributes minimally toward fruit horticultural and engineering aspects Mechanization of fruit and vegetable development once nutrient uptake and
80 ● January–March 2005 15(1)
JJan2005HT.indban2005HT.indb 8800 112/6/042/6/04 4:35:164:35:16 PMPM moisture demands are met. However, pruning. The trees can be pruned to plant interior. In both cases, a plant the woody mass continues to use the the desired shape before fruit set and system that presented the majority tree’s resources to maintain itself, allowed to grow during the remainder of the fruit at the canopy surface presenting obstructions to robotic har- of the year. In some limited cases, severe would improve harvestability. There vesting. Ben-Tal (1983) suggested that pruning is being tested. Under this are two possible solutions. The fi rst maximum yield per unit area would be practice, alternating sides of the tree would suggest a thin leaf canopy so achieved by a large number of relatively are pruned each year and allowed to that the detection systems could more small trees, suggesting that smaller set fallow, while the other side of the easily view the plant interior, and the robotic systems may actually provide tree produces the current year’s crop. second suggests a dense canopy that a better economic return. When the canopy returns the following might force more fruit to grow at the Scalability of robotic systems is year, the woody mass is covered by the surface. The two strategies seem to be an important economic factor, which new growth and a relatively uniform in confl ict under normal tree behavior. impacts the design of the plant growth vertical wall is achieved. Impact of an- Sparsely leafed trees tend to have more system. The productivity of large mul- nual fruit yield has not been reported interior fruit, which reduces fruit ac- tiple-arm systems vs. smaller, more ag- on this technique to date. cessibility, while densely leafed trees ile, human-like robots is an important Experiments conducted on ap- will make it more diffi cult to sense the economic question. Large equipment ples demonstrated that tree shape interior fruit. A tree which is naturally systems require wide row spacings contributes toward the suitability of fruited at the limb extremities with while smaller systems can work in a mechanical harvesting (Zocca, 1983). minimal interior fruit might resolve more confi ned grove confi guration. Modifi cations to cultural practices this problem. Optimally, the fruit should be grown for growing and harvesting fruit are Another primary concern is in a hedge row confi guration where the important for successful mechanical canopy uniformity. Factors affect- plants produce a maximum number of harvesting. A mechanized pruner was ing uniformity in emergence, stand, fruits over the surface area (Ben-Tal, developed that not only reduced the growth, and maturity must be clearly 1983). This suggests that the trees labor required for pruning, but also understood in order to develop viable or plants be grown at a close spacing properly shaped the hedgerow for plant systems for mechanical harvesting so that the growth plane is uniform, maximum harvesting effi ciency of erect (Davis, 1969). Cultural practices have with minimal scalloping of the hedge cane fruits (Morris, 1983). been discussed that could produce a between plants. Ben-Tal (1983) pointed out sev- hedge-row system. However, trees PLANT SHAPE AND SIZE. The ideal eral problems that can arise when an that require severe hedging to main- confi guration for effi cient robotic har- orchard is prepared through pruning tain their shape often develop woody vesting would be a vertical or slightly for a specifi c kind of equipment, such structures near the surface, which could inclined hedge wall, 10 to 12 ft (3.0 to as reduced yield, fruit quality, and the be an obstacle to robotically harvesting 3.7 m) tall, that is relatively uniform, number of years of production. Ad- interior fruit. A tree that grew to an smooth, and continuous from start to ditional issues, such as canopy light appropriate mature height and shape row end. The fruit would be located exposure and maximum height of a and then maintained its size with ei- on the canopy surface with minimal tree for proper spraying, pruning, etc., ther minimal hedging or woody mass occlusion. In reality this would not should be considered. The question of build-up would be ideal. be the case, but the example provides plant geometry and its relationship to Several plant breeding projects some insight into what a robot would productivity needs to be thoroughly have contributed favorably to mechani- need in order to maintain fast harvest examined (Rohrbach, 1983). cal harvesting. Peach (Prunus persica) cycle times and maximum fruit re- TREE GENETICS FOR OPTIMAL breeders increased fruit harvest by moval. Deviations from the ideal will HARVESTING. Plant breeders developing releasing varieties with varying maturi- cost removal effi ciency and cycle time new varieties of fruit must consider if ties, effectively doubling or tripling the performance. the variety will be accepted at market length of the peach season in many Orchards should have uniform and if it will be durable under machine production areas (Carew, 1969). plant sizes and predictable shapes for handling. Attractive appearance and Dwarfi ng rootstocks in combination effi cient robotic harvesting (Cargill, long shelf life are imperative in the fresh with apple varieties have provided size 1983). Standardization of tree sizes market. Varieties must be resistant to control of apple trees. Plant improve- would signifi cantly improve harvesting bruising, cracking, and rupturing dur- ment through breeding can modify throughput and thus economic benefi t. ing machine handling. The fruit must crop characteristics and assist in the These standard sizes should consider be relatively easy to remove from the introduction of mechanical harvesting tree height, tree thickness, tree shape, plant and the peduncle must remain systems (Carew, 1969). and tree spacing within and between attached (Davis, 1969; Lapushner et rows so that the robotic equipment al., 1983). Engineering design aspects of can maintain continuous harvesting, In addition to fruit-related issues, robotic harveting with minimal idle harvest time when there are a number of tree factors Robotic systems developers from traveling between trees. A number of that can be improved genetically that the U.S., Europe, Israel, and Japan have these features are designed into the can enhance robotic harvestability. conducted independent research and grove at planting, while others must Two major obstacles impede effi cient development on harvesting systems for be maintained mechanically. A com- robotic harvesting: 1) locating fruit apples and citrus achieving harvesting mon modern approach for maintaining occluded by the leaf canopy; and 2) effi ciencies of 75%. These low levels both tree size and shape is mechanical harvesting fruit located in the tree or of performance were attributed to
● January–March 2005 15(1) 81
JJan2005HT.indban2005HT.indb 8811 112/6/042/6/04 4:35:164:35:16 PMPM WORKSHOP
poor fruit identifi cation and the in- varieties (C. unshiu) tended to be more still, however, susceptible to injury. ability to negotiate natural obstacles susceptible to undesirable detachment Injury is more prevalent in less mature inside the tree canopy (Sarig, 1993). conditions than other citrus varieties. oranges, as was found by Juste et al. Harvesting cycle times for citrus were ‘Clausellina’ exhibited 8% to 19% (1988). The resistance to pressure estimated at 2 s/fruit for a two-arm calyxless conditions, depending on was found to stabilize as the oranges machine (or 4 s/fruit for a single- fruit maturity, while ‘Navelina’ was matured at around 343.2 kPa (49.78 arm machine). Cycle time should be relatively constant at 2% to 3% calyx- psi) for ‘Salustiana’ and 441.3 kPa higher for apples due to a more open less. Fornes et al. (1994) also reported (64.01 psi) for ‘Washington’ using a canopy (Sarig, 1993). These levels of that detachment method and damage circular surface area of 1.1 cm2 (0.17 harvesting performance and economic varied with rotational speed for the inch2). Rind penetration tests were return prevented producer acceptance same three varieties. also performed using a 4.7-mm-diam- of robotic harvesting. Juste et al. (1988) researched the eter (0.19 inch) punch. Penetration PHYSICAL PROPERTIES AND FRUIT detachment forces of ‘Salustiana’ (C. si- force was found to be 28.9 N (6.50 REMOVAL. A robotic harvester must nensis) and ‘Washington’ navel oranges lbf) for ‘Salustiana’ oranges and 26.6 be able to quickly remove the fruit (C. sinensis). ‘Salustiana’ was found to N (5.98 lbf) for ‘Washington’ navel without damaging the fruit or the tree. have a detachment force 73.1 ± 0.6 N oranges. An integral part of the harvester is the (16.43 ± 0.13 lbf) and ‘Washington’ When manually harvesting or- end-effector, which is a tool or device was found to have a detachment force anges the fruit is detached using one attached to the end of the manipulator of 54.4 ± 4.7 N (12.23 ± 1.06 lbf). The of three methods, depending on the that grabs and removes the fruit from detachment force of ‘Washington’ was variety and cultural practice. The la- the tree. Because of its direct interac- found to substantially increase with an borer can use a set of clippers to detach tion with the fruit and tree structure, increase in maturity. All of these forces the fruit, usually leaving as short a it must be designed with the specifi c were measured along the stem axis. stem as possible. Secondly, the laborer physical properties of the commodity Fornes et al. (1994) also researched can lift the fruit so that the stem axis to be harvested in mind. detachment forces on ‘Clausellina’ and is rotated 90° and then pull down so There are several ways that a robot ‘Clemenules’ mandarins, and ‘Nave- that the force is perpendicular to the might damage the fruit or tree: 1) end- lina’ oranges. Detachment forces for all stem axis. Lastly, the laborer can add a effector applying excessive positive or three varieties were found to decrease twisting motion to the second method. negative pressure or force to the fruit as maturity increased. Although the end-effector does not during pick and place operations; 2) Juste et al. (1988) measured necessarily have to follow one of these inappropriate stem separation tech- torsional detachment by counting the methods, an understanding of manual niques for the type of fruit; 3) fruit number of turns required to detach the procedures gives insight into some of damage during retraction from the tree fruit. ‘Salustiana’ displayed an average the potential methods. canopy or conveyance to bulk storage; twist of 2.48 ± 0.12 revolutions and The fi rst type of robotic orange or 4) manipulator contact with the ‘Washington’ displayed an average harvesting end effectors that has been tree structure. Fruit damage may not twist of 2.36 ± 0.11 revolutions. The developed is the cutting end-effector. be physically evident immediately, yet maximum twist was 4.75 revolutions Several cutting end-effector designs bruising, scratches, cuts, or punctures for both varieties. Approximately 60% have been developed as described in will result in decreased shelf life. A of ‘Salustiana’ still had a stem, which Ito (1990), Sarig (1993), Pool and properly designed end-effector will was on average 3.82 ± 0.36 mm (0.150 Harrell (1991), and Bedford et al. attempt to minimize fruit damage. ± 0.014 inch) in length. Approximately (1998). This method is prevalent in The fruit removal technique em- 78% of ‘Washington’ navel oranges several agricultural applications since ployed is typically the largest cause of still had a stem, which was on aver- it produces the least amount of stress fruit injury. In the case of oranges, the age 6.33 ± 1.14 mm (0.249 ± 0.044 on the actual fruit. The basic premise fruit must be harvested with the calyx inch) in length. Rabatel et al. (1995) is to fi rst capture the fruit using a intact and the stem removed fl ush with stated that a stem length of 5.0 mm suction cup or gripper, and then use the calyx. If the peel is torn away from (0.20 inch) or less was desirable. It a cutting device to sever the stem that the caylx, the resulting fruit is unus- should be noted that in these tests the is holding the fruit onto the tree. This able for the fresh fruit market due to calyx remained intact on the torsional can either be done blindly by swing- contamination and reduced shelf life. detachments, but 70% of the direct ing a blade around the outer edge or This condition is referred to as “plug- pulling detachments had calyx separa- by detecting the stem’s location and ging.” If a long stem remains on the tion or displayed “plugging.” Coppock cutting it with a scissor device. The fruit, the packer will either reject the (1984) observed a near 50% reduction stem’s location can either be detected fruit or require stem removal post- in pulling detachment force when the through machine vision or through harvest. Fornes et al. (1994) reported force was applied at a 90° angle from force/torque sensors. citrus detachment conditions for fruit the stem axis for ‘Valencia’ oranges (C. In the blind system a blade passes removed using a prototype vacuum- sinensis). This reduction in detachment around the encased fruit to sever the grip rotational-separation end-effector. force might decrease the amount of stem without damaging adjacent fruit Fruit detachment conditions varied plugging exhibited. or the tree. The blade must be large with maturity for the three citrus va- The rind of oranges makes them enough to encircle the fruit, and must rieties reported: ‘Clausellina’ (Citrus one of the more durable fruits, in maintain sharpness to achieve a clean unshiu), ‘Clemenules’ (C. unshiu), and contrast with more delicate-skinned cut. The scissor method reduces the ‘Navelina’ (C. sinensis). The mandarin products, such as tomatoes. They are chance of fruit damage but is sub-
82 ● January–March 2005 15(1)
JJan2005HT.indban2005HT.indb 8822 112/6/042/6/04 4:35:174:35:17 PMPM stantially more complex, requiring a In robotic fruit harvesting sys- images that contained 673 oranges that larger end-effector, more sensors, and tems, machine vision has become one were taken at an actual orange grove. more time. of the most popular sensing systems Their algorithm correctly identifi ed The second type of end-effector for fruit identifi cation. With the advent 87% of the oranges, while 15% of the is the pull and cut end-effector. This of color cameras, differences between detected regions were incorrectly clas- method was proposed by Pool and leaf canopy and mature fruit can be sifi ed as oranges. Their approach had Harrell (1991). In this method, the discriminated. The vision system is able diffi culty with both brightly and poorly fruit is grasped either through suction to determine either two-dimensional lit oranges, brightly lit leaves, and or a type of collection sock. The stem (2D) or three-dimensional (3D) posi- certain types of occlusion. Bulanon et is severed as the end-effector retracts. tion in the canopy, depending on the al. (2001) presented an algorithm that This method disturbs the surrounding type of system employed. However, used a 240 × 240 pixel color image to limb structure, making subsequent the process of separating the desir- detect apples. The apples were detected harvesting more diffi cult since the fruit able fruit from other objects, such as by thresholding the image using both is in motion, and still has some of the leaves, branches, and unripe fruit, is the red color difference and luminance limitations of the cutting end effectors very diffi cult due to the large amount values. It was determined that the red previously mentioned. of information that must be processed. color difference values were much more The third type of end-effector This is a diffi cult task even with today’s effective at detecting the apples than design is the twisting method. This relatively high-speed computers. To the luminance values. method was suggested by Juste et al. put this into perspective, one 640 × There are three major problem (1998) and Rabatel et al. (1995) to be 480-pixel color image has almost one areas associated with the use of ma- the most promising of the three. This million data points, and a standard chine vision-based sensing: 1) partial involves twisting the fruit, preferably video camera runs at 30 frames per and totally occluded fruit are diffi cult perpendicular to its attachment axis, second, resulting in 30 million data to accurately detect; 2) light variability until the stem is severed. Twisting points to process every second. can result in low detection rates of ac- the fruit in this manner reduces the Machine vision systems common- tual fruit as well as high levels of false amount of disturbance to the tree ly applied to agricultural applications detections; and 3) the computational and thus to the surrounding fruit. acquire reflectance, transmittance, time required to process images as Twisting involves the least amount of or fl uorescence images of the object infl uences real-time control. force of the three methods and has the under ultraviolet, visible, or infrared Numerous other sensors are com- lowest plugging rate. Like the other illumination. A basic machine vision monly employed in robotic harvesting two types, fruit size is a consideration system includes a camera, optics, light- systems, such as ultrasonic range, laser here as well. Generally, the twisting ing, a data acquisition system, and a range, capacitive proximity, light emit- action is achieved by use of a rotating processor. Fujiura (1997) developed ting diode (LED) range, and so on. suction cup. This cup must be of the robots having a 3D machine vision It is not likely that a single sensor will right size to create a good seal while system for crop recognition. The vi- solve the complete sensing problem, still providing enough force to keep sion system illuminated the crop using but several sensors will need to be in- the orange from slipping. One of the red and infrared laser diodes and used tegrated together to form a complete major advantages of this method is that three position-sensitive devices to system. there is a large fl exibility in the angle of detect the refl ected light. The sensors ROBOTIC MANIPULATION AND approach. Except at the stem, the cup selected were suitable for agricultural CONTROL. The kinematic and geomet- can attach to any part of the fruit. robots that are required to measure the ric description of a robotic manipulator Tutle (1985) suggested an ap- 3D shape and size of targets within a is one of the key tasks in building a proach that combined the twisting limited measuring range. Jimenez et al. robotic system to work, especially in and pulling approach in U.S. Patent (2000) developed a laser-based vision unstructured environments. Harvest- 4,532,757. The end-effector design system for automatic fruit recognition ing of fruits (orange, apple, etc.) is selected for a given application should to be applied to an orange-harvesting highly unstructured and the robotic be developed in conjunction with the robot. The machine vision system arm should be fl exible enough to adapt manipulator, sensors, and control de- was based on an infrared laser range- to changes in the environment. Various velopment to optimize the capabilities fi nder sensor that provides range and geometric coordinate confi gurations of the harvester. refl ectance images and was designed used in industrial applications are avail- MACHINE VISION AND SENSING to detect spherical objects in non- able: cartesian, cylindrical, spherical, TECHNOLOGIES. According to Sarig structured environments. The sensor articulated, and redundant. (1993), “While major progress has output included 3D position, radius, In a 1968 review of mechanical been made with the identifi cation of and surface refl ectivity of each spheri- citrus-harvesting systems (Schertz and fruit on the tree and determination of cal target, and had good classifi cation Brown, 1968), the basic principles for its location, only 85% of the total fruits performance. utilizing robots to pick fruits were laid on the tree are claimed to be identifi ed. Plebe and Grasso (2001) pre- out. The earliest laboratory prototype Variability in lighting conditions and sented a color-based algorithm for was an apple harvester (Parrish and obscurity of fruits because of leaf and detecting oranges and determining the Goksel, 1997) consisting of a simple branch coverage (especially in citrus target centers. They also applied stereo arm with a pan-and-tilt mechanism trees), require further development imaging to these processed images to and a touch sensor in place of an end- of identifi cation techniques, or major determine the range to the detected effector, which made contact with changes in tree shape.” fruit. They presented results from 50 modeled fruit.
● January–March 2005 15(1) 83
JJan2005HT.indban2005HT.indb 8833 112/6/042/6/04 4:35:174:35:17 PMPM WORKSHOP
The fi rst fi eld prototype for har- vesting apples was developed in France (Grand D’Esnon, 1985). The mechani- cal system consisted of a telescopic arm that moved up and down in a vertical framework. The arm was mounted on a barrel that could rotate horizontally. In 1986, a new prototype (MAGALI) was built (Grand D’Esnon et al., 1987) that used a spherical manipulator ser- voed by a camera set at the center of the rotation axes. Figure 1 shows the spherical manipulator that can execute a pantographic prismatic movement (only rotational joints) along with two rotations. In 1986, the University of Florida, along with other collaborators, initi- ated a program to develop a robotic system for citrus harvesting (Harrell et al., 1988). The outcome of this research was a 3-degree of freedom (DOF) ma- nipulator actuated with servo-hydraulic drives. Joints 0 and 1 were revolute and joint 2 was prismatic. This geometry was characteristic of a spherical coordi- nate robot (Fig. 2). The feasibility of a Fig. 1. MAGALI apple-picking arm developed by Grand D’Esnon et al. (1987). robotic citrus harvester was ascertained by this research work. The French-Spanish Eureka project (Rabatel et al., 1995), started in 1991, was based on the feasibility study done at the University of Florida. The proposed robotic system had a dual harvesting arm confi guration to achieve greatest economic return; however, the prototype consisted of only one harvesting arm. The arm had two modules, an elevating arm and a picking arm. The picking arm was of a pantographic structure rather than a linear structure. The elevating arm supported the picking arm and the associated camera. The elevating arm was equipped with a lateral DOF to avoid collision with the picking arm with the vegetation, while acting as a fruit conveyor as well. Several manipulator architectures have been attempted for fruit harvest- Fig. 2. Citrus-picking robot developed by Harrell et al. (1988). ing. Of these, the articulated joint (6 DOF) seems to work the best, since reduced burden for the operator. Ve- with success, based on the degree of it closely resembles a human arm. In hicle position, heading, steering effort, accuracy required in the navigation order to avoid obstacles and to harvest and speed with respect to the desired system. There is a tradeoff between interior canopy fruit, the optimal con- path are the most important issues that accuracy and cost in the selection of fi guration for a robotic harvester may must be considered. Global positioning DGPS and RTKGPS, with the latter require more degrees of freedom than systems (GPS) in combination with being more accurate and expensive. a standard articulated joint. inertial navigation systems have been RTKGPS has been giving very accurate AUTONOMOUS VEHICLE GUID- widely used as positioning and heading results (Benson et al., 2001; Nagasaka ANCE. Autonomous vehicles are being sensors in traditional fi eld agriculture et al., 2002; Noguchi et al., 2002). developed for several applications, application. Both realtime kinematic Gyros have been widely used for in- including agricultural vehicles. The GPS (RTKGPS) and realtime differ- clination measurement (Mizushima et major advantages are precision and ential GPS (DGPS) have been tested al., 2002). Fiberoptic gyro (FOG) has
84 ● January–March 2005 15(1)
JJan2005HT.indban2005HT.indb 8844 112/6/042/6/04 4:35:184:35:18 PMPM given the best performance (Nagasaka al. (1992) pointed out that the fuzzy there are still major technological chal- et al., 2002). At present, gyros and controller could achieve better tracking lenges that need to be addressed, such inclinometers are available together as performance than the PI controller. It as locating occluded fruit, reaching the inertial measurement units (IMU) for has wider adaptability to all kinds of interior canopy, maintaining high cycle pitch, roll and yaw, and linear velocity inputs. Qiu et al. (2001) verifi ed that rates in diffi cult harvesting conditions, measurements. With the combination the fuzzy steering control provided cost affordability, and maintainability of RTKGPS and FOG, accuracy of a prompt and accurate steering rate of high-tech equipment. ±5 cm (2.0 inches) has been achieved control on the tractor. Kodagoda et al. The current citrus robotics devel- (Noguchi et al., 2002). GPS cannot (2002) found fuzzy control to be better opment initiative, which is being led by be used alone for positioning in citrus than PID for longitudinal control. PID the University of Florida, is seen as a applications, as it gives errors when the was also found to have large chatter, multidisciplinary effort where horticul- vehicle moves under tree canopies. high saturation. A combination of turalists, economists, and agricultural, In addition to sensing global fuzzy and PID control holds signifi cant mechanical, and electrical engineers are positions, the vehicle must be able to promise (Benson et al., 2001). Effi cient working together to solve some of the detect local obstacles that may impede guidance can be achieved using a fuzzy- complex problems mentioned previ- the path. Several sensing technologies PID control system with vision, laser ously. From a horticultural perspective, have been explored for this task. Ul- radar and IMU as sensors. research has begun to develop special trasonic sensors can map tree canopies orange varieties, which may be more while traveling at speeds of 1.8 m·s–1 Results and discussion favorable toward robotic harvesting. (5.91 ft/s); measurement accuracy is In Summer 2001, the Florida Additionally, research is beginning on better at lower speeds (Iida and Burks, Department of Citrus began an in- the development of a model grove, 2002). The development of machine vestigation into the potential for using which will attempt to optimize the vision guidance techniques has become robotics to harvest citrus. Current grove design and cultural practices for a very attractive sensing alternative, mass harvesting programs have proven mechanical/robotic harvesting and especially when combined with other viable for process citrus, but cannot maximum economic potential. From an proximity-based sensors (Benson et be used for fresh fruit markets and economic standpoint, studies are pro- al., 2001; Zhang et al., 1999). They remain questionable for late season posed that will evaluate the production have proven to be reliable in several ‘Valencia’, pending abscission chemical system for optimum economic return row-crop applications, but have not development and approval. A fact-fi nd- on investment. The potential labor performed well in sparsely populated ing team evaluated past horticultural productivity gains, projected capital crops. Their reliability reduces with robotics efforts, and talked to experts equipment cost, harvesting effi ciency, low lighting, shadows, dust, and fog. in robotics, agricultural mechaniza- value added benefi ts, and long-term Benson et al. (2001) overcame this tion, horticulture, and economics to yield impacts of robotic systems will be a by using artifi cial lighting. Laser radar determine if there had been suffi cient few of the factors that will be considered has been used for ranging and obstacle advances in technology and changes in evaluating the economic potential. avoidance. It has higher resolution in the economic potential for robotic In terms of engineering development than ultrasonic sensing, and requires harvesting to suggest that a renewed issues, the primary thrusts will be in fewer computations than vision. Its effort was warranted. The consensus the end-effector, manipulator system, performance degrades with dust and opinion of a Forum on Robotic Citrus sensing technology, material handling, rain like vision and it is costlier than Harvesting, held in Apr. 2002, was that vehicle guidance, and machine intel- ultrasound. It provides planar data there was an urgent need for harvest- ligence development. of the path, but can generate 3D by ing solutions for the fresh fruit market, There is a growing interest among rotating the laser source to give a 3D that signifi cant long-term fi nancial researchers working in other tree fruit view. O’Connor et al. (1995) found commitment would be required, and industries, along with their growers, to that sensor data is noisy, and can be although it is a diffi cult problem, pursue automation solutions to reduce fi ltered using Kalman fi lters to obtain enough technical progress has been the increasing disparity between U.S. robust sensor fusion. made in the past decade to warrant a production labor cost and those of Steering control is a major factor new robotics program. developing countries. However, it is for accurate guidance. PID control During the past two decades, clear that novel approaches need to (proportional, integral, derivative) has since the beginnings of research in be taken to solve robotics technology given satisfactory performance (Zhang agricultural robotics, there have been problems, as well as the manufacturing et al., 1999). Neural networks have numerous technological advances. and maintenance challenges that will the inherent disadvantage of learning The speed of computers has increased surface as high-tech equipment systems only what the driver does, so they are exponentially over the last 10 years. are implemented in harsh agricultural not robust. Behavior-based control is Manipulator technologies have im- environments. These challenges will a new development that has been suc- proved so that current manipulators require a high level of cooperation cessfully used in small mobile robots. A are faster and more dexterous. Machine among engineers, researchers, indus- behavior-based system in combination vision and other sensing technologies, try, and growers to insure that these with a real-time control system (RTCS) along with image and signal processing new systems will be able to meet their is expected to do well in vehicle guid- technologies, have greatly advanced. design objectives. An additional major ance. Fuzzy control has recently been Our understanding of horticultural issue is safety, since agriculture has been tried, with results comparable with practices and their impact on harvest- a notoriously dangerous workplace. PID (Benson et al., 2001). Senoo et ability has greatly improved. However, Thoughtful design and appropriate
● January–March 2005 15(1) 85
JJan2005HT.indban2005HT.indb 8855 112/6/042/6/04 4:35:204:35:20 PMPM WORKSHOP
industry standards will be required to ogy sharing between commodity Grand D’Esnon, A., G. Rabatel, R. Pel- build safety devices into the automated groups will offer the benefi t of lever- lenc, A. Journeau, and M.J. Aldon. 1987. systems that will protect operators and aged research and development dollars MAGALI: A self-propelled robot to pick casual observers. and reduced overall development time apples. Amer. Soc. Agr. Eng. Paper No. Some expected outcomes are the for multiple commodities. 87-1037. creation of new jobs associated with the Harrell, R.C., P.D. Adsit, T.A. Pool, and manufacture, operation, and service Literature cited R. Hoffman. 1988. The Florida robotic of machinery and instrumentation; grove-lab. Amer. Soc. Agr. Eng. Paper Bedford, R. Ceres, J.L. Pons, A.R. Jimenez, No. 88-1578. improvement of working conditions, J.M. Martin, and L. Calderon. 1998. and ability of laborers to become more Design and implementation of an aided Ito, N. 1990. Agricultural robots in Japan. skilled; and elimination of monotonous fruit-harvesting robot (Agribot). Ind. Inst. Electrical and Electronics Eng. Intl. and onerous tasks, such as heavy lifting, Robot 25(5):337–346. Wkshp. Intelligent Robots and Systems. digging, handling of sharp objects, and Ibaraki, Japan 1:249–253. Benson, E.R., J.F. Reid, and Q. Zhang. repetitive tasks. 2001. Machine vision based steering system Iida, M. and T.F. Burks. 2002. Ultrasonic Initial optimistic estimates have for agricultural combines. Amer. Soc. Agr. sensor development for automatic steering suggested that a 7- to 10-year multi- Eng. Paper No. 01-1159. control of orchard tractor. Proc. of the million dollar program will be required Conf. Automation Technol. for Off-Road to bring forth a market-ready citrus Ben-Tal, Y. 1983. Horticultural aspects Equipment. Amer. Soc. Agr. Eng. Publ. harvesting system. It is recognized of mechanical fruit harvesting. Proc. Intl. 701P0502:221–229. Symp. on Fruit, Nut, and Veg. Harvest- that signifi cant federal funding will be ing Mechanization. Amer. Soc. Agr. Eng. Jiminez, A.R., R. Ceres, and J.L. Pons. required to sustain the program for the SP-5:372–375. 2000. Vision system based on a laser duration of the development process. range-fi nder applied to robotic fruit har- Even with major federal research dol- Brown, G.K. 2002. Mechanical harvest- vesting. Machine Vision and Applications lars, there is a degree of uncertainty ing systems for the Florida citrus juice 11(6):321–329. regarding our ability to overcome all industry. Amer. Soc. Agr. Eng. Paper No. 021108. Juste, F., C. Gracia, E. Molto, R. Ibanez, technological barriers to high-speed and S. Castillo. 1988. Fruit bearing zones and high-effi ciency robotic harvesting. Bulanon, D.M., Kataoka, T., Zhang, S., and physical properties of citrus for me- The questions still remain of whether or Ota, Y., and Hiroma, T. 2001. Optimal chanical harvesting. Citriculture. Proc. not these new systems will be accepted thresholding for the automatic recognition Sixth Intl. Congr. Middle-East, Tel Aviv, by growers and provide an adequate of apple fruits. Amer. Soc. Agr. Eng. Paper Israel 4:1801–1809. return on investment. If not, these No. 01-3133. Kodagoda, K.R.S, W.S. Wijesoma, and robotics development efforts may meet Carew, J. 1969. The prospects for fruit E.K. Teoh. 2002. Fuzzy speed and steer- the same fate as the ones of the 20th and vegetable mechanization: Horticul- ing control of an AGV. Inst. Electrical and century. One major factor looms on ture outlook. Fruit and vegetable harvest Electronics Eng. Trans. on Control Systems the horizon: most major automation mechanization. Rural Manpower Ctr., Technol. 10(1):112–120. and mechanization efforts have been Michigan State Univ., East Lansing. Rpt. driven by high labor cost and/or in- No. 16:79–83. Lapushner, D., R. Frankel, and E. Edel- man. 1983. Genetical and cultural aspects adequate labor supply, both of which Cargill, B.F. 1983. Harvesting high density for mechanical harvested fresh market could become reality. red tart cherries. Proc. Intl. Symp. on Fruit, tomatoes. Proc. Intl. Symp. on Fruit, Nut, and Veg. Harvesting Mechanization. Nut, and Veg. Harvesting Mechanization Conclusions Amer. Soc. Agr. Eng. SP-5:195–200. 5(12):404–407. The solutions to agricultural Coppock, G.E. 1984. Robotic principles robotic mechanization problems are Mizushima, A., N. Noguchi, K. Ishii, and in the selective harvest of Valencia oranges. H. Terao. 2002. Automatic navigation of multidisciplinary in nature. Although Proc. First Intl. Conf. on Robotics and In- the agricultural vehicle by the geomag- there have been signifi cant technology telligent Machines in Agr., Amer. Soc. Agr. netic direction sensor and gyroscope. Proc. advances, many scientifi c challenges Eng., Tampa, Fla. ASAE. p. 138–145. Conf. Automation Technol. for Off-Road remain. Viable solutions will require Davis, V. 1969. Mechanization of fruit Equipment. Amer. Soc. Agr. Eng. Publ. engineers and horticultural scientists and pod vegetables introduction. Fruit and 701P0502:204–211. who understand crop-specifi c biologi- vegetable harvest mechanization. Rural Monselise, S.P. and E.E. Goldschmidt. cal systems and production practices, Manpower Ctr., Michigan State Univ., East 1982. Alternate bearing in fruit trees. Hort. as well as the machinery, robotics, Lansing. Rpt. No. 16:205–213. Rev. 4:155–158. and controls issues associated with Fornes, I., E. Molto, F. Juste, and A. automated production systems. Clearly Morris, J.R. 1983. Effects of mechanical Segura. 1994. Mechanical harvesting of harvesting on the quality of small fruits focused multidisciplinary teams are citrus: Evolution of damage produced and needed to address the full range of and grapes. Proc. Intl. Symp. on Fruit, fruit behaviour during post-harvest. EurAg Nut, and Veg. Harvesting Mechanization commodity-specifi c technical issues Eng 94. Rpt. N-94-G061. 5(12):332–348. involved. Although there will be com- Fujiura, T. 1997. Agricultural robots us- mon technology components, such as Nagasaka, Y., N. Umeda, and Y. Kanetai. ing 3D vision sensor. Robot. July 1997. 2002. National Agr. Res. Center, Japan: machine vision, robotic manipulation, 117:32–38. vehicle guidance, and so on, each ap- Automated rice transplanter with GPS and plication will be specialized due to the Grand D’Esnon, A. 1985. Robotic FOG. Proc. Conf. Automation Technol. harvesting of apples. Proc. Agri-Mation for Off-Road Equipment. ASAE Publ. unique nature of the biological system. 701P0502:190–195. However, collaboration and technol- 1:210–214.
86 ● January–March 2005 15(1)
JJan2005HT.indban2005HT.indb 8866 112/6/042/6/04 4:35:204:35:20 PMPM Noguchi, N., M. Kise, K. Ishii, and H. Rabatel, G., A. Bourely, F. Sevila, and F. Sims, W.L. 1969. Cultural practices for fruit Terao. 2002. Field automation using robot Juste. 1995. Robotic harvesting of citrus: vegetables. Fruit and Vegetable Harvest tractor. Proc. Conf. Automation Technol. State-of-art and development of the French Mechanization. Rural Manpower Ctr., for Off-Road Equipment. ASAE Publ. Spanish Eureka Project. Harvest and Post- Michigan State Univ., East Lansing. Rpt. 701P0502:239–245. harvest Technol. for Fresh Fruits and Veg.: No. 16:225–237. Amer. Soc. Agr. Eng. Proc. Intl. Conf., O’Connor, M., G. Elkaim, Jr., and B. Par- Guanajunto, Mexico: 232–239. Tutle, E. 1985. Robotic fruit harvester. kinson. 1995. Kinematic GPS for closed- U.S. Patent No. 4,532,757. U.S. Patent loop control of form and construction Rohrbach, R.P. 1983. Mechanized main- Offi ce, Washington, D.C. vehicles. Stanford Univ., Stanford, Calif. tenance of blueberry quality. Proc. Intl. Paper No. 10NGPS-95. Symp. on Fruit, Nut, and Veg. Harvest- Wolf, I. and Y. Alper. 1983. Mechaniza- ing Mechanization Amer. Soc. Agr. Eng. tion of paprika harvest. Proc. Intl. Symp. Parrish, E.A., Jr., and A.K. Goksel. 1997. SP-5:134–140. on Fruit, Nut, and Veg. Harvesting Pictorial pattern recognition applied to fruit Mechanization Amer. Soc. Agr. Eng. SP- harvesting. Trans. ASAE 20(5):822–827. Sarig, Y. 1993. Robotics of fruit harvesting: 5:265–275. A state-of-the-art review. J. Agr. Eng. Res. Peterson, D.L., B.S. Bennedsen, W.C. 54(4):265–280. Zhang, Q., J. Reid, and N. Noguchi. 1999. Anger, and D. Wolford. 1999. A systems ap- Automatic guidance control for agricul- proach to robotic bulk harvesting of apples. Schertz, C.E. and G.K. Brown. 1968. tural tractor using redundant sensor. Soc. Trans. of the ASAE 42(4):871–876. Basic consideration in mechanizing cit- Automation Eng. Trans., J. Commercial rus harvest. Trans. Amer. Soc. Agr. Eng. Vehicles (108):27–31. Plebe, A. and G. Grasso. 2001. Localiza- 11:343–346. tion of spherical fruits for robotic harvest- Zocca, A. 1983. Shake harvesting trials with ing. Machine Vision and Applications. Senoo, S., M. Mino, and S. Funabiki. 1992. market apples. Proc. Intl. Symp. on Fruit, 13(2):70–79. Steering control of automated guided Nut, and Veg. Harvesting Mechanization vehicle for steering energy saving by fuzzy 5(12):65–72. Pool, T.A. and R.C. Harrell. 1991. An reasoning. Proc. 1992 Inst. Electrical and end-effector for robotic removal of citrus Electronics Eng. Ind. Applications Soc. from the tree. Trans. ASAE 34(3/4):373– Annu. Mtg.:1712–1716. 378. Qiu, H., Q. Zhang, and J. Reid. 2001. Fuzzy control of electrohydraulic steering systems for agricultural vehicles. Trans. ASAE 44(6):1397–1402.
● January–March 2005 15(1) 87
JJan2005HT.indban2005HT.indb 8877 112/6/042/6/04 4:35:214:35:21 PMPM