You are viewing a javascript disabled version of the site. Please enable Javascript for this site to function properly.
Go to headerGo to navigationGo to searchGo to contentsGo to footer
In content section. Select this link to jump to navigation

Road detection in image by fusion laser points based on fuzzy SVM for a small ground mobile robot

Abstract

Road detection is still full of challenge for a small ground mobile robot with limited load capacity and computing resources which works in complex outdoor environment. This paper proposes a road detection method based on fuzzy support vector machine with on-line updating and retraining strategy. The algorithm extracts multi feature in image and trains a fuzzy support vector machine road classifier off-line by using few training samples. Then it detects road in laser points using a fuzzy clustering method based on maximum entropy principle. After calibrating the camera and laser range finder, and project laser points into the image, the algorithm chooses road samples with high confidence automatically according to range data and designs a rule to retraining the FSVM on-line when needed to improve its environmental adaptability. Experiments in outdoor campus environment indicate that the proposed algorithm is effective.

1Introduction

Reliable road detection is crucial to the success of many road scene understanding applications, such as unmanned ground mobile robot, autonomous driving, and driver assistance [3]. Furthermore, it can serve as a preprocessing stage for higher level reasoning and understanding of road scenes. Despite recent progress such as lots of high performance sensors have been used in this area, we are still far from having ideal solutions. Road detection remains a challenging computer vision problem due to the large variability of images acquired at different times of the day, with changing illumination, weather, different environments, and variable road conditions.

The common ground mobile robot or autonomous driving system is built on a vehicle type platform [10, 16, 17, 36]. Sensors can get relatively good field of vision on this kind of platform, which means both roadside can be seen in urban environment, so the algorithm can detect road curbs, road markings, and estimate road shape, etc. In this paper, we focus on another kind of robot platform, i.e. the small ground mobile robot as shown in Fig. 1. This is a kind of very flexible platform which makes it is suitable to complete lots of tasks, and it usually moves in structured or semi-structured road environment as its pass through ability is not strong. Compared with vehicle type platform, the small robot has to face following obvious shortcomings:

  • it gets very poor visual field and can only see a very local area in front of it, which makes it is more practical to detect road surface than detect road curb for it.

  • it is impossible for it to carry lots of high performance sensors as the load capacity and computing resources are very restricted in this kind of platform.

Under this circumstance, the small robot almost cannot see road curbs, so finding where does road locates is crucial for it in order to ensure its safety when moving on the road. It usually carries fewer vision sensors, typically including a low coast camera and a small laser range finder. According to this kind of sensor configuration, we present a practical road detection algorithm based on fuzzy theory in this paper. It extracts color, texture and gradient in image and train a road classifier off-line based on fuzzy SVM model at first, and then the algorithm uses a fuzzy clustering method to find road points in laser range finder data and maps road points into image after extrinsic calibrating of the camera and laser range finder. The algorithm selects road samples with high confidence (higher fuzzy membership) to replace low confidence samples in training set and retrain the FSVM classifier on-line when necessary.

The main contribution of the proposed algorithm is that we use high confidence laser detection result to guide reliable road samples chosen in image automatically and to retrain the FSVM on-line when needed to make the algorithm get high adaptability. So it is an off-line and on-line learning based algorithm, and for the on-line learning part, the algorithm can select and update reliable training samples by itself. This algorithm is mainly depending on on-line learning, which means we can use few samples to do the off-line learning. Actually, we only select samples in the first frame to do the off-line training in our experiment and this can be very clear to improve the training efficiency. We introduce fuzzy theory into the algorithm in order to improve its capability of anti-noise.

The organization of this paper is as follows: in Section 2, we summarize some related robot systems and road detection methods; Section 3 shows how to extract multi image feature to train a road classifier based on FSVM; the sample updating and classifier retraining rules is introduced in Section 4; our experiments and results are presented in Section 5; we finish the paper in Section 6 with a brief analysis of the proposed method and introducing our future work.

2Related works

Road detection for unmanned ground mobile robot has been researched in the past two decades. The earlier systems only use camera to detect road in gray or colored image, such as NAVLAB [12], VaMP [33], ARGO GOLD [1], etc., then lots of other sensors like radar or lidar have been used to deal with this problem with the development of sensor technology. Table 1 shows some typical systems that detecting road based on sensor fusion.

As far as we know, there is no universal road detection algorithm until now. Researchers usually develop their own road detection system to meet the needs of a specific application, so there are lots of road detection methods. More than one classification criteria can be used to categorize these methods. For one hand, according to sensors being used, road detection method can be categorized into camera-based [1, 4, 6, 7, 12, 14, 15, 21–23, 25, 27, 31, 33, 37– 39, 43, 46], Lidar-based [2, 9, 20, 28, 35, 47, 45], and fusion-based [11, 19, 26, 32, 40– 42]. From the beginning, the camera was used for road detection [1, 12, 21, 33], and has been extended to nowadays [6, 7, 14, 15, 22, 23, 25, 27, 31, 36– 39, 43, 46]. One can extracts road feature in image to train a classifier, but the detection results may be unstable in variable illumination, textureless, or cluttered scenes. With the development of laser measurement technology, researchers find it is easier to detect road in range data than in image and Lidar-based methods can achieve reliable and accurate results in their valid range. But range sensor usually takes low resolution and small visual field compare to camera. References [11, 19, 26, 32, 40– 42] fuse different kind of sensors to improve adaptability and robustness of the detection algorithms. Depth camera like Kinect is also used for indoor robot to detect object [40, 42] in recent years, but this kind of sensor is still not suitable for outdoor working.

For another hand, road detection methods can also be categorized into feature-based [15, 18, 19, 42], model-based [25, 44, 46] and region-based [11, 19]. Feature-based method extract road features such as color, texture, boundary, etc. to detect road, and it usually get high accuracy. But it requires the road has distinct feature identification and is sensitive to outliers. Model-based method is based on the assumption of the road model and matches the road template and image to find road. This method gets high robustness when the road model is good, but the shape of road is ever-changing in real world which makes it is very hard to build a universal road model. Region-based methods try to segment the image with road and non road area by training a classifier using multi-feature. For semi-structured or unstructured road in complex outdoor environment, region-based method usually has strong adaptability. In this paper, we focus on how to detect road by using a low cost camera and a small range finder based on image segmentation.

Road detection methods based on image feature are more easily affected by changing environmental factors such as illumination, so nowadays researchers either add prior knowledge such as road shape, or fuse camera with other kind of sensors such as lidar or IR camera, etc. At the same time, existing algorithms which use laser data are usually based on dense point cloud, especially since some high definition Lidar systems have been developed recent years. Dense point cloud contains abundant geometric information which can be used to extract more road feature or build road model, but it is also need great consumption of computing resources.

Comparing our small mobile system to vehicle platform based robot, we get poor visual field, weak performance sensors, and low computational power, so we believe that a method colligates sensor fusion, machine learning and region-based is a good way to solve our problem.

3Off-line training of FSVM

Detecting road from image can be considered as an image segmentation problem. We segment image into road area and non road area according to road color, road texture and road boundary. Generally speaking, road has different color from other area in semi-structured environment like campus or urban. The color of road usually bias blue, and gets greater brightness than non road area. Because of influenced by plants and soil, the color of non road area usually bias red and green. Texture is an expression of the intrinsic properties of the surface of the object. Texture generally exists in nature, such as fingerprints, water, wood etc., and the road texture is showed as the change of pixel gray or color. Road boundary is also an important feature as it limits the road area.

Based on the above analysis, we use color, texture and boundary as road feature in this paper and employ a FSVM model to train a classifier off-line with three main steps which are feature extraction, fuzzy SVM training and refinement. The detail of each step is expressed as follows:

3.1Image feature extraction

We extract color, texture and gradient as image feature to train a classifier. The algorithm splits an image into patches with size of 16×16 pixels at first, and then extracts each patch’s feature:

  • Color

R, G and B color components are highly correlated in RGB model, so we cannot distinguish two colors based on their RGB color distance. In contrast with RGB, HSI has three independent components as shown in Fig. 2. It separates intensity out of hue and saturation, so H and S components are less influenced by the change of illumination. Based on this, we use HSI color model in this paper and calculate the average HSI color value of pixels in one patch as that patch’s color.

  • Texture

We compute four kinds of Gray-level co-occurrence matrix (GLCM) as texture of road:

Angular second moment: ASM reflects the uniformity of the gray level distribution of the image. When the distribution of gray level is uniform, the elements are concentrated in the main diagonal of the GLCM, and the ASM value is large, which means the texture is uniformly distributed.

(1)
ASM=i=1nj=1nG2(i,j)

Contrast: Contrast reflects the clarity of image texture. The distance from the diagonal of GLCM is large when the texture with strong contrast.

(2)
CON=i=1nj=1n[(i-j)2G(i,j)]

Correlation: Correlation reflects the correlation of regional gray value. The distribution of the elements is more uniform and equal in GLCM with higher correlation.

(3)
COR=i=1nj=1nijG(i,j)-μiμjσi2σj2
where
μi=i=1nij=1nG(i,j),μj=j=1nji=1nG(i,j)σi2=i=1n(i-μi)2j=1nG(i,j),σj2=j=1n(j-μj)2i=1nG(i,j)

Entropy: Entropy reflects quantity of information of the image. The GLCM and entropy is zero without texture. If one image contains rich texture information, each patch’s entropy is approximately equal which means the entropy is large. If the differences among patches are large, then the entropy is small.

(4)
ENT=ijG(i,j)log2G(i,j)

We compress the gray level of the image into 16. The algorithm computes texture of four directions{0°, 45°, 90°, 135°} to eliminate the effect of direction of texture and take the average value of four direction as the texture feature.

  • Gradient

We compute the gradient of each pixel according to its four neighbor pixels in a patch and take the average gradient of pixels in a patch as the patch’s gradient g.

The feature vector is expressed as Equation (5) after feature extracting.

(5)
F=[H,Ss,I,ASM,CON,COR,ENT,g]

3.2Training of FSVM

The algorithm employs a FSVM [8] to train the road classifier. FSVM can solve data set with noise better than traditional SVM. Our robot carries a low cost camera with low resolution and image quality, so FSVM is suitable for processing such images.

Each sample has a fuzzy membership in FSVM which indicates how much the sample belongs to a certain class. Value of membership decides the contribution to the decision function of one training sample. By introducing membership, the FSVM can decrease the effect of samples with noise to the trained classifier. Suppose the training samples are expressed as:

(6)
S={(s1,l1,a1),(s2,l2,a2),...(si,li,ai),...(sm,lm,am)}
in which s i represents training samples, s i  ∈ R n . l i is class label and l i  ∈ {-1, 1}. a i is the membership of one sample belongs to a certain class and a i  ∈ (0, 1]. The optimal solution of the FSVM is formulated as Equation (7)
(7)
min12ω2+Ci=1maiξi
with constraint
(8)
li((ωsi)+b)+ξi1,ξi0
then the optimal discriminate function of the FSVM is
(9)
f(x)=sgn[(ω*x)+b*],xɛRn
in which
ω*=ai*lisi,b*=li-liai*(sisj),a*=maxai-12aiajliljK(sisj),subjectoliai=0
and we take Radial Basis Function (RBF) as the kernel function K in this paper. Then we rewrite the feature vector according to FSVM model as
(10)
F=[H,S,I,ASM,CON,COR,ENT,g,a]
in which the last variable a indicates the membership of one image patch. The algorithm takes the average membership of pixels in a patch as the patch’s membership value. We use the following function to compute membership
(11)
sc=1ni=1nsi,r=maxsc-si,a(si)=1-si-scr+δ

x c represents the central of a class (road or non-road), r is radius, δ indicates a small positive constant to make sure a (s i ) >0.

The algorithm selects road and non road samples manually in image as showing in Fig. 3(a) and train the FSVM. We want to emphasize here that we use few samples to do the off-line training. More precisely, we select training samples only in the first image frame in our experiment. This obviously can greatly improve the efficiency of off-line learning, but also bring higher error rate when on-line use. So we design a rule to update training samples and retrain the classifier on-line which will be explained in Section 4. The whole algorithm is mainly depending on on-line training to get better environmental adaptability.

3.3Refinement of road detection result

We use relatively small samples to train the FSVM in order to get high efficiency, and we find there are lots of wrong classified patches after initial classification (see Fig. 3(b)). So the algorithm uses following rules to refine the initial result:

  • take a 8×8 template to do morphological filtering to the result image to eliminate small and isolated patches in segmented binary image (see Fig. 3(c));

  • road is located in the 60% of the lower part of an image and patches belong to road should be connected, so we only take the biggest area which located in the lower part of the image as road (see Fig. 3(c));

  • as we use patches instead of pixels to do computation, the road boundary in binary image is jagged. The algorithm takes median filtering to smooth the boundary (see Fig. 3(d)).

We can see the result of refinement step is good by contrasting (d) and (b) of Fig. 3.

4Update training samples on-line automatically by fusion laser points

As laser range finder gets high ranging accuracy, so we use road information detected in laser points to help us choosing high confidence road samples in image to improve the on-line performance of the algorithm. We detect road in laser points and register the range finder and camera to project laser points into the image at first, and then, reliable road samples are chosen according to pixels projected by laser points which are dart on road. The FSVM will be retrained using these road samples if needed.

4.1Road detection in laser points

Laser points are collected by a UTM-30LX laser range finder. The range finder takes an inclined angle when installing in order to “look on” the road. We employ a fuzzy clustering algorithm based on Maximum Entropy Principle (MEP) [44] to cluster laser points in a data frame and find points which are located on road.

It is assumed the threshold of prediction error is δ, the next predicted laser point’s position is (xˆi,yˆi), and the real measured position is (x i , y i ). The Euclidean distance between (xˆi,yˆi) and (x i , y i ) is defined as

(12)
ei=(xi-xˆi)2+(yi-yˆi)2

The point will be considered to be the start of another cluster if e i  > δ. The parameter δ should be determined according to the distance between range finder and the target. We use Equation (2) to compute δ.

(13)
δ=max{δt}
where δt=2axt2+yt2sinθ, t = i - 1, i, i + 1.

Parameter a in Equation (13) is a scale factor to counteract noise. Parameter θ is half of angular resolution of azimuth of UTM-30LX. (x t , y t ) is the coordinate of a point p i .

Suppose every frame of range data gets N points and the system model can be characterized by M cluster centers C1,C2,...,CM in an N-dimension space. Each cluster center Ci can be represented by a vector that is composed of a pair of component vectors: the input vector Qi and the output vector Yi. For Ci let the first p dimensions correspond to p input variables that constitute the input vector Qi and the other m-p dimensions correspond to m-p output variables that form the output vector Yi.

There will be a prediction error at each time when predict the position of the next point, and the accumulated error problem is on the table if we ignore this error. In our algorithm, each vector in input vector Qi is presented a

(14)
Xi=[(xi,yi),ei]
where error item e i take part in prediction process too, and then minus the last predicted error in next prediction to reduce the effect of accumulate errors. Then our cluster centers are formed as follows:

(15)
C1:Q1=[Xj-p-M+1,Xj-p-M+2,...,Xj-M]TY1=[Xj-M+1]TC2:Q2=[Xj-p-M+2,Xj-p-M+3,...,Xj-M+1]TY2=[Xj-M+2]TCM:QM=[Xj-p,Xj-p+1,...,Xj-1]TYM=[Xj]T

For data point measured at time t, the probability of it belongs to the cluster centers can be viewed as its fuzzy membership μ i , where μ i  ∈ [0, 1] and i = 1, 2, . . . , M. The summation of all μ i ’s is equal to one. The clustering process can be formulated as an optimization problem and the corresponding cost function to be minimized is defined as:

(16)
E=i=1MμiP-Xi2
where Xi is the input vector of fuzzy centers.

The distribution of μ i is unknown in our problem. According to the information theory, the MEP is the most unbiased prescription to choose the values of the membership, μ i , for which Shannon entropy, i.e., the expression

(17)
H=i=1Mμilnμi
is maximal under the constrains by Equation (17). This optimization problem can be reformulated as the maximization of the Lagrange
(18)
L=H-λE
where λ is the Lagrange multiplier.

The final solved probabilities are of the Gibbs distribution [30, 29], i.e.

(19)
μi=exp(-λP-Xi2)i=1Mexp(-λP-Xi2)

Take Equation (20) into Equation (16), and versus λ differentiate, and then take the experiential parameter [29] we get:

(20)
λ=100M/min(P-Xi2)

Weighting reasoning mechanism is used to compute the output vector as our system present a linear trend. The following formula will account for that trend well.

(21)
Y=i=1MwlμlYl+1M1l=1M-1(Yl+1-Yl)
where w l is the weighting term:
(22)
wl=lM/l=1Ml

Hereto, we have introduced the clustering step. It does not require training which enables it to work on-line completely.

The ideal road model is shown in Fig. 4, but usually we cannot see the both side of the road because of occlusion. So it is unreliable to detect left and right curb in one laser frame data to find the road. Fortunately, the road in campus and urban is usually flat, so laser points dart on road satisfies linear distribution. We do linear fitting of each cluster of points and take clusters with small linear fitting mean square error as the candidate road cluster. There are lots of linear distributed point clusters are not belong to road obviously as any sample points from regular surface are linear distributed. We have to design some filters to find real road in range data.

We known the angle of yaw, roll and pitch of the robot by using a gyro and we transform points to the robot coordinates O R , The road surface around the robot should be horizontal in O R so we search horizontal linear distributed clusters of points after project them into ground plane. Suppose the included angle between the horizontal and a linear distributed cluster r i is β and the average height of points in r i is h r i , the height of the robot located is h R . The length of r i is L and the least width that the robot can be passed through is W. Then the first three road filters are expressed in Equation (23). Δβ is the threshold of the error of horizontal and h is the highest height of road surface. Δh is the threshold of the height differences between the robot current located area and the front road area.

(23)
F1:|β|<ΔβF2:hri<hF3:L>W

(24)
F4=|hri-hR|<Δh
(25)
R={ri|riF1F2F3F4}

Equation (23) is used to analyze the single data frame of range finder. Points in one frame are sparse, so we use the temporal and spatial correlation in the process of robot motion to increase the confidence of road detection. The height of the road should be consistent in a local area, and here will be obstacle exist if situation shown in Fig. 5 appeared. We use Equation (24) to limit the change of the road height at adjacent moment.

At last, clusters of points r i satisfy Equation (25) will be chosen as road. Figure 6 shows the road detection result in laser points. Road detected by range data usually get higher confidence than that detected in image so we use this information to help find reliable road samples in image automatically.

4.2Registration of the ranger finder and camera

The calibration of the camera and laser range finder is shown in Fig. 7(a) and we use the method proposed in [34] to solve this extrinsic calibration problem. Laser points can be mapped onto the image after calibration as shown in Figs. 7(b) and 7(c).

4.3Updating training samples on-line

We use only 200 training samples in order to improve training efficiency which will inevitably leads to the classification error become higher and higher along with the robot’s movement and the change of environment. So we design the rule to update the training samples on-line to improve the adaptability of the proposed algorithm.

The algorithm takes the patches in image under the mapped road line after projecting road laser points onto the image (shown in Fig. 7) as road samples with high confidence. This is reasonable because the field of vision of camera mounted on our robot is very low. Figure 8 shows an ideal situation without obstacles on road. The laser point dart on road projected onto image will become multi-line with obstacle and the algorithm only take patches under road lines as the road samples under this condition. The algorithm uses the upper left and upper right area as the high confidence non-road samples.

According to the on-line estimated classification error computed the algorithm decides when to update samples and retraining the FSVM. It takes the coarse segmentation image subtraction the corresponding refined one to get error classified patches S road and S non - road and who will be chosen as updating samples C road and C non - road if they satisfy following constraints: (1) S road and S non - road are not locate on around the boundary between road and non-road area in image as the boundary may not clearly enough to be choose as high confidence samples (gray area in Fig. 2(e)); (2) The area of S road and S non - road should be big enough. These tow constraints can be expressed as:

(26)
Croad={s1,s2,...,sn|siSnon-road&&area(si)>th&&siSboundary}Cnon-road={s1,s2,...,sm|sjSroad&&area(sj)>th&&sjSboundary}
(27)
r1=i=1nNUM(si)NUM(Snon-road),siCroadr2=j=1mNUM(sj)NUM(Sroad),sjCnon-roadp=αr1+βr2
where r 1 represents error rate of classified non-road as road, r 2 represents error rate of classified road as non-road, S road and S non - road represent area that be classified as road and non-road, α and β are weights. When r 1 or r 2 bigger than eth, or p is bigger than p th, we updating training samples of non-road or road and retraining the FSVM. We use the samples with higher value of fuzzy membership to replace samples with lower fuzzy membership in training set.

So far, we have introduced out road detection method. We summarized the proposed algorithm as the flow chart shown in Fig. 9.

5Experiments

We use the robot platform shown in Fig. 1 to collect experimental data in real outdoor campus environment. This robot is a very small ground mobile platform with size of 0.36 m (L)×0.35 m (W)×0.45 m (H), and equips a small DAHENG Mercury camera, a HOKUYO UTM-30LX laser range finder, a IG-500 N GPS aided AHRS and odometer.

Fig. 10 shows some experiment result: Original pictures of the scene with extracted road boundary are shown in Figure 10(a). Result of road detection by SVM and the proposed method based on FSVM are shown in Figs. 10(b) and 10(c) respectively. Figure 10(d) shows the final result. We can see that Fig. 10(c) got better result than Fig. 10(b), which means the proposed method has stronger environment adaptability. Updating and retraining FSVM can deal with factors of environmental changing.

We test the algorithm by using continuous frames while the robot moving in campus. Figures 11(a) and 11(b) shows original image and result of road detection in 8 continuous frames. There exist an intersection, shadow and changing illumination in these 8 frames during the robot moving. Road detection result shows the proposed algorithm has the ability to deal with these changing environment factors.

Figures 11(c) and 11(d) are error rate of detection results of 100 frames by using FSVM and SVM respectively. The lateral axis indicates number of frames and vertical axis indicates detection error rate of one frame. Suppose N is the total number of patches in one image, and N err is the difference number of patches between FSVN or SVM classification result and manual labeled ground truth, then the error rate is defined as r err  = N err /N. We can see the average classification error rate of FSVM is 6.3% , which is better than 8.5% gotten by SVM. We give full consideration to the samples for each category of membership degree when use FSVM, so the error rate is low at the first frame in (c) compared to (d). And the error rate is relatively stable in continuous detection of 100 frames. Thus the proposed algorithm is not only improves the accuracy of classification, but also can better adapting to environmental changes.

The effect of refinement and the comparison between without and with on-line training is tested. Form Fig. 12(b) we can see that the sky is easily classified as the road without refinement, and the results are improved a lot after refinement as shown in Fig. 12(c). Road detection results are further improved when on-line learning is used by comparing Figs. 12(d) and (c).

Previous experimental results show the algorithm has good environmental adaptability to deal with road scenes. In urban environment, obstacles are one of the factors that affect road detection result, so we test the proposed algorithm in campus when there are different obstacles on road. Figure 13 shows typical detection results, and we can see all obstacles that on road are detected. We know that color and texture of obstacles in urban environment usually different from the road, so it is not very hard for the proposed algorithm to recognize obstacles theoretically, and experiments in real environments also validate this according Fig. 13.

6Conclusion

Road detection is still full of challenge, especially for a small ground mobile robot with limited load capacity and computing resource who works in complex outdoor environment. In this paper we present a road detection method based on fuzzy theory. It extracts multi image feature and trains a FSVM classifier with few training samples off-line to get higher training efficiency. Then an on-line training sample updating and retraining strategy is used to make the algorithm has strong adaptability to adapt dynamic outdoor environment. As we use low performance sensors, we introduce fuzzy theory in our algorithm to improve its anti-noise capability. Experiments in the real campus environment validate the method.

Our future research will focus on tilting the laser range finder to get a range image and use this kind of data to navigate the outdoor mobile robot. We have built a tilting system and registered multi-frame laser data in static state already, but it is hard to get high-accuracy range image when the robot moves on rugged ground because of the big random shaking. We will try our best to deal with this problem and design a small and high-performance outdoor environmental detection and understanding system for intelligent ground mobile robot.

Acknowledgments

This work was supported by the Natural Science Foundation of Jiangsu Province of China under Grant No. BK2012399, the doctoral program of higher education of specialized research fund of China under Grant No. 20123219120028, the Key Laboratory of Intelligent Perception and Systems for High-Dimensional Information (Nanjing University of Science and Technology), Ministry of Education under Grant No. 30920140122006 and the Fundamental Research Funds for the Central Universities under Grant No. 30915011321.

References

1 

Broggi A, Pavia U, Bertozzi M, Fasciol A 1999 ARGO and the MilleMiglia in automatic tour Intelligent Systems and their Applications 14 55 64

2 

Hervieu A, Soheilian B 2013 Road side detection and reconstruction using LIDAR sensor In Proceedings of the IEEE on the Intelligent Vehicles Symposium, Gold Coast 1247 1252

3 

Geiger A, Lenz P, Urtasun R 2012 Are we ready for autonomous driving? The kitti vision benchmark suite RI, USA CVPR, Providence

4 

Seibert A, Hahnel M, Tewes A, Rojas R 2013 Camera based detection and classification of soft shoulders, curbs and guardrails IEEE Intelligent Vehicles Symposium, Gold Coast 853 858

5 

Kim BS, Xu SL, Savarese S 2013 Accurate localization of 3D objects from RGB-D data using segmentation hypotheses, IEEE Conference on Computer Vision and Pattern Recognition 3182 3189 Oregon, USA

6 

Wang B, Fremont V 2013 Fast road detection from color images IEEE Intelligent Vehicles Symposium (IV) 1209 1214 Gold Coast City, Australia

7 

Wang B, Fremont V, Rodriguez SA 2014 Color-based road detection and its evaluation on the KITTI road benchmark,IEEE Intelligent Vehicles Symposium Proceedings 31 36 Dearborn, Michigan, USA

8 

Lin CF, Wang SD 2002 Fussy support vector machine IEEE Trans. on Neural Networks 13 464 471

9 

Habermann D, Carcia C 2010 Obstacle detection and tracking using laser 2D Latin American Robotics Symposium and Intelligent Robotic Meeting (LARS), Sao Bernardo do Campo 120 125 Brazil

10 

Hong D, Kimmel S, Boehling R 2008 Development of a semi-autonomous vehicle by the visually-impaired IEEE International Conference on Multisensot fusion and Integration for Intelligent Systems 539 544 Seoul

11 

Maier D, Bennewitz M, Stachniss C 2011 Self-supervised obstacle detection for humanoid navigation using monocular vision and sparse laser data, IEEE International Conference on Robotics and Automation 1263 1269 Beijing, China

12 

F. Dellaert, D. Pomerlau and C. Thorpe, Model-based car tracking integrated with a road-follower, IEEE International Conference on Robotics and Automation, Leuven, 1998, pp. 1889–1894, On Intelligent Vehicles Symposium, Tokyo, 1996, pp. 391–396.

13 

Arbeiter G, Fuchs S, Hampp J, Bormann R 2014 Efficient segmentation and surface classification of range images IEEE International Conference on Robotics and Automation 5502 5509 Hong Kong, China

14 

Vitor GB, Victorino AC, Ferreira JV 2014 A histogram-based joint boosting classification for determining urban road, IEEE International Conference on Intelligent Transportation Systems (ITSC) 2245 2246 Qingdao, China

15 

Vitor GB, Lima DA, Victorino AC, Ferreira JV 2013 A 2D/3D vision based approach applied to road detection in urban environments IEEE Intelligent Vehicles Symposium (IV) 952 957 Gold Coast City, Australia

16 

Chris U 2009 The urban challenge, International Conference on Information Fusion 1 10 Seattle, Washington, USA

17 

Stanek G, Langer D, Muller-Bessler B, Huhnke B 2010 Junior 3: A test platform for advanced driver assistance systems IEEE Conference on Intelligent Vehicles Symposium 143 149 San Diego

18 

Zhang G, Zheng NN, Cui C, Yan YZ 2009 An efficient road detection method in noisy urban environment IEEE Intelligent Vehicles Symposium 556 561 Xi’an, China

19 

Zhang HH, Wang JL, Fang T, Quan L 2014 Joint segmentation of images and scanned point cloud in large-scale street scenes with low-annotation cost IEEE Transactions on Image Processing 23 4763 4772

20 

Guan HY, Li J, Yu YT, Chapman M, Wang C 2015 Automated road information extraction from mobile laser scanning data IEEE Transactions on Intelligent Transportation Systems 16 194 205

21 

Ulrich I, Nourbakhsh I 2000 Appearance-based obstacle detection with monocular color vision 866 871 Austin, Texas, USA AAAI

22 

Alvarez JM, Lopez AM, Gevers T, Lumbreras F 2014 Combining priors, appearance, and context for road detection IEEE Transactions on Intelligent Transportation Systems 15 1168 1178

23 

Alvarez JM, Salzmann M, Barnes N 2014 Data-driven road detection, IEEE Winter Conference on Applications of Computer Vision (WACV), Steamboat Springs 1134 1141 CO, USA

24 

Alvarez JM, Gevers T, Lopez AM 2009 Vision-based road detection using road models, IEEE International Conference on Image Processing 2073 2076 Cairo, Egypt

25 

Alvarez JM, Gevers T, Diego F, Lopez AM 2013 Road geometry classification by adaptive shape models IEEE Transactions on Intelligent Transportation Systems 14 459 468

26 

Tan J, Li J, An XJ, He HG 2014 Robust curb detection with fusion of 3D-lidar and camera data Sensors 14 9046 9073

27 

Lu K, Li J, An XJ, He HG 2014 A hierarchical approach for road detection IEEE International Conference on Robotics and Automation (ICRA) 517 522 Hong Kong, China

28 

Yeonsik K, Chiwon R, Seung-Beum S, Bongsob S 2012 A lidar-based decision-making method for road boundary detection using multiple Kalman filters IEEE Trans Ind Electron 59 4360 4368

29 

Rose K 1998 Deterministic annealing for clustering, compression, classification, regressions and related optimization problems Proceedings of the IEEE 86 2210 2239

30 

Rose K, Gurewitz E, Fox GC 1990 Statistical mechanics and phase transitions in clustering Phys Rev Lett 65 945 948

31 

Beyeler M, Mirus F, Verl A 2014 Vision-based robust road lane detection in urban environments IEEE International Conference on Robotics and Automation (ICRA) 4920 4925 Hong Kong, China

32 

Enzweiler M, Greiner P, Knoppel C, Franke U 2013 Towards multi-cue urban curb recognition In Proceedings of the IEEE Intelligent Vehicles Symposium 902 907 Gold Coast City, Australia

33 

Maurer M, Dickmanns ED 1997 A system architecture for autonomous visual road vehicle guidance IEEE Conference on Intelligent Transportation System 578 583 Boston

34 

Zhang QL, Pless R 2004 Extrinsic calibration of a camera and laser range finder (improves camera calibration), IEEE International Conference on Intelligent Robots and Systems 2301 2306 Sendai, Japan

35 

Matthaei R, Lichte B, Maurer M 2013 Robust grid-based road detection for ADAS and autonomous vehicles in urban environments, International Conference on Information Fusion (FUSION) 938 944 Istanbul, Turkey

36 

Tedrake R, Fallon M, Karumanchi S 2014 IEEE International Conference on Robotics and Automation A summary of team MIT’s approach to the virtual robotics challenge 2087 Hong Kong

37 

Zhou SY, Gong JW, Xiong GM, Chen HY 2010 Road detection using support vector machine based on-line learning and evaluation IEEE Intelligent Vehicles Symposium (IV) 21 256 261

38 

Zhou SY, Iagnemma K 2010 Self-supervised learning method for unstructured road detection using fuzzy support vector machines, IEEE/RSJ International Conference on Intelligent Robots and Systems 1183 1189 Taipei, Taiwan

39 

Kuhnl T, Fritsch J 2014 Visio-spatial road boundary detection for unmarked urban and rural roads, IEEE Intelligent Vehicles Symposium Proceedings 1251 1256 Dearborn, Michigan, USA

40 

Huang WQ, Gong XJ, Liu JL 2013 Integrating visual and range data for road detection, IEEE International Conference on Image Processing (ICIP) 4136 4140 Melbourne, Australia

41 

Huang WQ, Gong XJ, Xiang ZY 2014 Road scene segmentation via fusing camera and lidar data IEEE International Conference on Robotics anf Automation 1008 1013 Hong Kong, China

42 

Hu X, Rodriguez FSA, Gepperth A 2014 A multi-modal system for road detection and segmentation, IEEE Intelligent Vehicles Symposium Proceedings 1365 1370 Dearborn, Michigan, USA

43 

Liu X, Shang KK, Liu J, Zhou CY 2014 Unstructured road detection based on fuzzy clustering arithmetic, International Conference on Fuzzy Systems and Knowledge Discovery (FSKD) 114 118 Xiamen, China

44 

Yuan X, Zhao CX 2010 Traversable area extraction using LIDAR sensor Acta Armamentarii 31 1702 1707

45 

Zhong XY, Peng XF, Zhou JH 2011 Detection of moving obstacles for mobile robot using laser sensor, The 30th Chinese Control Conference 4002 4006 Yantai, China

46 

He Z, Wu T, Xiao ZP, He HG 2013 Robust road detection from a single image using road shape prior, IEEE International Conference on Image Processing (ICIP) 2757 2761 Melbourne, Australia

47 

Liu Z, Wang J, Liu D 2013 A new curb detection method for unmanned ground vehicles using 2D sequential laser data Sensors 13 1102 1120

Figures and Tables

Fig.1

A small ground mobile robot platform with size of 0.36 m (L)×0.35 m (W)×0.45 m (H).

A small ground mobile robot platform with size of 0.36 m (L)×0.35 m (W)×0.45 m (H).
Fig.2

RGB and HSI color, (a) RGB image, (b) H component, (c) S component, (d) I component.

RGB and HSI color, (a) RGB image, (b) H component, (c) S component, (d) I component.
Fig.3

Road detection result in image: (a) training sample selection, (b) road detection after initial classification, (c) road detection after refine step 1and 2, (d) result after smoothing (c), (e) a demonstration of S boundary mentioned in Section 4.3.

Road detection result in image: (a) training sample selection, (b) road detection after initial classification, (c) road detection after refine step 1and 2, (d) result after smoothing (c), (e) a demonstration of S boundary mentioned in Section 4.3.
Fig.4

Ideal road model.

Ideal road model.
Fig.5

Change of road height between time t and t + 1.

Change of road height between time t and t + 1.
Fig.6

Road detection in laser points, (a) original scenes, (b) points clustering with red points are start or end point of a cluster, (c) results of road detection shown in red line.

Road detection in laser points, (a) original scenes, (b) points clustering with red points are start or end point of a cluster, (c) results of road detection shown in red line.
Fig.7

Registration of a camera and laser range finder Laser points. (a) the coordinate of two sensor, (b) and (c) laser points mapped into image after registration (green pixels).

Registration of a camera and laser range finder Laser points. (a) the coordinate of two sensor, (b) and (c) laser points mapped into image after registration (green pixels).
Fig.8

High confidence road area after mapping laser points into image. The red line is the road detected in laser point projected into image and white area under red line is road samples with high confidence level. The blue line is the outline of the road detected by FSVM.

High confidence road area after mapping laser points into image. The red line is the road detected in laser point projected into image and white area under red line is road samples with high confidence level. The blue line is the outline of the road detected by FSVM.
Fig.9

Flow chart of the proposed algorithm.

Flow chart of the proposed algorithm.
Fig.10

Results of road detection, (a) original scenes and road boundary, (b) results of SVM, (c) results of FSVM, (d) results after refinement based on (c), i.e. result of the proposed algorithm.

Results of road detection, (a) original scenes and road boundary, (b) results of SVM, (c) results of FSVM, (d) results after refinement based on (c), i.e. result of the proposed algorithm.
Fig.11

Results of continuous frames, (a) original scenes, (b) results of road detection, (c) error rate of FSVM, (d) error rate of SVM.

Results of continuous frames, (a) original scenes, (b) results of road detection, (c) error rate of FSVM, (d) error rate of SVM.
Fig.12

Comparison of results between without and with on-line training. From top to down: frame no. 21, 31, 41, 49, 61 and 89. (a) original scenes, (b) results of without on-line training and without refinement, (c) results of without on-line training after refinement, (d) results of with on-line training after refinement.

Comparison of results between without and with on-line training. From top to down: frame no. 21, 31, 41, 49, 61 and 89. (a) original scenes, (b) results of without on-line training and without refinement, (c) results of without on-line training after refinement, (d) results of with on-line training after refinement.
Fig.13

Result of road detection with obstacles on the road, (a) original scenes, (b) result of road detection.

Result of road detection with obstacles on the road, (a) original scenes, (b) result of road detection.
Table 1

Some typical unmanned ground vehicle systems

System nameYearSensorsAdaptive environment
Odin [10]2007Colored camera, lidarOutdoor nature scene
BOSS [16]2008Laser range finder, HDL 64E Lidar, binocular camera systemUrban
Junior 3 [17]2010Laser range finder,Riegl laser, HDL 64E Lidar, Bosch Radae, colored cameraUrban
VRC [36]2013Laser range finder, binocular camera systemComplex environment