Pedestrian crossing intention prediction in the wild: A survey AITranslate

[1]J. Fang, F. Wang, J. Xue, and T. -S. Chua, "Behavioral intention prediction in driving scenes: A survey," IEEE Transactions on Intelligent Transportation Systems, vol. 25, no. 8, pp. 8334–8355, 2024.

[2]N. Sharma, C. Dhiman, and S. Indu, "Pedestrian intention prediction for autonomous vehicles: A comprehensive survey," Neurocomputing, vol. 508, pp. 120–152, 2022.

[3]H. Razali, T. Mordan, and A. Alahi, "Pedestrian intention prediction: A convolutional bottom-up multi-task approach," Transportation Research Part C: Emerging Technologies, vol. 130, p. 103259, 2021.

[4]Y. Ling, Z. Ma, B. Xie, Q. Zhang, and X. Weng, "SA-BiGCN: Bi-stream graph convolution networks with spatial attentions for the eye contact detection in the wild," IEEE Transactions on Intelligent Transportation Systems, vol. 25, no. 2, pp. 2089–2100, 2023.

[5]Y. Belkada, L. Bertoni, R. Caristan, T. Mordan, and A. Alahi, "Do pedestrians pay attention? eye contact detection in the wild," arXiv preprint arXiv:2112.04212, 2021.

[6]V. Onkhar, P. Bazilinskyy, J. C. Stapel, D. Dodou, D. Gavrila, and J. C. de Winter, "Towards the detection of driver–pedestrian eye contact," Pervasive and Mobile Computing, vol. 76, p. 101455, 2021.

[7]R. Korbmacher and A. Tordeux, "Review of pedestrian trajectory prediction methods: Comparing deep learning and knowledge-based approaches," IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 12, pp. 24126–24144, 2022.

[8]X. Song, K. Chen, X. Li, J. Sun, B. Hou, Y. Cui, B. Zhang, G. Xiong, and Z. Wang, "Pedestrian trajectory prediction based on deep convolutional LSTM network," IEEE Transactions on Intelligent Transportation Systems, vol. 22, no. 6, pp. 3285–3302, 2020.

[9]M. Golchoubian, M. Ghafurian, K. Dautenhahn, and N. L. Azad, "Pedestrian trajectory prediction in pedestrian-vehicle mixed environments: A systematic review," IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 11, pp. 11544–11567, 2023.

[10]S. Zhang, M. Abdel-Aty, Y. Wu, and O. Zheng, "Pedestrian crossing intention prediction at red-light using pose estimation," IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 3, pp. 2331–2339, 2021.

[11]T. Chen, R. Tian, and Z. Ding, "Visual reasoning using graph convolutional networks for predicting pedestrian crossing intention,"2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), Montreal, BC, Canada, 11–17 October, 2021, pp. 3103–3109.

[12]H. Kim, K. Lee, G. Hwang, and C. Suh, "Crash to not crash: Learn to identify dangerous vehicles using a simulator,"in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 33, no. 01, 2019, pp. 978–985.

[13]J. F. P. Kooij, N. Schneider, F. Flohr, and D. M. Gavrila, "Contextbased pedestrian path prediction,"Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6–12, 2014.

[14]A. Rasouli, I. Kotseruba, and J. K. Tsotsos, "Are they going to cross? A benchmark dataset and baseline for pedestrian crosswalk behavior,"in Proceedings of the IEEE International Conference on Computer Vision Workshops, Venice, Italy, 22–29 October, 2017, pp. 206–213.

[15]A. Rasouli, I. Kotseruba, T. Kunic, and J. K. Tsotsos, "Pie: A large-scale dataset and models for pedestrian intention estimation and trajectory prediction,"in Proceedings of the IEEE/CVF International Conference on Computer Vision, Seoul, Korea (South), 27 October–02 November, 2019, pp. 6262–6271.

[16]B. Liu, E. Adeli, Z. Cao, K.-H. Lee, A. Shenoi, A. Gaidon, and J. C. Niebles, "Spatiotemporal relationship reasoning for pedestrian intent prediction," IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 3485–3492, 2020.

[17]H. Wu, L. Wang, S. Zheng, Q. Xu, and J. Wang, "Crossing-road pedestrian trajectory prediction based on intention and behavior identification,"in 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), Rhodes, Greece, 20–23 September, 2020, pp. 1–6.

[18]T. Chen, T. Jing, R. Tian, Y. Chen, J. Domeyer, H. Toyoda, R. Sherony, and Z. Ding, "PSI: A pedestrian behavior dataset for socially intelligent autonomous car," arXiv preprint arXiv:2112.02604, 2021.

[19]A. Rasouli, T. Yau, P. Lakner, S. Malekmohammadi, M. Rohani, and J. Luo, "PePScenes: A novel dataset and baseline for pedestrian action prediction in 3d," arXiv preprint arXiv:2012.07773, 2020.

[20]B. Völz, K. Behrendt, H. Mielenz, I. Gilitschenski, R. Siegwart, and J. Nieto, "A data-driven approach for pedestrian intention estimation,"2016 IEEE 19th International Conference on Intelligent Transportation Systems (ITSC). Rio de Janeiro, Brazil, 1–4 November, 2016, pp. 2607–2612.

[21]C. Zhang, A. H. Kalantari, Y. Yang, Z. Ni, G. Markkula, N. Merat, and C. Berger, "Cross or wait? predicting pedestrian interaction outcomes at unsignalized crossings,"in 2023 IEEE Intelligent Vehicles Symposium (IV). Rio de Janeiro, Brazil, 01–04 November, 2016, 2023, pp. 1–8.

[22]I. Kotseruba, A. Rasouli, and J. K. Tsotsos, "Benchmark for evaluating pedestrian action prediction,"in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, Waikoloa, HI, USA, 03–08 January, 2021, pp. 1258–1268.

[23]M. Azarmi, M. Rezaei, H. Wang, and S. Glaser, "PIP-Net: Pedestrian intention prediction in the wild," arXiv preprint arXiv:2402.12810, 2024.

[24]J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, "You only look once: Unified, real-time object detection,"in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, Las Vegas, NV, USA, June 27–June 30 2016, pp. 779–788.

[25]W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. -Y. Fu, and A. C. Berg, "SSD: Single shot multibox detector," Computer Vision– ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016.

[26]R. Girshick, "Fast R-CNN,"in Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 07–13 December, 2015, pp. 1440–1448.

[27]D. Yang, H. Zhang, E. Yurtsever, K. A. Redmill, and Ü. Özgüner, "Predicting pedestrian crossing intention with feature fusion and spatiotemporal attention," IEEE Transactions on Intelligent Vehicles, vol. 7, no. 2, pp. 221–230, 2022.

[28]Y. Ling, Z. Ma, Q. Zhang, B. Xie, and X. Weng, "PedAST-GCN: Fast pedestrian crossing intention prediction using spatial–temporal attention graph convolution networks," IEEE Transactions on Intelligent Transportation Systems, vol. 25, no. 10, pp. 13277–13290, 2024.

[29]P. R. G. Cadena, Y. Qian, C. Wang, and M. Yang, "Pedestrian graph+: A fast pedestrian crossing prediction model based on graph convolutional networks," IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 11, pp. 21050–21061, 2022.

[30]Y. Zhou, G. Tan, R. Zhong, Y. Li, and C. Gou, "PIT: Progressive interaction transformer for pedestrian crossing intention prediction," IEEE Transactions on Intelligent Transportation Systems, vol. 24, no. 12, pp. 14213–14225, 2023.

[31]J. Carreira and A. Zisserman, "Quo vadis, action recognition? A new model and the kinetics dataset,"in proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, Honolulu, HI, USA, 21–26 July 2017, pp. 6299–6308.

[32]Z. Cao, T. Simon, S. -E. Wei, and Y. Sheikh, "Realtime multi-person 2d pose estimation using part affinity fields,"in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, Honolulu, HI, USA, 21–26 July, 2017, pp. 7291–7299.

[33]G. Hidalgo, Y. Raaj, H. Idrees, D. Xiang, H. Joo, T. Simon, and Y. Sheikh, "Single-network whole-body pose estimation,"in Proceedings of the IEEE/CVF International Conference on Computer Vision, 27 Oct.–2 Nov., 2019, Seoul, Korea (South), 2019, pp. 6982–6991.

[34]K. Sun, B. Xiao, D. Liu, and J. Wang, "Deep high-resolution representation learning for human pose estimation,"in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019, Seoul, Korea (South), 27 Oct.–2 Nov., 2019, pp. 5693–5703.

[35]S. Kreiss, L. Bertoni, and A. Alahi, "PifPaf: Composite fields for human pose estimation,"in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, Seattle, WA, USA, 13–19 June, 2020, pp. 11977–11986.

[36]A. Bucksch, R. Lindenbergh, and M. Menenti, "SkelTre: Robust skeleton extraction from imperfect point clouds," The Visual Computer, vol. 26, pp. 1283–1300, 2010.

[37]Y. Ling, Q. Zhang, X. Weng, and Z. Ma, "STMA-GCN pedcross: Skeleton based spatial-temporal graph convolution networks with multiple attentions for fast pedestrian crossing intention prediction,"in 2023 IEEE 26th International Conference on Intelligent Transportation Systems (ITSC), Bilbao, Spain, 13 February, 2024, pp. 500–506.

[38]X. Zhang, P. Angeloudis, and Y. Demiris, "ST crossingpose: A spatial-temporal graph convolutional network for skeleton-based pedestrian crossing intention prediction," IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 11, pp. 20773–20782, 2022.

[39]T. Mordan, M. Cord, P. Pérez, and A. Alahi, "Detecting 32 pedestrian attributes for autonomous vehicles," IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 8, pp. 11823–11835, 2021.

[40]Z. Zhang, R. Tian, and Z. Ding, "TrEP: Transformer-based evidential prediction for pedestrian intention with uncertainty,"in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 37, no. 3, pp. 3534–3542, 2023.

[41]X. Guo, S. Li, J. Yu, J. Zhang, J. Ma, L. Ma, W. Liu, and H. Ling, "PFLD: A practical facial landmark detector," arXiv preprint arXiv:1902.10859, 2019.

[42]W. Wu, C. Qian, S. Yang, Q. Wang, Y. Cai, and Q. Zhou, "Look at boundary: A boundary-aware face alignment algorithm,"in Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, June 18–23, 2018, pp. 2129–2138.

[43]H. Jin, S. Liao, and L. Shao, "Pixel-in-pixel net: Towards efficient facial landmark detection in the wild," International Journal of Computer Vision, vol. 129, no. 12, pp. 3174–3194, 2021.

[44]X. Dong, Y. Yan, W. Ouyang, and Y. Yang, "Style aggregated network for facial landmark detection,"in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, Salt Lake City, UT, USA, June 18–23, 2018, pp. 379–388.

[45]E. Ilg, N. Mayer, T. Saikia, M. Keuper, A. Dosovitskiy, and T. Brox, "Flownet 2.0: Evolution of optical flow estimation with deep networks,"in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, Honolulu, HI, USA, 21–26 July 2017, pp. 2462–2470.

[46]L.-C. Chen, G. Papandreou, F. Schroff, and H. Adam, "Rethinking atrous convolution for semantic image segmentation," arXiv preprint arXiv:1706.05587, 2017.

[47]O. Ronneberger, P. Fischer, and T. Brox, "U-Net: Convolutional networks for biomedical image segmentation,"in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5–9, 2015.

[48]T.-Y. Lin, P. Doll´ar, R. Girshick, K. He, B. Hariharan, and S. Belongie, "Feature pyramid networks for object detection,"in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2017, pp. 2117–2125.

[49]W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao, "Pyramid vision transformer: A versatile backbone for dense prediction without convolutions,"in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 568–578.

[50]D. Eigen, C. Puhrsch, and R. Fergus, "Depth map prediction from a single image using a multi-scale deep network," Advances in Neural Information Processing Systems, vol. 3, no. 9, pp. 2366–2374, 2014.

[51]J. Watson, O. Mac Aodha, V. Prisacariu, G. Brostow, and M. Fir-man, "The temporal opportunist: Self-supervised multi-frame monocular depth,"in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 20–25 June, 2021, pp. 1164–1174.

[52]K. Simonyan and A. Zisserman, "Two-stream convolutional networks for action recognition in videos," Advances in Neural Information Processing Systems, vol. 27, pp. 568–576, 2014.

[53]P. R. G. Cadena, M. Yang, Y. Qian, and C. Wang, "Pedestrian graph: Pedestrian crossing prediction based on 2d pose estimation and graph convolutional networks,"in 2019 IEEE Intelligent Transportation Systems Conference (ITSC), Auckland, New Zealand, 27–30 October, 2019, pp. 2000–2005.

[54]F. U. H. A. Mattoo, U. S. Khan, T. Nawaz, and N. Rashid, "Deep learning-based feature fusion for action recognition using skeleton information,"in 2023 International Conference on Robotics and Automation in Industry (ICRAI), Peshawar, Pakistan, 03–05 March, 2023, pp. 1–6.

[55]Y. Ito, Q. Kong, K. Morita, and T. Yoshinaga, "Efficient and accurate skeleton-based two-person interaction recognition using inter-and intrabody graphs,"in 2022 IEEE International Conference on Image Processing (ICIP), Bordeaux, France, 16–19 October, 2022, pp. 231–235.

[56]I. Kotseruba, A. Rasouli, and J. K. Tsotsos, "Do they want to cross? understanding pedestrian intention for behavior prediction,"in 2020 IEEE Intelligent Vehicles Symposium (IV), Las Vegas, NV, USA, 19 October–November, 2020, pp. 1688–1693.

[57]M. Dong, "Pedestrian cross forecasting with hybrid feature fusion,"in Asian Conference on Machine Learning. ACML, 2024, Hanoi, Vietnam, December 5–8, 2024, pp. 327–342.

[58]T. W. Wang and S.-H. Lai, "Pedestrian crossing intention prediction with multi-modal transformer-based model,"in 2023 Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), Taipei, Taiwan, 20 November, 2023, pp. 1349–1356.

[59]X. Shi, Z. Chen, H. Wang, D.-Y. Yeung, W.-K. Wong, and W.-C. Woo, "Convolutional lstm network: A machine learning approach for precipitation nowcasting," NIPS'15: Proceedings of the 28th International Conference on Neural Information Processing Systems, vol. 1, pp. 802–810.

[60]J. Yue-Hei Ng, M. Hausknecht, S. Vijayanarasimhan, O. Vinyals, R. Monga, and G. Toderici, "Beyond short snippets: Deep networks for video classification,"in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, arXiv preprint arXiv:1503.08909, 2015.

[61]Y. Du, W. Wang, and L. Wang, "Hierarchical recurrent neural network for skeleton based action recognition,"in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Boston, MA, USA, 07–12 June, 2015, pp. 1110– 1118.

[62]D. Tran, L. Bourdev, R. Fergus, L. Torresani, and M. Paluri, "Learning spatiotemporal features with 3d convolutional networks,"in Proceedings of the IEEE International Conference on Computer Vision, Santiago, Chile, 18 February, 2016, pp. 2380–7504.

[63]A. Bhattacharyya, M. Fritz, and B. Schiele, "Long-term on-board prediction of people in traffic scenes under uncertainty,"in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Salt Lake City, UT, USA, 18–23 June, 2018, pp. 4194–4202.

[64]A. Rasouli, I. Kotseruba, and J. K. Tsotsos, "Pedestrian action anticipation using contextual feature fusion in stacked RNNs," arXiv preprint arXiv:2005.06582, 2020.

[65]F. Piccoli, R. Balakrishnan, M. J. Perez, M. Sachdeo, C. Nunez, M. Tang, K. Andreasson, K. Bjurek, R. D. Raj, E. Davidsson, et al., "Fussi-Net: Fusion of spatio-temporal skeletons for intention prediction network,"in 2020 54th Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, CA, USA, 01–04, November, 2020, pp. 68–72.

[66]J. Gesnouin, S. Pechberti, B. Stanciulcscu, and F. Moutarde, "Trouspinet: Spatio-temporal attention on parallel atrous convolutions and ugrus for skeletal pedestrian crossing prediction,"in 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), Jodhpur, India, 15–18 December, 2021.

[67]B. Yang, Z. Wei, H. Hu, R. Wang, C. Yang, and R. Ni, "Dpcian: A novel dual-channel pedestrian crossing intention anticipation network," IEEE Transactions on Intelligent Transportation Systems, vol. 25, no. 6, pp. 6023–6034, 2023.

[69]X. Chen, S. Zhang, J. Li, and J. Yang, "Pedestrian crossing intention prediction based on cross-modal transformer and uncertainty-aware multi-task learning for autonomous driving," IEEE Transactions on Intelligent Transportation Systems, vol. 25, no. 9, 12538–12549, 2024.

[70]N. Lv, Y. Huang, H. Zhang, and F. Wu, "Pedestrian crossing prediction with pathwise feature fusion and stacked gate recurrent unit," IEEE Sensors Letters, vol. 8, no. 2, pp. 2475–1472, 2024.

[71]K. Simonyan and A. Zisserman, "Very deep convolutional networks for large-scale image recognition," arXiv preprint arXiv:1409.1556, 2014.

[72]K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition,"in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Las Vegas, NV, USA, 12 December, 2016, pp. 770–778.

[73]A. Graves and A. Graves, "Long short-term memory," Supervised Sequence Labelling with Recurrent Neural Networks, Springer, 2012, pp. 37–45.

[74]J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, "Empirical evaluation of gated recurrent neural networks on sequence modeling," arXiv preprint arXiv:1412.3555, 2014.

[75]T. N. Kipf and M. Welling, "Semi-supervised classification with graph convolutional networks," arXiv preprint arXiv:1609.02907, 2016.

[76]X. Wei, S. Yao, C. Zhao, D. Hu, H. Luo, and Y. Lu, "Graph convolutional networks (GCN)-based lightweight detection model for dangerous driving behavior,"in International Conference on Wireless Algorithms, Systems, and Applications. Springer, 2022, pp. 27–39.

[77]T. Sofianos, A. Sampieri, L. Franco, and F. Galasso, "Space-time separable graph convolutional network for pose forecasting,"in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, Montreal, QC, Canada, Oct. 10–Oct. 17 2021, pp. 11209–11218.

[78]A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, "Attention is all you need," arXiv preprint arXiv:1706.03762, 2017.

[79]S. Dai, J. Liu and N.-M. Cheung, "Uncertainty-aware pedestrian crossing prediction via reinforcement learning," IEEE Transactions on Circuits and Systems for Video Technology, vol. 34, no. 10, pp. 9540–9549, 2024.

[80]J. Gesnouin, S. Pechberti, B. Stanciulescu, and F. Moutarde, "Assessing cross-dataset generalization of pedestrian crossing predictors,"in 2022 IEEE Intelligent Vehicles Symposium (IV), Aachen, Germany, 04–09 June, 2022, pp. 419–426.

[81]J. Gawlikowski, C. R. N. Tassi, M. Ali, J. Lee, M. Humt, J. Feng, A. Kruspe, R. Triebel, P. Jung, R. Roscher, et al., "A survey of uncertainty in deep neural networks," Artificial Intelligence Review, vol. 56, no. Suppl. 1, pp. 1513–1589, 2023.

[82]K. Terzidou, I. Krontiris, K. Grammenou, M. Zacharopoulou, M. Tsikintikou, F. Baladima, C. Sakellari, and K. Kaouras, "Autonomous vehicles: Data protection and ethical considerations," Journal of the Association for Computing Machinery, 2020.

[83]I. Krontiris, K. Grammenou, K. Terzidou, M. Zacharopoulou, M. Tsikintikou, F. Baladima, C. Sakellari, and K. Kaouras, "Autonomous vehicles: Data protection and ethical considerations,"in Proceedings of the 4th ACM Computer Science in Cars Symposium, Feldkirchen Germany, 2 December, 2020, pp. 1–10.

Basic Information:

DOI：10.23919/CHAIN.2024.000008

Chinese Library Classification Number:

Citation Information:

In real-world driving scenarios, understanding the intentions of pedestrians in real-time is critical for the built environment safety when operating intelligent vehicles on the roads. Pedestrians crossing the street is a common behavior that can easily lead to accidents. This paper presents a comprehensive review of the prediction of pedestrian crossing intentions, focusing on data, model structure, data representation, information extraction, prediction function, and associated models and challenges. The review highlights that data types, model generalization ability, and prediction uncertainty are key challenges on pedestrian crossing intention prediction. It identifies open challenges and opportunities for future research in pedestrian crossing intention prediction.