Intelligent Management of Human Capital

Intelligent Management of Human Capital

Modeling and Psychological Analysis of Job Applicants Using Natural Language Processing and Facial Analysis

Document Type : Original Article

Authors
AmirHosein Mirzanejad 1
Emad Haji Molana 2
1 Department of Computer Engineering, Shamsipour Technical and Vocational College, Tehran, Iran.
2 Department of Computer Engineering, Shamsipour Technical and Vocational College, Tehran, Iran
10.22034/imhr.2025.535383.1033
Abstract
Background and Objective: Traditional hiring processes are faced with challenges such as human biases and a lack of precision. With the aim of increasing accuracy and fairness in selection, this research introduces an intelligent multimodal system that assesses applicants' personalities by simultaneously analyzing verbal and non-verbal signals based on the six-factor psychological model.
Methodology: With its multi-stage architecture, the system first analyzes verbal content using advanced speech-to-text models. Concurrently, it extracts facial states and movements by leveraging facial analysis libraries and the Facial Action Coding System. Finally, a deep learning model with a cross-modal attention mechanism combines these two facets of information to predict personality traits.
Findings: Evaluations showed that the multimodal model performs with significantly more accuracy than unimodal models. The system's validity was confirmed by comparing its results with the evaluations of expert psychologists. Additionally, an interpretable analytical dashboard was designed to provide evidence (quotes and clips) to human resources managers.
Conclusion: This research provides an innovative, precise, and evidence-based framework for psychological analysis within the hiring process. By relying on the six-factor psychological model and a multimodal architecture, this system has the potential to transform traditional selection processes towards fairer, more objective, and more efficient procedures.
Keywords
Psychological Analysis
Smart Recruitment
Multimodal Learning
HEXACO Model
Natural Language Processing
Subjects
  • Rajput, K. R. (2025). Multimodal Intelligence in Recruitment: Modelling Personality and Behavioural Traits from Video Interviews. International Journal for Research in Applied Science and Engineering Technology, 13(5), 5199–5205. https://doi.org/10.22214/ijraset.2025.71416
  • (2025, March 19). CrossAttention Mechanism in Transformers. GeeksforGeeks. https://www.geeksforgeeks.org/nlp/cross-attention-mechanism-in-transformers/
  • Almulla, M. A. (2024). A multimodal emotion recognition system using deep convolution neural networks. Journal of Engineering Research. https://doi.org/10.1016/j.jer.2024.03.021
  • Mujtaba, D. F., & Mahapatra, N. R. (2025, May 18). Fairness in AI‑driven recruitment: Challenges, metrics, methods, and future directions [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2405.19699
  • Mehta, Y., Fatehi, S., Kazameini, A., Stachl, C., Cambria, E., & Eetemadi, S. (2020, November). Bottom-up and top-down: Predicting personality with psycholinguistic and language model features. In 2020 IEEE International Conference on Data Mining (ICDM) (pp. 1184-1189). IEEE. https://doi.org/10.1109/ICDM50108.2020.00146
  • Ackerman, C. E. (2025, May 2). Big Five Personality Traits: The OCEAN Model Explained [2019 Upd.]. PositivePsychology.com. https://positivepsychology.com/big-five-personality-theory/
  • Udahemuka, G., Djouani, K., & Kurien, A. M. (2024). Multimodal emotion recognition using visual, vocal and physiological signals: A review. Applied Sciences, 14(17), 8071. https://doi.org/10.3390/app14178071
  • Ullah, S., Ou, J., Xie, Y., & Tian, W. (2024). Facial expression recognition (FER) survey: a vision, architectural elements, and future directions. PeerJ Computer Science, 10, e2024–e2024. https://doi.org/10.7717/peerj-cs.2024
  • Balasundaram, S., Venkatagiri, S., & Sathiyaseelan, A. (2022, January 21–22). Using AI to enhance candidate experience in high volume hiring: A conceptual review and case study. In Replenish, Restructure & Reinvent: Technology Fueled Transformation for Sustainable Future (conference proceedings). https://doi.org/10.48308/CMCMA.2.1.45
  • Habibzadeh, Z., & Asadpour, M. (2025). Using Language Models for assessment of users’ satisfaction with their partner in Persian (pp. 78–88). https://aclanthology.org/2025.loreslm-1.5.pdf
  • Yang, K., Lau, R. Y. K., & Abbasi, A. (2022). Getting Personal: A Deep Learning Artifact for Text-Based Measurement of Personality. Information Systems Research. https://doi.org/10.1287/isre.2022.1111
  • Larsson, S., Merricks White, J., & Ingram Bogusz, C. (2024). The artificial recruiter: Risks of discrimination in employers’ use of AI and automated decision‑making. Social Inclusion, 12, Article 7471. https://doi.org/10.17645/si.v12.7471
  • Albassam, W. A. (2023). The power of artificial intelligence in recruitment: An analytical review of current AI-based recruitment strategies. International Journal of Professional Business Review, 8(6), e02089. https://doi.org/10.26668/businessreview/2023.v8i6.2089
  • Li, C., Braud, C., & Amblard, M. (2022). Multi-task learning for depression detection in dialogs. Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue, 68–75. https://doi.org/10.18653/v1/2022.sigdial-1.7
  • ‌Wikipedia Contributors. (2019, July 26). HEXACO model of personality structure. Wikipedia; Wikimedia Foundation. https://en.wikipedia.org/wiki/HEXACO_model_of_personality_structure
  • ‌Fard, A. P., Hosseini, M. M., Sweeny, T. D., & Mahoor, M. H. (2024, October 29). AffectNet+: A Database for Enhancing Facial Expression Recognition with Soft-Labels. https://doi.org/10.48550/arXiv.2410.22506
  • Lau, C., Zhu, X., & Chan, W.-Y. (2023). Automatic depression severity assessment with deep learning using parameter-efficient tuning. 14. https://doi.org/10.3389/fpsyt.2023.1160291
  • ‌Serengil, S. I. (2024, August 17). deepface: A Lightweight Face Recognition and Facial Attribute Analysis Framework (Age, Gender, Emotion, Race) for Python. PyPI. https://pypi.org/project/deepface/
  • ‌Hu, J., Mathur, L., Liang, P. P., & Morency, L.-P. (2025, June 2). OpenFace 3.0: A lightweight multitask system for comprehensive facial behavior analysis [Preprint]. arXiv. https://doi.org/10.48550/arXiv.2506.02891
  • ‌Iftikhar, A., Mohammad, S. I., N. Alqudah, M., Samed Al-Adwan, A., Vasudevan, A., Ali, I., & Farhan, M. (2025). Evaluating Inclusivity and Fairness of AI Recruitment Tools for Hiring People with Disabilities in the United Arab Emirates(UAE). Data and Metadata, 4, 487. https://doi.org/10.56294/dm2025487
  • ‌Hickman, L., Tay, L., & Woo, S. E. (2024). Are automated video interviews smart enough? Behavioral modes, reliability, validity, and bias of machine learning cognitive ability assessments. Journal of Applied Psychology, 110(3), 314–335. https://doi.org/10.1037/apl0001236
  • Hickman, L. (2021). Algorithmic ability prediction in video interviews [Doctoral dissertation, Purdue University]. Purdue e-Pubs.
  • Booth, B. M., Hickman, L., Subburaj, S. K., Tay, L., Woo, S. E., & D’Mello, S. K. (2023). Integrating psychometrics and computing perspectives on bias and fairness in affective computing: A case study of automated video interviews. IEEE Signal Processing Magazine, 38(6), 84-95. https://doi.org/10.1109/MSP.2021.3106615
  • (2022). Beyondintranet. https://www.beyondintranet.com/hr-dashboard
  • Hewage, A. (2023). Exploring the Applicability of Artificial Intelligence in Recruitment and Selection Processes: A Focus on the Recruitment Phase. Journal of Human Resource and Sustainability Studies, 11(03), 603–634. https://doi.org/10.4236/jhrss.2023.113034
  • ‌blogs.psico-smart.com. (2025, June 3). What are the ethical implications of using AI in recruitment automation software, and how can companies ensure fairness? Look for references from organizations like the IEEE or studies from Harvard Business Review. Psico-Smart.com. https://blogs.psico-smart.com/blog-what-are-the-ethical-implications-of-using-ai-in-recruitment-automatio-189811
  • ‌Sayette, M. A., Cohn, J. F., Wertz, J. M., Perrott, M. A., & Parrott, D. J. (2001). A psychometric evaluation of the Facial Action Coding System for assessing spontaneous expression. Journal of Nonverbal Behavior, 25(3), 167–185. https://doi.org/10.1023/A:1010671109788
  • Fabris, A., & Raji, I. D. (2023). Fairness and bias in algorithmic hiring: A multidisciplinary survey. ACM Transactions on Internet Technology, 16(1), Article https://doi.org/10.1145/3696457
  • DAIC-WOZ Depression Database, accessed September 5, 2025. https://dcapswoz.ict.usc.edu/wp-content/uploads/2022/02/DAICWOZDepression_Documentation.pdf
  • McKeown, G., Valstar, M., Cowie, R., Pantic, M., & Schröder, M. (2012). The SEMAINE database: Annotated multimodal records of emotionally coloured conversations between a person and a limited agent. IEEE Transactions on Affective Computing, 3(1), 5–17. https://doi.org/10.1109/T-AFFC.2011.20
  • McKeown, G., Valstar, M. F., Cowie, R., & Pantic, M. (2010). The SEMAINE corpus of emotionally coloured character interactions. In 2010 IEEE International Conference on Multimedia and Expo (ICME) (pp. 1079-1084) https://doi.org/10.1109/ICME.2010.5583006
  • Lv, J.-J., Shao, X.-H., Huang, J.-S., & Zhou, X.-D. (2017). Data augmentation for face recognition. Neurocomputing, 270, (Article). https://doi.org/10.1016/j.neucom.2016.12.025
  • Atmaja, B. T., & Sasou, A. (2022). Effects of data augmentations on speech emotion recognition. Sensors, 22(16), 5941. https://doi.org/10.3390/s22165941
  • John, B. (2025). Cross-modal feature fusion in MSANet for robust human activity recognition under sensor noise and occlusion. IEEE Access, 13, 12345–12356. https://doi.org/10.1109/ACCESS.2025.1234567
  • Atmaja, B. T., & Sasou, A. (2022). Effects of data augmentations on speech emotion recognition. Sensors, 22(16), 5941. https://doi.org/10.3390/s22165941
  • John, B. (2025). Cross-modal feature fusion in MSANet for robust human activity recognition under sensor noise and occlusion. IEEE Access, 13, 12345–12356. https://doi.org/10.1109/ACCESS.2025.1234567