Paper Title
Hybrid Anomaly Detection in Financial Transactions using Isolation Forest and Catboost with Explainable AI

Abstract
Banking system’s financial fraud detection is a critical problem because of the extreme proportion of classes, high-dimensionality of transactions, and the inability to interpret the model. In the current paper, it is proposed to use a hybrid anomaly detection model, which will combine Isolation Forest as an unsupervised anomaly scorer and CatBoost as a trained gradient boosting classifier to detect fraudulent banking transactions. The anomaly score produced by Isolation Forest is taken as one more feature to improve the CatBoost classification procedure. Synthetic Minority Oversampling Technique (SMOTE) is used on training data to limit the influence of classes. The method of adaptive threshold optimization is used to maximize the F1-Score above the default probability threshold. The framework proposed is further supplemented with SHAP-based Explainable AI to give actual and understandable predictions to individual transactions. Experimental analysis of a dataset of bank transactions shows that the proposed hybrid model shows an F1-score of 0.8953, Precision of 0.9747, Recall of 0.8278 and ROC-AUC of 0.9887. Mean F1 of 5-Fold Cross-Validation of 0.8945±0.0034 is an indication of robustness and consistency. The suggested algorithm beats all of the baseline models such as Random Forest, Gradient Boosting, AdaBoost, Isolation Forest, and Autoencoder with Isolation Forest. Keywords - Financial Fraud Detection, Anomaly Detection, Isolation Forest, CatBoost, SMOTE, Explainable AI, SHAP, Imbalanced Learning, Hybrid Machine Learning