fAIr is an open AI-assisted mapping service developed by the Humanitarian OpenStreetMap Team (HOT), with the aim of improving and assisting mapping for humanitarian aid and disaster relief. The proposal illustrates the research undertaken to assess the performance of fAIR underlining ML model training, from the training datasets selection process, the choice of the metrics used to measure accuracy, and finally the analysis of the results obtained testing for different metrics. The research falls within the broader spectrum of research on understanding the fine tuning process for geographic domain adaptation in image analysis validation, particularly for building footprints detection.