Date Log
This work is licensed under a Creative Commons Attribution 4.0 International License.
Satellite Image Fusion Using a Hybrid Traditional and Deep Learning Method
Corresponding Author(s) : Mahmoud M. Hammad
Geomatics and Environmental Engineering,
Vol. 17 No. 5 (2023): Geomatics and Environmental Engineering
Abstract
Due to growing demand for ground-truth in deep learning-based remote sensing satellite image fusion, numerous approaches have been presented. Of these approaches, Wald’s protocol is the most commonly used. In this paper, a new workflow is proposed consisting of two main parts. The first part targets obtaining the ground-truth images using the results of a pre-designed and well-tested hybrid traditional fusion method. This method combines the Gram–Schmidt and curvelet transform techniques to generate accurate and reliable fusion results. The second part focuses on the training of a proposed deep learning model using rich and informative data provided by the first stage to improve the fusion performance. The demonstrated deep learning model relies on a series of residual dense blocks to enhance network depth and facilitate the effective feature learning process. These blocks are designed to capture both low-level and high-level information, enabling the model to extract intricate details and meaningful features from the input data. The performance evaluation of the proposed model is carried out using seven metrics such as peak-signal-to-noise-ratio and quality without reference. The experimental results demonstrate that the proposed approach outperforms state-of-the-art methods in terms of image quality. It also exhibits the robustness and powerful nature of the proposed approach which has the potential to be applied to many remote sensing applications in agriculture, environmental monitoring, and change detection.
Keywords
Download Citation
Endnote/Zotero/Mendeley (RIS)BibTeX
- Xiao G., Bavirisetti D. P., Liu G., Zhang X.: Image Fusion. Springer, Singapore 2020.
- Kaur H., Koundal D., Kadyan V.: Image fusion techniques: A survey. Archives of Computational Methods in Engineering, vol. 28, 2021, pp. 4425–4447. https://doi.org/10.1007/s11831-021-09540-7.
- Tsagkatakis G., Aidini A., Fotiadou K., Giannopoulos M., Pentari A., Tsakalides P.: Survey of deep-learning approaches for remote sensing observation enhancement. Sensors, vol. 19(18), 2019, 3929. https://doi.org/10.3390/s19183929.
- Dong C., Loy C. C., He K., Tang X.: Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38(2), 2016, pp. 295–307. https://doi.org/10.1109/TPAMI.2015.2439281.
- Masi G., Cozzolino D., Verdoliva L., Scarpa G.: Pansharpening by convolutional neural networks. Remote Sensing, vol. 8(7), 2016, 594. https://doi.org/10.3390/rs8070594.
- Zhong J., Yang B., Huang G., Zhong F., Chen Z.: Remote sensing image fusion with convolutional neural network. Sensing and Imaging, vol. 17, 2016, 10. https://doi.org/10.1007/s11220-016-0135-6.
- Yang J., Fu X., Hu Y., Huang Y., Ding X., Paisley J.: PanNet: A Deep Network Architecture for Pan-Sharpening. [in:] 2017 IEEE International Conference on Computer Vision ICCV 2017: Proceedings: 22–29 October 2017, Venice, Italy 2017, IEEE, Piscataway 2017, pp. 1753–1761. https://doi.org/10.1109/ICCV.2017.193.
- Nguyen H.V., Ulfarsson M.O., Sveinsson J.R., Mura M.D.: Deep SURE for unsupervised remote sensing image fusion. IEEE Transactions on Geoscience and Remote Sensing, vol. 60, 2022, pp. 1–13. https://doi.org/10.1109/TGRS.2022.3215902.
- Wei Y., Yuan Q.: Deep residual learning for remote sensed imagery pansharpening. [in:] RSIP 2017: International Workshop on Remote Sensing with Intelligent Processing: Proceedings: May 19–21, Shanghai, China, IEEE, Piscataway 2017, pp. 1–4. https://doi.org/10.1109/RSIP.2017.7958794.
- Shao Z., Cai J.: Remote sensing image fusion with deep convolutional neural network. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 11(5), 2018, pp. 1656–1669. https://doi.org/10.1109/JSTARS.2018.2805923.
- Liu X., Liu Q., Wang Y.: Remote sensing image fusion based on two-stream fusion network. Information Fusion, vol. 55, 2020, pp. 1–15. https://doi.org/10.1016/j.inffus.2019.07.010.
- Wang J., Shao Z., Huang X., Lu T., Zhang R.: A dual-path fusion network for pan-sharpening. IEEE Transactions on Geoscience and Remote Sensing, vol. 60, 2022, pp. 1–14. https://doi.org/10.1109/TGRS.2021.3090585.
- Guo A., Dian R., Li S.: Unsupervised blur kernel learning for pansharpening. [in:] IGARSS 2020 – 2020 IEEE: International Geoscience and Remote Sensing Symposium: International Geoscience and Remote Sensing Symposium: September 26 – October 2, 2020: Virtual Symposium, IEEE, Piscataway 2020, pp. 633–636. https://doi.org/10.1109/IGARSS39084.2020.9324543.
- Luo S., Zhou S., Feng Y., Xie J.: Pansharpening via unsupervised convolutional neural networks. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 13, 2020, pp. 4295–4310. https://doi.org/10.1109/JSTARS.2020.3008047.
- Benzenati T., Kessentini Y., Kallel A.: Pansharpening approach via two-stream detail injection based on relativistic generative adversarial networks. Expert Systems with Applications, vol. 188, 2022, 115996. https://doi.org/10.1016/j.eswa.2021.115996.
- Ye F., Li X., Zhang X.: FusionCNN: a remote sensing image fusion algorithm based on deep convolutional neural networks. Multimedia Tools and Applications, vol. 78, 2019, pp. 14683–14703. https://doi.org/10.1007/s11042-018-6850-3.
- Hammad M., Ghoniemy T., Mahmoud T., Amein A.: Hybrid fusion using Gram Schmidt and Curvelet transforms for satellite images. IOP Conference Series: Materials Science and Engineering, vol. 1172, 2021, 012016. https://doi.org/10.1088/1757-899X/1172/1/012016.
- Wang X., Yu K., Wu S., Gu J., Liu Y., Dong C., Qiao Y. et al.: Esrgan: Enhanced super-resolution generative adversarial networks. [in:] Leal-Taixé L., Roth S. (eds.), Computer Vision – ECCV 2018 Workshops: Munich, Germany, September 8–14, 2018: Proceedings, Part V, Lecture Notes in Computer Science, vol. 11133, Springer, Cham 2019, pp. 63–79. https://doi.org/10.1007/978-3-030-11021-5_5.
- Vivone G., Alparone L., Chanussot J., Dalla Mura M., Garzelli A., Licciardi G.A., Restaino R. et al.: A critical comparison among pansharpening algorithms. IEEE Transactions on Geoscience and Remote Sensing, vol. 53(5), 2014, pp. 2565–2586. https://doi.org/10.1109/TGRS.2014.2361734.
References
Xiao G., Bavirisetti D. P., Liu G., Zhang X.: Image Fusion. Springer, Singapore 2020.
Kaur H., Koundal D., Kadyan V.: Image fusion techniques: A survey. Archives of Computational Methods in Engineering, vol. 28, 2021, pp. 4425–4447. https://doi.org/10.1007/s11831-021-09540-7.
Tsagkatakis G., Aidini A., Fotiadou K., Giannopoulos M., Pentari A., Tsakalides P.: Survey of deep-learning approaches for remote sensing observation enhancement. Sensors, vol. 19(18), 2019, 3929. https://doi.org/10.3390/s19183929.
Dong C., Loy C. C., He K., Tang X.: Image super-resolution using deep convolutional networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 38(2), 2016, pp. 295–307. https://doi.org/10.1109/TPAMI.2015.2439281.
Masi G., Cozzolino D., Verdoliva L., Scarpa G.: Pansharpening by convolutional neural networks. Remote Sensing, vol. 8(7), 2016, 594. https://doi.org/10.3390/rs8070594.
Zhong J., Yang B., Huang G., Zhong F., Chen Z.: Remote sensing image fusion with convolutional neural network. Sensing and Imaging, vol. 17, 2016, 10. https://doi.org/10.1007/s11220-016-0135-6.
Yang J., Fu X., Hu Y., Huang Y., Ding X., Paisley J.: PanNet: A Deep Network Architecture for Pan-Sharpening. [in:] 2017 IEEE International Conference on Computer Vision ICCV 2017: Proceedings: 22–29 October 2017, Venice, Italy 2017, IEEE, Piscataway 2017, pp. 1753–1761. https://doi.org/10.1109/ICCV.2017.193.
Nguyen H.V., Ulfarsson M.O., Sveinsson J.R., Mura M.D.: Deep SURE for unsupervised remote sensing image fusion. IEEE Transactions on Geoscience and Remote Sensing, vol. 60, 2022, pp. 1–13. https://doi.org/10.1109/TGRS.2022.3215902.
Wei Y., Yuan Q.: Deep residual learning for remote sensed imagery pansharpening. [in:] RSIP 2017: International Workshop on Remote Sensing with Intelligent Processing: Proceedings: May 19–21, Shanghai, China, IEEE, Piscataway 2017, pp. 1–4. https://doi.org/10.1109/RSIP.2017.7958794.
Shao Z., Cai J.: Remote sensing image fusion with deep convolutional neural network. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 11(5), 2018, pp. 1656–1669. https://doi.org/10.1109/JSTARS.2018.2805923.
Liu X., Liu Q., Wang Y.: Remote sensing image fusion based on two-stream fusion network. Information Fusion, vol. 55, 2020, pp. 1–15. https://doi.org/10.1016/j.inffus.2019.07.010.
Wang J., Shao Z., Huang X., Lu T., Zhang R.: A dual-path fusion network for pan-sharpening. IEEE Transactions on Geoscience and Remote Sensing, vol. 60, 2022, pp. 1–14. https://doi.org/10.1109/TGRS.2021.3090585.
Guo A., Dian R., Li S.: Unsupervised blur kernel learning for pansharpening. [in:] IGARSS 2020 – 2020 IEEE: International Geoscience and Remote Sensing Symposium: International Geoscience and Remote Sensing Symposium: September 26 – October 2, 2020: Virtual Symposium, IEEE, Piscataway 2020, pp. 633–636. https://doi.org/10.1109/IGARSS39084.2020.9324543.
Luo S., Zhou S., Feng Y., Xie J.: Pansharpening via unsupervised convolutional neural networks. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 13, 2020, pp. 4295–4310. https://doi.org/10.1109/JSTARS.2020.3008047.
Benzenati T., Kessentini Y., Kallel A.: Pansharpening approach via two-stream detail injection based on relativistic generative adversarial networks. Expert Systems with Applications, vol. 188, 2022, 115996. https://doi.org/10.1016/j.eswa.2021.115996.
Ye F., Li X., Zhang X.: FusionCNN: a remote sensing image fusion algorithm based on deep convolutional neural networks. Multimedia Tools and Applications, vol. 78, 2019, pp. 14683–14703. https://doi.org/10.1007/s11042-018-6850-3.
Hammad M., Ghoniemy T., Mahmoud T., Amein A.: Hybrid fusion using Gram Schmidt and Curvelet transforms for satellite images. IOP Conference Series: Materials Science and Engineering, vol. 1172, 2021, 012016. https://doi.org/10.1088/1757-899X/1172/1/012016.
Wang X., Yu K., Wu S., Gu J., Liu Y., Dong C., Qiao Y. et al.: Esrgan: Enhanced super-resolution generative adversarial networks. [in:] Leal-Taixé L., Roth S. (eds.), Computer Vision – ECCV 2018 Workshops: Munich, Germany, September 8–14, 2018: Proceedings, Part V, Lecture Notes in Computer Science, vol. 11133, Springer, Cham 2019, pp. 63–79. https://doi.org/10.1007/978-3-030-11021-5_5.
Vivone G., Alparone L., Chanussot J., Dalla Mura M., Garzelli A., Licciardi G.A., Restaino R. et al.: A critical comparison among pansharpening algorithms. IEEE Transactions on Geoscience and Remote Sensing, vol. 53(5), 2014, pp. 2565–2586. https://doi.org/10.1109/TGRS.2014.2361734.