VisTA-SR: Improving the Accuracy and Resolution of Low-Cost Thermal

Imaging Cameras for Agriculture

Heesup Yun, Sassoum Lo, Christine H. Diepenbrock, Brian N. Bailey, J. Mason Earles
University of California, Davis
1 Shields Ave, Davis, CA 95616
arXiv:2405.19413v1 [cs.CV] 29 May 2024

{hspyun, ssslo, chdiepenbrock, bnbailey, jmearles}@ucdavis.edu

Abstract ature compared to other temperature measurement devices.

These benefits can help identify areas where crops are expe-
Thermal cameras are an important tool for agricul- riencing disease or stress, allowing for timely intervention.
tural research because they allow for non-invasive mea- Previous studies using thermal cameras in agriculture
surement of plant temperature, which relates to important have utilized high-resolution industrial-grade thermal cam-
photochemical, hydraulic, and agronomic traits. Utilizing eras [16]. However, these cameras are very expensive, often
low-cost thermal cameras can lower the barrier to intro- costing over $10,000 which limits their accessibility. This
ducing thermal imaging in agricultural research and pro- low accessibility can restrict the widespread deployment of
duction. This paper presents an approach to improve the thermal cameras in agriculture, especially for researchers
temperature accuracy and image quality of low-cost ther- who cannot afford costly sensors.
mal imaging cameras for agricultural applications. Lever- An alternative approach is to use low-cost sensors.
aging advancements in computer vision techniques, par- Recent developments in thermal image sensors and im-
ticularly deep learning networks, we propose a method, age processing technologies have made various affordable
called VisTA-SR (Visual & Thermal Alignment and Super- consumer-grade thermal cameras available. These thermal
Resolution Enhancement) that combines RGB and thermal cameras have the advantages of being relatively lightweight
images to enhance the capabilities of low-resolution ther- and easy to operate. Therefore, there have been attempts to
mal cameras. The research includes calibration and vali- use low-cost thermal cameras in agriculture [5, 14, 15]. For
dation of temperature measurements, acquisition of paired example, Bhandari [4] obtained an image mask from visi-
image datasets, and the development of a deep learning net- ble light images and applied it to thermal images to mea-
work tailored for agricultural thermal imaging. Our study sure wheat canopy temperature and estimate water stress.
addresses the challenges of image enhancement in the agri- Another study used a low-cost thermal camera to calcu-
cultural domain and explores the potential of low-cost ther- late crop canopy temperature automatically [15]. However,
mal cameras to replace high-resolution industrial cameras. these low-cost thermal cameras have not been able to com-
Experimental results demonstrate the effectiveness of our pletely replace high-resolution industrial thermal cameras
approach in enhancing temperature accuracy and image due to their lower pixel count and resolution.
sharpness, paving the way for more accessible and efficient Thermal camera resolution has a significant impact on
thermal imaging solutions in agriculture. the capability and accuracy of agricultural research. For ex-
ample, low-resolution thermal cameras may only be able to
recognize crops at the plant-level rather than organ-level,
1. Introduction making it challenging to observe temperature differences
between leaves, stems, flowers, and fruits, for instance. This
Agricultural research often uses crop temperature mea- limited feature resolution will limit the temperature mea-
surement to detect abnormal plant characteristics, calculate surement capability at various phenological stages, which
crop water stress indices, or model complex biophysical in- is essential for developing precise crop biophysical models.
teractions. Since various methods have been attempted to Therefore, improving the quality of low-cost thermal im-
measure crop temperature, thermal imaging cameras are ages can increase the feasibility of using low-cost thermal
widely used because they can quickly measure the temper- cameras in agriculture.
ature at many points in the image [20]. Also, thermal imag- Enhancing the resolution of low-resolution thermal im-
ing can quickly and non-invasively measure crop temper- ages is a challenging task. It is an ill-posed problem, as
 RGB Visual & Thermal Alignment and Super Resolution (VisTA-SR)
Stage 1: Domain Transfer and Alignment
CycleGAN

RGB2IR IR2RGB
Output
RGB 𝐺! (𝐼) 𝐺" (𝐼) RGB’

Concat HR Target

Aligned
Thermal Template Matching RGB SR Discriminator
Fake or Real
SR Output [0…1]
Aligned 𝐷(𝐼)
RGB2IR
ResNet Generator
Resize
LR IR x4 SR IR
𝐺!" (𝐼)

Stage 2: Super Resolution Combining RGB and Thermal

Figure 1. Structure of the proposed VisTA-SR network. The network has two main stages: the Image Alignment and the Super-Resolution
Network. The Image Alignment aligns the RGB and thermal images, while the Super-Resolution Network enhances the resolution of the
thermal image.

multiple high-resolution ground truths can exist for a sin- 2. Related Work
gle low-resolution image. Nevertheless, various computer
vision and machine learning techniques have been proposed 2.1. Traditional Image Enhancement
to overcome the challenge. Particularly with the recent ad- Before the advent of deep learning-based image sharp-
vancements in deep learning, there have been many re- ening approaches, filter-based techniques were used to en-
ported cases of upsampling low-resolution images to high- hance image quality, including fundamental Gaussian ker-
resolution. Some researchers have used ResNet and GANs nels and image sharpening kernels such as Bilinear filter-
to perform image super-resolution [21]. Others have com- ing [28], Bilateral filtering [30], and Lanczos filtering [11].
bined multiple low-resolution images to create a single Despite their ability to reduce image noise and enhance ob-
high-resolution image [26]. Some have also used multi- ject edges, these approaches have been criticized for intro-
modal data to improve the resolution of the data [2, 8, 17]. ducing artificial noises not present in the original image or
However, research on improving the quality of thermal im- producing unsatisfactory sharpness.
ages in the agricultural domain has been limited. Applying
these techniques to agricultural thermal images could po- 2.2. Deep Learning Based Super Resolution
tentially improve the image quality of low-resolution ther-
mal cameras, allowing them to replace high-resolution ther- Recently, there have been attempts to improve sharp-
mal cameras. ness using deep learning. These attempts include making
Therefore, this paper studies how computer vision tech- super-resolution images from low-resolution images using
niques can improve the image quality of low-resolution ResNet and GANs, resulting in various developed meth-
thermal cameras for agricultural applications. We propose ods [9, 21, 31]. These methods have shown the ability to
a deep learning network that leverages complementary in- restore low-resolution images with higher quality compared
formation from RGB and thermal image domains for both to traditional filter-based algorithms. However, they still
image alignment and super-resolution enhancement. face challenges in overcoming the ill-posed problem of cre-
The specific contributions of this paper are as follows: ating shapes that do not exist in the original image.
• Calibration and validation of the temperature measure-
2.3. Multi-Image or Multi-Modal Super Resolution
ment of a low-cost thermal camera in the agricultural do-
main To address the ill-posed problem in super-resolution
• Acquisition of a paired low-resolution thermal camera methods, attempts have been made to create a single high-
image dataset, as well as RGB and high-resolution ther- resolution image using various low-resolution or comple-
mal camera data in the agricultural domain mentary information. For example, one approach is to uti-
• Proposal of an integrated image alignment and super- lize the high-resolution panchromatic channel of satellite
resolution deep learning algorithm to improve the image imagery to enhance the sharpness of lower-resolution chan-
quality of low-resolution thermal cameras by combining nels [10]. Another approach is to combine information from
RGB and thermal images multiple frames to improve the sharpness of thermal im-
ages [7]. Additionally, multi-modal super-resolution tech- employed to capture high-resolution thermal image data in
niques that combine RGB and thermal information have the field. Positioned between high-end and consumer-grade
also been tried [2, 8, 17]. thermal cameras in terms of price, the FLIR Boson camera
offered a lightweight form factor and flexible video output
2.4. Use Cases of Thermal Cameras in Agriculture interface for easy field image capture. FLIR Boson images
Most agricultural research studies have traditionally re- were collected from the ROS-based system on Ubuntu PC.
lied on high-resolution thermal cameras in their research. Lastly, the FLIR One Pro, a low-cost and low-resolution
For example, Gonzalez-Dugo et al. [16] showed promis- thermal camera, was used in this study. It has a thermal
ing results assessing water stress within a commercial or- resolution of 160x120 and an RGB camera resolution of
chard using a high-resolution thermal camera, which costs 1440x1080. FLIR One Pro image acquisition and storage
more than $20,000. Yan et al. [33] recently employed a Pro were performed using a custom Swift-based app developed
SC TIR camera (640x512 resolution, $17,250) to estimate with the FLIR Mobile API on an iPhone.
evaporation, transpiration, and evapotranspiration, crucial 3.2. Low Cost Thermal Camera Calibration
parameters for understanding water dynamics in agricul-
tural systems. However, these cameras can be prohibitively Radiometric thermal cameras have a logarithmic rela-
expensive, limiting their accessibility for many researchers tionship between the digital number and temperature [23,
and farmers. 29]. The parameters for converting the digital number to
In recent years, the emergence of low-cost thermal cam- temperature are stored in the EXIF tag information of the
eras has opened up new possibilities for agricultural ap- FLIR radiometric JPEG images. These parameters, which
plications. Several studies have explored the use of low- are pre-calibrated values from the factory, are used to con-
cost thermal cameras in agricultural research. Garcı́a-Tejero vert the digital numbers of the thermal imaging camera to
et al. [13] compared the performance of a low-cost FLIR temperatures using Equation 1. Upon comparing the factory
One camera (80x60 resolution, $400) with a high-end FLIR parameters of different thermal imaging cameras, it was ob-
SC660 camera (640x480 resolution, $20,000) for assessing served that only the values of R1 and O differed, while the
crop water status. They found that the low-cost camera was values of R2 , B, and F remained constant. The parameter B
able to provide valuable insights, demonstrating the poten- is derived from the Planck constant h and Boltzmann con-
tial for more affordable thermal imaging solutions. Simi- stant kb , and the parameter F value is 1. For the FLIR One
larly, Iseki et al. [18] used a FLIR C2 camera (80x60 res- Pro cameras, R2 was fixed at 0.0125, and R1 and O are
olution, $500) to estimate leaf stomatal conductance, a key empirically calibrated depending on the individual camera.
indicator of plant water status. Parihar et al. [25] utilized a
FLIR E6 camera (240x180 resolution, $2,000) for irrigation
scheduling of horticultural plants, demonstrating its utility \text {Temperature }(\SI {}{\celsius }) = \frac {B}{\ln (\frac {R_1}{R_2(DN+O)})+F} - 273.15 \label {eq:flir_one_pro} (1)
in optimizing water use. While low-cost thermal cameras
offer an attractive alternative, their lower resolution and im- However, the accuracy of these factory parameters can-
age quality than their high-end counterparts may limit their not be fully trusted as the manufacturer does not fully guar-
ability to provide the same level of detailed information. antee the temperature accuracy of the low-cost thermal cam-
Additionally, the temperature accuracy of low-cost cameras eras. To ensure the accuracy of temperature measurements,
in various environmental conditions and crop types needs it is necessary to recalibrate the parameters of the thermal
further investigation. Nonetheless, the studies reviewed here imaging camera. Therefore, the optimization process fo-
highlight the potential of low-cost thermal cameras in agri- cused on optimizing the values of R1 and O. The optimiza-
cultural research. tion was performed using the Nelder-Mead method [24],
which is a widely used optimization algorithm, with a toler-
3. Materials and Methods ance of 1e − 6. The optimization process was implemented
using the ‘scipy.optimize.minimize’ function in Python.
3.1. Thermal Cameras
Experiments were conducted to verify the temperature
In this study, three types of thermal cameras were uti- accuracy of the FLIR One Pro thermal imaging camera.
lized. Table 1 shows the specifications of the thermal cam- The surface temperature of a controlled water bath was
eras. The VarioCAM HD camera, known for its high spatial measured using a thermocouple with a digital data logger.
resolution and temperature accuracy, was primarily used to The thermocouple measured the temperature starting from
create a dataset for temperature accuracy validation. The 4.0 ◦ C, the initial temperature of the cold water, and reach-
VarioCAM HD images were collected using their propri- ing 100.0 ◦ C, the boiling point of water. The air temperature
etary software on the Windows Operating System. The and relative humidity were maintained during the experi-
FLIR Boson camera, with a resolution of 640x512, was ment at 24.0 ◦ C and 40%.
 VarioCam HD Head 800 FLIR Boson FLIR One Pro
Spectral Range 7.5 - 14 µm 8 - 14 µm 8 - 14 µm
Detector Resolution 1,024 × 768 640 × 512 160 × 120
Temperature Measuring Range -40 - 2,000 °C Non Radiometric -20 - 120 °C
Measurement Accuracy ±1.5°C or ±1.5% Non Radiometric ±3°C or ±5%
Temperature Sensitivity 30mK 40mK 70 mK
Frame Rate 30 Hz & 60 Hz 9 Hz 8.7 Hz
Dimensions 221 × 90 × 94 mm 21 x 21 x 11 mm 68 × 34 × 14 mm
Weight 1.15 kg 21g 36.5 g
Price (Approx.) $20,000 $4,000 $400

Table 1. Specifications of thermal cameras used in this study

100 100
Thermocouple Temperature 1:1 Line

Thermal Camera Measurement (C°)

Factory Parameters Factory Parameters
80 Calibrated Parameters 80 Calibrated Parameters
Air Temperature
Temperature (C°)

60 60

40 40

20 20

0 0
0 200 400 600 800 1000 1200 20 40 60 80 100
Time (s) Thermocouple Temperature (C°)

Figure 2. Comparison between thermocouple, factory, and cali- Figure 3. Comparison between factory and calibrated temperature
brated temperature values in a time series values in a 1:1 plot

Figure 2 shows the temperatures calculated using the fac- 3.3. Matching Low-Resolution and High-
tory parameters. The results showed that the temperatures Resolution Thermal Imaging Cameras
calculated using the factory parameters were higher than
reference temperatures below 30 ◦ C. However, at tempera- As part of a more extensive set of breeding experi-
tures near the boiling point of water, the measured tempera- ments, Cowpea (Vigna unguiculata L. Walp.) and Common
tures were almost 20 ◦ C lower than the actual temperatures. Bean (Phaseolus vulgaris) images were collected from June
This indicates that the low-cost thermal camera’s tempera- to September 2022 in Davis, California, to obtain high-
ture values are inaccurate, especially at high temperatures. resolution and low-resolution thermal images in the field.
The original and optimized parameters are shown in Ta- To match the low-resolution and high-resolution thermal
ble 2, and the temperatures calculated using the optimized imaging datasets, camera calibration was performed to cal-
new parameters are shown in Figure 2. The temperatures culate each camera’s intrinsic parameters, and camera ex-
calculated using the new parameters are more accurate than trinsics were also measured. The two cameras were installed
the results using the factory parameters, and they are almost at a height of approximately 1.5m from the ground, and the
identical to the temperatures measured by the thermocouple distance between the centers of the two camera lenses was
(Figure 3). 5cm.
However, the high-resolution and low-resolution thermal
R1 B F O R2 images were captured on different platforms at different
frame rates, so matching the two datasets was challenging.
Factory 18333.4 1435 1 -2284 0.0125 Initially, SIFT [22] feature extraction and matching were at-
Optimized 12755.4 1435 1 -6707 0.0125 tempted, similar to the previous temperature accuracy test.
However, the quality and the number of the extracted fea-
Table 2. Comparison of factory and lab calibrated parameters tures in the images sometimes incorrectly estimate the ho-
mography between the low-resolution and high-resolution
pair, which led to unstable matching and alignment results. results compared to other methods. Figure 5 illustrates the
Since the field of view difference between the two im- input RGB image, the RGB-to-thermal image translated by
ages is only due to the scale difference based on the image Cycle GAN, and the low-resolution thermal image to be
resolution and the transitional offsets caused by the capture aligned. Inspired by the approach of Arar et al. [3], the RGB
timings, template matching [27] was performed to robustly image was first translated to the thermal imaging camera’s
match the images by setting the high-resolution image as domain using Cycle GAN [34]. Then, template matching
the template image T and calculating the Normalized Cross was performed between the domain-translated RGB image
Correlation (NCC) [6] between the template image and the and the input low-resolution thermal image. The maximum
low-resolution image I, finding the coordinates x∗ and y ∗ correlation value was calculated based on the image convo-
where the NCC value was maximized, and resizing the tem- lution operation from one image to another, which can be
plate image to a predefined scale for this process. hardware-accelerated and integrated into a super-resolution
module using PyTorch.
After aligning the domain-transformed RGB image with
the thermal image, the original RGB image was also trans-
R(x,y)=\sqrt {\frac {\sum _{x',y'}(T(x',y')-I(x+x',y+y'))^2}{\sum _{x',y'}T(x',y')^2\cdot \sum _{x',y'}I(x+x',y+y')^2}}
formed using the alignment result. Subsequently, the RGB,
(2) domain-transformed, and low-resolution thermal images
were combined and inputted into a ResNet-based Convolu-
(x^*, y^*) = \text {argmax}_{\substack {0 \leq x < M \\ 0 \leq y < N}} R(x,y) (3) tional Neural Network (CNN). The output image was then
fed into a Discriminator CNN for Generative Adversarial
The template matching was performed using Python Network (GAN) training. This architecture is depicted in
OpenCV code. Figure 4 illustrates matching the low- Figure 1, referred to as VisTA SR.
resolution and high-resolution thermal imaging cameras. Except for CycleGAN [34] and Template Matching, the
For cases where the NCC value was 0.75 or higher, the implementation followed that of SRGAN [21] and ESR-
bounding box was calculated and limited to the area within GAN [31], and the loss function used is as follows:
the padding of the low-resolution thermal image coordi- Cycle Consistency Loss [34]:
nate system. Then, it was converted to the coordinate sys-
l_{\text {Consi}}^{\text {Cycle}} = \left |I_{\text {RGB}} - G_{\text {IR2RGB}}(G_{\text {RGB2IR}}(I_{\text {RGB}}))\right | (4)
tem before resizing the template image. Image cropping
was performed using the original resolution of the tem- Identity Loss [34]:
plate and background images. The FLIR One Pro also has
an integrated RGB camera, allowing simultaneous acquisi- l_{\text {MSE}}^{\text {Cycle}} = \left |\left |I_{\text {HR}} - G_{\text {RGB2IR}}(I_{\text {RGB}})\right |\right | (5)
tion of RGB images. Therefore, the RGB images were also
cropped using the Template Matching results. MSE Loss [21, 31]:
3.4. Improving Image Resolution by Combining l_{\text {MSE}}^{\text {SR}} = \left |\left |I_{\text {HR}} - G_{\text {SR}}(I_{\text {LR}}, I_{\text {RGB}})\right |\right | (6)
RGB and Thermal Imaging
In this paper, complementary information from the RGB Content Loss ([21, 31]):
image’s structural details and the thermal imaging camera’s
l_{\text {VGG}}^{\text {SR}} = \left |\left |\phi _{\text {VGG}}(I_{\text {HR}}) - \phi _{\text {VGG}}(G_{\text {SR}}(I_{\text {LR}}, I_{\text {RGB}}))\right |\right | (7)
intensity information is utilized to enhance the resolution
of the low-resolution thermal imaging camera. The RGB Adversarial Loss [21, 31]):
and thermal images obtained from the FLIR One Pro have
the same field of view, but they are not perfectly pixel- l_{\text {Adv}}^{\text {SR}} = -\log D_{\text {SR}}(G_{\text {SR}}(I_{\text {LR}}, I_{\text {RGB}})) (8)
aligned due to differences in camera lens position and video
stream delays, which poses a challenge in combining the Total Loss:
two modalities for resolution improvement. We tested deep-
learning based image registration methods such as Spatial l_{\text {total}, G} = (l_{\text {Re}}^{\text {Cycle}} + l_{\text {MSE}}^{\text {Cycle}}) + (l_{\text {MSE}}^{\text {SR}} + l_{\text {VGG}}^{\text {SR}} + {\alpha } l_{\text {Adv}}^{\text {SR}}) (9)
Transformer Networks [19] and Deformable Field-based
approaches [35]. However, those methods tended to learn 4. Results
a shortcut existing in the dataset, which is a mean offset of
4.1. Low-Cost Thermal Camera Field Validation
the images rather than the differences between the input im-
with High Fidelity Thermocouple Camera
ages, resulting in unstable experimental results.
Therefore, a template matching method based on image Field data was collected to validate the temperature ac-
intensity was employed to align the domain-transformed curacy of the low-resolution thermal imaging camera in a
image and the thermal image, which yielded more stable real-world environment with crops and soil. The data was
 High Res Thermal Resized High Res Low Res Thermal+ Padding
512 128 120 + padding
Resize
Use
x0.25* Bo u n di
Template ng B
ox

160 + padding
Matching

160
Matching Result
Crop
640

Cropped Cropped
Low Res High Res
Thermal Thermal

Figure 4. Matching and aligning process of low-resolution and high-resolution thermal images

Convolution Based Template Matching

∆𝑥, ∆𝑦 = 𝑎𝑟𝑔 max (𝐼%! ∗ 𝐼!"#$%! )(∆𝑥,∆𝑦)
∆',∆(
𝐼!"# CycleGAN 𝐼!"#$%! 𝐼%!

*
Figure 6. An example of feature matching based temperature com-
parison between FLIR One Pro and VarioCam HD Camera
Image Shift
𝐼)*+,-./ 𝑥, 𝑦 = 𝐼(𝑥 − ∆𝑥, 𝑦 − ∆𝑦)
the Root Mean Square Error (RMSE) was also improved
Figure 5. Matching RGB and thermal images using CycleGAN from 1.52 ◦ C to 1.40 ◦ C. Since using the factory param-
and template matching eters tends to overestimate the temperature when it is be-
low 20 ◦ C, as shown in Figure 3, the temperature values ob-
tained using the factory parameters in Figure 6 also showed
collected in the Garbanzo bean (Cicer arietinum) field lo- higher temperature measurements than the actual tempera-
cated in Davis, California. The ground truth temperature tures.
values were measured using a VarioCAM HD camera and Table 3 also indicates that when calculating RMSE and
compared with the temperature measured by the FLIR One R2 using only data between 15 ◦ C and 30 ◦ C, the tempera-
Pro thermal camera, and a total of 170 image pairs were col- ture measurements with calibrated parameters showed bet-
lected on April 5, 2022. Image feature points were extracted ter accuracy. Considering the typical leaf temperature of
from both images using the SIFT [22] feature extractor, and plants, the accuracy within this temperature range is cru-
they were matched using the Flann matching algorithm [1]. cial for thermal cameras used in agriculture. Therefore, the
Then, the homography between the two images was calcu- thermal camera calibration in this study demonstrates the
lated, and outliers were removed using the RANSAC algo- potential to enhance temperature measurement accuracy in
rithm [12]. As a result, the temperature values from the cor- agricultural research.
responding points in the two images were compared (Figure
6). 4.2. VisTA SR Result
The matching result for the 170 image pairs is shown In 2022, a total of 2612 image pairs were collected from
in Figure 7, and Table 3 summarizes the results. It indi- a warm-season grain legume field across the growing season
cates that the temperature measurement accuracy was im- by matching low-resolution(160x120, FLIR One Pro) ther-
proved from R2 = 0.86 to R2 = 0.89 after calibration, and mal images with high-resolution (640X512, FLIR Boson)
 High Fidelity vs Low Cost Thermal Camera ilarity Index (SSIM[32]) and the lowest Peak Signal-to-
40
1:1 Noise Ratio (PSNR), while SRGAN and VisTA SR demon-
Factory strated similar performance with an RMSE of 2.75 ◦ C. It
Low Cost Thermal Camera (°C)

35
Calibrated can be inferred that the higher RMSE value of the Bilin-
30 ear algorithm is because SRGAN and VisTA SR learned the
temperature distribution of the training dataset, and Bilin-
25 ear’s higher SSIM value is believed to be a result of the orig-
inal dataset already being aligned with the template match-
20 ing process. Additionally, SRGAN showed a higher PSNR
value than VisTA SR, but VisTA SR exhibited excellent vi-
15
sual quality, indicating that evaluating the performance of
15 20 25 30 35 40 the Super-Resolution (SR) algorithm solely based on these
High Fidelity Thermal Camera (°C) image metrics is not ideal.

Figure 7. Feature matching based temperature comparison result.

Technique RMSE (◦ C) SSIM PSNR
Plotted all matched temperature points for a total of 170 images
Bilinear 2.84 0.74 23.84
SRGAN[31] 2.74 0.63 24.26
All data 15 ◦ C - 30 ◦ C
VisTA SR (Ours) 2.75 0.63 23.67
R2 RMSE (◦ C ) R2 RMSE (◦ C)
Factory 0.86 1.52 0.83 1.52 Table 4. RMSE, SSIM, and PSNR comparison of Bilinear, SR-
Calibrated 0.89 1.40 0.86 1.39 GAN, and VisTA SR algorithms

Table 3. Low-cost thermal camera (FLIR One Pro) temperature

accuracy validation result before and after parameter calibration
5. Conclusion & Future Work
This paper proposes a method to enhance temperature
thermal images. 80% of these pairs were used as training accuracy and image sharpness using a low-resolution ther-
data, while the remaining 20% were used for validation. The mal imaging camera for agricultural image acquisition.
network was trained over 200 epochs with a batch size of First, we conducted a calibration process to improve the
4. Figure 8 demonstrates the image conversion quality and temperature accuracy of the low-resolution thermal imag-
image alignment performance of the CycleGAN module, ing camera, followed by field experiments for validation. It
which was trained simultaneously with the SR Network. is confirmed that the temperature accuracy improved when
As depicted in the example images, CycleGAN success- using the calibrated parameters. We propose the VisTA-SR
fully translated the image domain and template matching algorithm for converting low-resolution thermal images to
successfully aligned low-resolution thermal images based high-resolution ones by aligning and combining RGB and
on image intensity. low-resolution images. Through such improvements in tem-
Figure 9 presents the results from the multiple input im- perature accuracy and image sharpness, we will be able to
age scales obtained from the VisTA SR algorithm using detect small temperature differences between crop tissues
the input of the combined RGB image aligned with Cy- or parts, and analyze them in relation to genotypes, growth
cleGAN and Template Matching, compared to the results environments, growth stages, and various other factors.
of the Super-Resolution Generative Adversarial Network One limitation was the difficulty of evaluating the per-
(SRGAN) algorithm [21, 31] that utilizes only the exist- formance of super-resolution algorithms in agricultural data
ing thermal image modality. Our VisTA-SR demonstrated using existing image metrics. Since most super-resolution
higher sharpness by leveraging higher-frequency structural studies generate low-resolution images by down-sampling
information from the RGB image. This demonstrates that the high-resolution images. In this case, the pixels of
VisTA-SR improved the performance of capturing thermal the low-resolution and high-resolution pairs are perfectly
properties of smaller features at the organ level, as opposed aligned, so the image metrics are proportion to the al-
to the plant level. gorithm’s super-resolution performance. However, in our
Table 4 compares the performance of Bilinear interpo- study, low-resolution thermal images were actually col-
lation, Super-Resolution Generative Adversarial Network lected with high-resolution images. Therefore, the output
(SRGAN), and our proposed VisTA SR algorithm, where result of our algorithm from low-resolution input may not
the Bilinear interpolation method exhibited the highest Root have a perfect pixel match with the high-resolution image.
Mean Square Error (RMSE) but the highest Structural Sim- Considering the characteristics of those image metrics that
 Figure 8. Input low-resolution images, domain translated images, and aligned images using CycleGAN and template matching

ized thermal image metrics for agricultural data that reflect

these features for performance evaluation in future research
is necessary. In future studies, we will examine whether
the thermal image improvement algorithm maintains, im-
proves, or hallucinates temperature information in thermal
images. Also, using the thermal images processed with the
algorithm developed in this paper, we will estimate biophys-
ical parameters such as stomatal conductance in plants and
compare accuracy with original and high-resolution image
inputs.

6. Acknowledgement
This work was financially supported by the Bill and
Melinda Gates Foundation, Project ID: INV- 002830,
G×E×M Innovation in Intelligence for Climate Adapta-
Figure 9. Comparison of input RGB, low-resolution thermal im- tion.
age input, SRGAN[21] output in multiple image scales (64x64,
128x128, and 256x256), VisTA-SR output, and ground truth high- References
resolution thermal image
[1] Fast Approximate Nearest Neighbors with Automatic Algo-
rithm Configuration. In Proceedings of the Fourth Interna-
tional Conference on Computer Vision Theory and Applica-
change significantly even by a few pixel changes, it can tions, pages 331–340, Lisboa, Portugal, 2009. SciTePress -
Science and and Technology Publications. 6
be inferred that the image evaluation metrics used in Ta-
[2] Feras Almasri and Olivier Debeir. RGB Guided Thermal
ble 4 reflected errors derived from multiple camera systems Super-Resolution Enhancement. In 2018 4th International
problem, even if the VisTA-SR had excellent visual quality Conference on Cloud Computing Technologies and Applica-
result than others. However, from an agricultural research tions (Cloudtech), pages 1–5, 2018. 2, 3
perspective, temperature accuracy and the ability to detect [3] Moab Arar, Yiftach Ginger, Dov Danon, Amit H. Bermano,
plants are important for understanding their complex bio- and Daniel Cohen-Or. Unsupervised Multi-Modal Im-
physical characteristics. In other words, developing special- age Registration via Geometry Preserving Image-to-Image
 Translation. In Proceedings of the IEEE/CVF Conference der a Permanent Shade Net House. Sensors, 23(6):2915,
on Computer Vision and Pattern Recognition, pages 13410– 2023. 1
13419, 2020. 5 [16] V. Gonzalez-Dugo, P. Zarco-Tejada, E. Nicolás, P. A. Nortes,
[4] Mahendra Bhandari. Use of infrared thermal imaging for J. J. Alarcón, D. S. Intrigliolo, and E. Fereres. Using high
estimating canopy temperature in wheat and maize. Master’s resolution UAV thermal imagery to assess the variability in
thesis, West Texas A&M University, 2016. 1 the water status of five fruit tree species within a commercial
[5] Pedro José Blaya-Ros, Vı́ctor Blanco, Rafael Domingo, Ful- orchard. Precision Agriculture, 14(6):660–678, 2013. 1, 3
gencio Soto-Valles, and Roque Torres-Sánchez. Feasibility [17] Honey Gupta and Kaushik Mitra. Toward Unaligned Guided
of Low-Cost Thermal Imaging for Monitoring Water Stress Thermal Super-Resolution. IEEE Transactions on Image
in Young and Mature Sweet Cherry Trees. Applied Sciences, Processing, 31:433–445, 2022. 2, 3
10(16):5461, 2020. 1 [18] Kohtaro Iseki and Olajumoke Olaleye. A new indicator of
[6] Kai Briechle and Uwe D. Hanebeck. Template matching us- leaf stomatal conductance based on thermal imaging for field
ing fast normalized cross correlation. In Aerospace/Defense grown cowpea. Plant Production Science, 23(1):136–147,
Sensing, Simulation, and Controls, pages 95–102, Orlando, 2020. 3
FL, 2001. 5 [19] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and
[7] Pasquale Cascarano, Francesco Corsini, Stefano Gandolfi, koray kavukcuoglu. Spatial Transformer Networks. In Ad-
Elena Loli Piccolomini, Emanuele Mandanici, Luca Tavasci, vances in Neural Information Processing Systems. Curran
and Fabiana Zama. Super-Resolution of Thermal Images Us- Associates, Inc., 2015. 5
ing an Automatic Total Variation Based Method. Remote [20] Azar Khorsandi, Abbas Hemmat, Seyed Ahmad Mireei, Ra-
Sensing, 12(10):1642, 2020. 3 soul Amirfattahi, and Parviz Ehsanzadeh. Plant temperature-
[8] Xiaohui Chen, Guangtao Zhai, Jia Wang, Chunjia Hu, and based indices using infrared thermography for detecting wa-
Yuanchun Chen. Color guided thermal image super resolu- ter status in sesame under greenhouse conditions. Agricul-
tion. In 2016 Visual Communications and Image Processing tural Water Management, 204:222–233, 2018. 1
(VCIP), pages 1–4, 2016. 2, 3 [21] Christian Ledig, Lucas Theis, Ferenc Huszar, Jose Caballero,
[9] Xiangyu Chen, Xintao Wang, Jiantao Zhou, Yu Qiao, and Andrew Cunningham, Alejandro Acosta, Andrew Aitken,
Chao Dong. Activating More Pixels in Image Super- Alykhan Tejani, Johannes Totz, Zehan Wang, and Wenzhe
Resolution Transformer. In 2023 IEEE/CVF Conference on Shi. Photo-Realistic Single Image Super-Resolution Using
Computer Vision and Pattern Recognition (CVPR), pages a Generative Adversarial Network. In Proceedings of the
22367–22377, Vancouver, BC, Canada, 2023. IEEE. 2 IEEE Conference on Computer Vision and Pattern Recogni-
[10] Farzaneh Dadrass Javan, Farhad Samadzadegan, Soroosh tion, pages 4681–4690, 2017. 2, 5, 7, 8
Mehravar, Ahmad Toosi, Reza Khatami, and Alfred Stein. [22] David G. Lowe. Distinctive Image Features from Scale-
A review of image fusion techniques for pan-sharpening of Invariant Keypoints. International Journal of Computer Vi-
high-resolution satellite imagery. ISPRS Journal of Pho- sion, 60(2):91–110, 2004. 4, 6
togrammetry and Remote Sensing, 171:101–117, 2021. 2 [23] Waldemar Minkina and Sebastian Dudzik. Infrared Ther-
[11] Claude E. Duchon. Lanczos Filtering in One and Two Di- mography: Errors and Uncertainties. Wiley, 1 edition, 2009.
mensions. Journal of Applied Meteorology and Climatology, 3
18(8):1016–1022, 1979. 2 [24] J. A. Nelder and R. Mead. A Simplex Method for Function
[12] Martin A. Fischler and Robert C. Bolles. Random sample Minimization. The Computer Journal, 7(4):308–313, 1965.
consensus: A paradigm for model fitting with applications to 3
image analysis and automated cartography. Communications [25] Gunjan Parihar, Sumit Saha, and Lalat Indu Giri. Application
of the ACM, 24(6):381–395, 1981. 6 of infrared thermography for irrigation scheduling of hor-
[13] Iván Garcı́a-Tejero, Carlos Ortega-Arévalo, Manuel Iglesias- ticulture plants. Smart Agricultural Technology, 1:100021,
Contreras, José Moreno, Luciene Souza, Simón Tavira, and 2021. 3
Vı́ctor Durán-Zuazo. Assessing the Crop-Water Status in Al- [26] Rafael Rivadeneira, Angel Sappa, and Boris Vintimilla.
mond (Prunus dulcis Mill.) Trees via Thermal Imaging Cam- Multi-Image Super-Resolution for Thermal Images:. In Pro-
era Connected to Smartphone. Sensors, 18(4):1050, 2018. 3 ceedings of the 17th International Joint Conference on Com-
[14] Jaime Giménez-Gallego, Juan D. González-Teruel, Ful- puter Vision, Imaging and Computer Graphics Theory and
gencio Soto-Valles, Manuel Jiménez-Buendı́a, Honorio Applications, pages 635–642. SciTePress, 2022. 2
Navarro-Hellı́n, and Roque Torres-Sánchez. Intelligent ther- [27] J.N. Sarvaiya, Suprava Patnaik, and Salman Bombaywala.
mal image-based sensor for affordable measurement of crop Image Registration by Template Matching Using Normal-
canopy temperature. Computers and Electronics in Agricul- ized Cross-Correlation. In 2009 International Conference
ture, 188:106319, 2021. 1 on Advances in Computing, Control, and Telecommunication
[15] Jaime Giménez-Gallego, Juan D. González-Teruel, Pedro J. Technologies, pages 819–822, 2009. 5
Blaya-Ros, Ana B. Toledo-Moreo, Rafael Domingo-Miguel, [28] P. R. Smith. Bilinear interpolation of digital images. Ultra-
and Roque Torres-Sánchez. Automatic Crop Canopy microscopy, 6(2):201–204, 1981. 2
Temperature Measurement Using a Low-Cost Image-Based [29] Glenn J. Tattersall. Infrared thermography: A non-invasive
Thermal Sensor: Application in a Pomegranate Orchard un- window into thermal physiology. Comparative Biochemistry
 and Physiology Part A: Molecular & Integrative Physiology,
202:78–98, 2016. 3
[30] C. Tomasi and R. Manduchi. Bilateral filtering for gray and
color images. In Sixth International Conference on Com-
puter Vision (IEEE Cat. No.98CH36271), pages 839–846,
1998. 2
[31] Xintao Wang, Ke Yu, Shixiang Wu, Jinjin Gu, Yihao Liu,
Chao Dong, Chen Change Loy, Yu Qiao, and Xiaoou Tang.
ESRGAN: Enhanced Super-Resolution Generative Adver-
sarial Networks, 2018. 2, 5, 7
[32] Z. Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Im-
age Quality Assessment: From Error Visibility to Structural
Similarity. IEEE Transactions on Image Processing, 13(4):
600–612, 2004. 7
[33] Chunhua Yan, Jiao Xiang, Longjun Qin, Bei Wang, Zhe Shi,
Weiyang Xiao, Muhammad Hayat, and Guo Yu Qiu. High
temporal and spatial resolution characteristics of evapora-
tion, transpiration, and evapotranspiration from a subalpine
wetland by an advanced UAV technology. Journal of Hy-
drology, 623:129748, 2023. 3
[34] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A.
Efros. Unpaired Image-To-Image Translation Using Cycle-
Consistent Adversarial Networks. In Proceedings of the
IEEE International Conference on Computer Vision, pages
2223–2232, 2017. 5
[35] Jing Zou, Bingchen Gao, Youyi Song, and Jing Qin. A re-
view of deep learning-based deformable medical image reg-
istration. Frontiers in Oncology, 12, 2022. 5