Environmental stresses such as drought and heat can cause substantial yield loss in agriculture. As such, hybrid crops which are tolerant to drought and heat stress would produce more consistent yields compared to the hybrids which are not tolerant to these stresses. In the 2019 Syngenta Crop Challenge, Syngenta released several large datasets that recorded the yield performances of 2,452 corn hybrids planted in 1,560 locations between 2008 and 2017 and asked participants to classify the corn hybrids as either tolerant or susceptible to drought stress, heat stress, and combined drought and heat stress. As one of the winning teams, we designed a two-step approach to solve this problem in an unsupervised way since no data was provided that classified any set of hybrids as tolerant or susceptible to any type of stress. First, we designed a deep convolutional neural network (CNN) that took advantage of state-of-the-art modeling and solution techniques to extract stress metrics for each type of stress. Our CNN model was found to successfully distinguish between the low and high stress environments due to considering multiple factors such as planting/harvest dates, daily weather, and soil conditions. Then, we conducted a linear regression of the yield of hybrid against each stress metric, and classified the hybrid based on the slope of the regression line, since the slope of the regression line showed how sensitive a hybrid was to a specific environmental stress. Our results suggested that only 14 % of the corn hybrids were tolerant to at least one type of stress.