Quantifying species interactions can be a daunting task given the size, diversity, and variability of interacting communities. The incompleteness of interaction sampling is particularly apparent among studies that construct interaction networks (Fründ et al. 2016). This shortcoming stems from both the immense sampling effort required to comprehensively sample unique interactions and the ephemeral nature of interactions which rewire through time (Chacoff et al. 2012; CaraDonna et al. 2017). Some studies have approached these issues by developing analytical methods that are robust to sampling effort (Poisot et al. 2012), while others have attempted to increase sampling effort of traditional methods (CaraDonna et al. 2017, Chacoff et al. 2018). Another possible solution would be to utilize technology to change the way observational data are collected and processed. For instance, Spiesman and Gratton (2016) used HD video cameras to remotely record flower visitor observations that were processed at a later date. The use of cameras in the field frees researchers to perform other tasks or simultaneously operate multiple cameras. Running many cameras at once increases the number of “eyes” making observations, but the film must still be reviewed by a human at a later date to separate interactions from inactivity. I propose using computer vision applications to identify interactions contained within large amounts of film that is mostly inactivity. In this project, I determine whether current machine learning APIs can consistently and correctly detect insect visitors from photos of pollinators visiting flowers.

Question and Problem

Collecting observations of species interactions in the field is inherently time consuming and labor intensive. Can Google’s Vision API or Microsoft Azure’s Vision API consistently and correctly detect insects present in pictures of flowers?

Location and Grain

This work will focus on data I generated from a central Pennsylvania grassland. However, if my methods prove effective, they may be utilized in future studies of species interactions.


Film of potential flower visitors was collected by cameras trained on flowering plants in the summer of 2017. This footage was taken from directly above the flower to obtain the clearest image of insect visitors. I use Google’s Cloud Vision API and Microsoft’s Azure Vision API to attempt to identify visitors from six sets of 60 frames of insects visitations. The RoogleVision package allowed communication with Google’s Vision API and the packages jsonlite and httr allowed communication with Microsoft Azure’s Vision API.