A new collaborative research paper in a special issue of Frontiers in Marine Science describes the potential use of artificial intelligence and machine learning in fisheries surveys. Specifically, it looks at how it could increase efficiency for operations of two fishery-independent surveys led by the Southeast Fisheries Science Center.
NOAA’s Southeast Fisheries Science Center has been conducting video surveys in the Gulf of Mexico for more than 30 years. These long-term video surveys help provide scientists insight into the population status of ecologically and economically important species over long time frames. They provide data on habitat and water quality that are critical to maintain the health of the vital snapper-grouper fisheries of the Gulf of Mexico. While video surveys have increased in importance to NOAA Fisheries and the Southeast Fisheries Science Center, processing the video remains a manual process. It represents a significant bottleneck to providing timely guidance for fisheries managers. Recent advancements in automated image analysis show promise to address the processing bottleneck providing timely data for fisheries assessments.
We partnered with Mississippi State University-Northern Gulf Institute and Kitware Inc. to develop an Artificial Intelligence and Machine Learning process to automate the processing of videos collected in the Gulf. This software system is called Video Image Analytics for the Marine Environment (VIAME). It uses the videos and images to:
- Search for fish
- Detect objects and fish
- Track species movement
- Enhance video
- Annotate video
- Generate products quickly
- Become smarter with the collected data
Matthew Campbell, co-author and co-lead of the project explained, “This artificial intelligence software system will be a huge stepping stone for us in terms of timeliness and efficiency delivering analytical products from these critical surveys.”
Accelerating Efficiency
Manual processing of video surveys conducted by scientists takes a significant amount of time. For example, combining one year of sampling from the Gulf Fishery Independent Survey of Habitat and Ecosystem Resources video surveys results in approximately 2,000 camera deployments, approximately 1,000 hours of footage, and around 30 terabytes of data. It typically takes a full year of manual video analysis (identifying and enumerating species) to turn the video collections into usable data for assessing fish.
With the new artificial intelligence in VIAME, video processing is projected to get faster over time as the automation is optimized for specific species. It will be expanded to include additional species. We anticipate that processing will be reduced to only a few months of work. Scientists will still have to ensure the precision and accuracy of that data compared to the manual processes used historically.
Accessibility and Functions
The project is taking an Open Science approach by making the software accessible to anyone who would like to use it. This will allow individuals to access VIAME as an online platform to test and build models, or for download and use on local computing. However, the online platform does not have the server space for large batch processes and is not intended to serve in that regard. The decision to have an open source format was made by Automated Image Analysis Strategic Initiative as a way to facilitate uptake and use of Artificial Intelligence and Machine Learning processing. Our center is using this open source model to provide this solution to groups that otherwise wouldn't have the resources to compile the imagery and create models themselves. This is particularly important once we start extending to the Caribbean.
Dr. Campbell explained, “Our team sticks to evaluating fish populations, so I hope by using this open source format that habitat scientists could access the imagery and begin to create habitat labels to automate that component.”
The software received its original data from the available imagery from previously processed video surveys. It consisted of 600 thousand individual annotations across about 140 fish species in the Gulf of Mexico. This accumulation of imagery data is a strong start for the Artificial Intelligence and Machine Learning classification pipeline. However, the project will need to establish a cloud-based workflow to be able to store, access, and archive large video collections.
Human Involvement
The establishment and continuous advancements of this software will take the weight off of scientists’ shoulders, but not entirely. Due to the critical status of the survey to assess snapper and grouper in the Gulf of Mexico, we must take great care to assure data processed using automated approaches are consistently accurate. This project is a step towards evaluating the accuracy of the automated system, thereby our team will be manually monitoring it for precision. We anticipate that, as machine learning grows to a high confidence level of precision, the human aspect of this process will be minimized. That will free up our scientists’ time to focus on analysis of the data and providing improved guidance to resource managers.
This project is revealing the productivity of integrating Artificial Intelligence and Machine Learning into NOAA Fisheries’ video surveys. The addition of these high computation tools will accelerate our center’s ability to understand critical aspects of important natural resources, habitats, and fish populations efficiently.
