Course: DAT255
Team Members: Mathias and Safiya
Institution: Høgskulen på Vestlandet
This project utilizes deep learning techniques to classify satellite images for the presence of ships. By using a binary classifier model, we process satellite images to detect and predict the presence of ships in standard optical imagery. The application leverages a sliding window algorithm to scale and evaluate small sections of the input image, ensuring that each part is analyzed for accuracy and robustness. Our application also supports ship detection for SAR (Synthetic Aperture Radar) images, where we utilize an object detection model from Facebook AI Researchs open-source software library, Detectron2. Our models can be deployed using Flask on localhost, where we also have a blogpost up.
- Optical satellite image of San Fransisco Bay area from "Ships in Satellite Imagery" [1]
- Image with probability of ship object in SAR image from "SARscope: Synthetic Aperture Radar Maritime Images". [2]
To recreate the models, visit the Kaggle repositories for each model:
-
For optical images: https://www.kaggle.com/code/mathiasskaugelarsen/dat255-project-optical
-
For SAR images: https://www.kaggle.com/code/mathiasskaugelarsen/dat255-project-sar
For optical images only, just clone the repository and run the main.py class. Connect on http://127.0.0.1:5000
For SAR images, its a bit more difficult, due to the Detectron2 needing to be used in a conda environment. Here are the steps to set up a conda environment, and run the model
Have some anaconda version installed, miniconda etc.
Run (in a bash shell):
conda init bash
conda create -n detectron2_env python=3.9
Click 'y' if asked
conda activate detectron2_env
Make sure detectron2_env is active the in terminal, if it later says (base) etc run conda activate detectron2_env again
Click 'y' if asked
conda activate detectron2_env
conda install pytorch torchvision torchaudio cudatoolkit=11.3 -c pytorch
pip install cython
pip install git+https://github.com/philferriere/cocoapi.git#subdirectory=PythonAPI
git clone https://github.com/facebookresearch/detectron2.git
python -m pip install -e detectron2
pip install flask fastai opencv-python
Select the detectron2_env as your python enterpreter
Uncomment the code needed in the main.py class, its 3 blocks of code thats commented out. Two blocks at the top of the class and one at the bottom.
Use shift+alt+A on windows or Cmd + Alt + Shift + C on mac to uncomment the whole block.
Finally, run the main class and connect on http://127.0.0.1:5000
The ability to accurately detect ships in satellite imagery has significant implications across several domains. It assists in monitoring sea lanes and detecting unauthorized ships like warships and even pirates, monitoring fishing activities to avoid illegal fishing or protect fish population, detection of lost ships, boats or debris in the ocean, and ship traffick for ports as well. [3] The use cases are many.
Satellite imagery provides a unique vantage point for monitoring vast oceanic areas that would otherwise be challenging and costly to survey from the ground or by sea. The comprehensive coverage and the ability to capture images over remote or inaccessible regions make satellite data a very valuable tool. Coupled with deep learning techniques, we can utilize it even more effective.
Synthetic Aperture Radar (SAR) images are particularly advantageous for ship detection due to their ability to acquire data irrespective of weather conditions and lighting. Unlike optical satellite images, which rely on sunlight and can be obstructed by clouds or fog, SAR images use radar waves to illuminate their targets and capture reflections. This capability ensures that SAR imagery can provide reliable, continuous observations during night-time and through poor weather conditions, crucial for year-round maritime surveillance.
- This image was generated with the assistance of AI.
- Creating model for optical satellite images: Kaggle, FastAI, resnet34
- Creating model for SAR satellite images: Kaggle, Detectron 2, Faster R-CNN
- Web Framework: Flask (Python), HTML
- Other Libraries: NumPy, OpenCV, matplotlib, torch
The binary classifier for optical images is designed to predict two classes:
- Ship - Presence of a ship in the image segment.
- No Ship - Absence of a ship in the image segment.
The model for optical images was trained on the dataset "Ships in Satellite Imagery" with labeled satellite images, that can be found in the kaggle dataset library. We utilized trasfer learning with a resnet-34 for model from fastai, which we fine-tune for a few epochs. This gives us a high performing model, and saves time and computing resources.
The model for SAR images was trained on the "SARscope: Synthetic Aperture Radar Maritime Images", that can also be found on kaggle. The model we used was one from Detectron2, which is a computer vision model zoo written in PyTorch by the FAIR Facebook AI Research group. The specific model we used was Faster_rcnn_R_50. We used this sinnce its listed as one of the state-of-the-art object detection frameworks [5] , and we wanted to gain experience using such a framework. It did cause us some trouble when using the model in our flask app, since the framework needs a conda environment to be used, but we learned a lot from using this aswell. The runtime was however quite long, which caused us to user a smaller subset of the data in the SAR dataset, to lower the runtime to about 6 hours to create the weights for our model.
The Faster_rcnn_R_50 model got decent results, but due to using a smaller porsion of the dataset it could have been better. Here is the breakdown from COCO evaluation metrics:
AP @[ IoU=0.50:0.95 | area= all | maxDets=100 ] = 0.434:
This is the mean Average Precision over IoU thresholds from 0.5 to 0.95 (with 0.05 increments) across all areas (sizes of objects) and considering the top 100 detections per image. An AP of 0.434 means the model has moderate accuracy overall.
AP @[ IoU=0.50 | area= all | maxDets=100 ] = 0.755:
This is the AP at a specific IoU threshold of 0.5. A higher AP at 0.75 indicates that the model performs well at this less strict threshold.
AP @[ IoU=0.75 | area= all | maxDets=100 ] = 0.450:
This is the AP at a more strict IoU threshold of 0.75. A lower AP here compared to IoU of 0.5 indicates the model is less accurate when stricter overlap criteria are used.
AP @[ IoU=0.50:0.95 | area= small/medium/large | maxDets=100 ]:
These lines show the AP for different object sizes (small, medium, large). It seems the model performs similarly for medium and large objects but worse for small objects.
Average Recall (AR) measures the average recall of the model at different IoU thresholds. Recall is the ratio of correctly predicted positive observations to the all observations in the actual class (how many of the relevant objects were found).
AR @[ IoU=0.50:0.95 | area= all | maxDets=1/10/100 ]:
These lines indicate the maximum recall given 1, 10, or 100 detections per image. As the number of detections allowed increases, the recall also increases because the model has more opportunities to make correct predictions.
AR @[ IoU=0.50:0.95 | area= small/medium/large | maxDets=100 ]:
These lines show the AR for different object sizes (small, medium, large). The model has higher recall for medium and large objects, indicating it's better at detecting them compared to small objects.
The Flask-based web interface employs a sliding window technique to:
-
Scale each uploaded image to the input size required by the trained model.
-
Move a sliding window across the image, segment by segment.
-
Feed each segment independently to the model for ship detection.
-
Aggregate results to determine the presence of ships in the entire image.
- Our deployment of the models with the optoin of uploading, using sample images, and SAR or Optical options
- Robert Hammell. (2018). Ships in Satellite Imagery [Data set]. Kaggle. https://doi.org/10.34740/KAGGLE/DSV/61115
- https://www.kaggle.com/datasets/kailaspsudheer/sarscope-unveiling-the-maritime-landscape/data
- https://ieeexplore.ieee.org/document/8767844
- https://colab.research.google.com/github/styler00dollar/Colab-Detectron2/blob/resnest/Colab-Detectron2.ipynb#scrollTo=_MZbDQNfx2DB (For some of the downloading cells for Detectron2
- https://roboflow.com/model-task-type/object-detection