We at Humans in the Loop are constantly on the lookout for the best image annotation platform that offers multiple functionalities, project management tools and optimization of the annotation process (even 1 second less per image matters when you have to annotate 50k images!).
Based on our experience with each one of these platforms, we are sharing here our honest reviews, hoping that this would be of use for data scientists looking to manually label their data.
Here are our criteria:
- Price
- Variety of functions, tools and formats
- Project management and ease of use
LabelImg is an open source image labeling tool that has pre-built binaries for Windows so it’s extremely easy to install.
- Price: Free
- Functionalities: only supports bounding boxes (there is also a version in the RotatedRect format and an optimized version for one-class tagging) but nothing more advanced. The format is PascalVoc XML and annotation files are saved separately for each image in the source folder.
- Project management: It has virtually no project management properties but it does allow an easy way to import and visualize annotations and correct them if necessary. The simple offline interface makes the annotation process pretty fast, even though it does not support many hotkey shortcuts.
VGG is an open-source tool that, just like LabelImg, can do an amazing job for straightforward tasks that do not require project management. It is available as an online interface and can also be used offline as an HTML file. In its most recent version, it also offers a wide variety of video labeling tools.
- Price: Free
- Functionalities: Offers a lot more tools, including dots, lines, polygons, circles and ellipses (only platform in this list supporting circles and ellipses!). Also has the option of adding object and image attributes/tags. The annotations can be downloaded as one JSON file containing all annotations, or as one CSV file, and can be uploaded afterwards if there is a need to review them.
- Project management: Nothing too advanced in terms of dataset management and users but their interface is one of the most efficient and precise ones for polygon annotation because it allows you to see the line of the polygon and nothing else. They support some hotkey shortcuts and the application is very lightweight in general.
3. Supervise.ly
Supervisely is an awesome web-based platform that offers an advanced annotation interface but also covers the entire process of computer vision training, including a deep learning models library that can be directly trained, tested, and improved within the platform.
- Price: Free community edition and enterprise pricing for the self-hosted version
- Functionalities: A great array of tools, including dots, lines, boxes, polygons, and a bitmap brush for semantic segmentation (we haven’t found their smart tool too useful though). Also includes the possibility to draw holes in polygons, which has been incredibly valuable. Another very useful feature is the option to add image and object tags and to order figures in layers. Output is in JSON files for each image or PNG masks and the platform also allows you to upload formats such as Cityscapes and COCO. In addition, there is an option to do data transformation directly on the platform.
- Project management: The platform offers tons of options for project management on various levels (teams, workspaces, datasets) and annotator management (labeling jobs, permissions, statistics). They also have a Data Transformation Language and a Python Notebooks option for managing the data which come in handy a lot. A couple of things missing are time statistics, as well as quality control mechanisms. Their tech support team is always available in the case of problems. The interface allows for very precise work and supports customizable hotkey shortcuts but performance has recently been slow at times, which can be pretty frustrating if the platform takes a lot of time to switch between images and record annotations.
4. Labelbox
Labelbox is another great web-based platform that launched in early 2018 and ever since then has been constantly updating and improving its functionalities. It also offers the possibility to integrate a human-in-the-loop by importing model predictions and seeing the consensus between the labelers and the model.
- Pricing: Free community edition limited to 5000 images and an enterprise version
- Functionalities: Offers a complete array of tools for annotation, such as points, lines, boxes and polygons, and has recently added an awesome new feature for their semantic segmentation brush — a superpixel coloring option that makes life so much easier when boundaries are clear (much like this and this open source tools). Output is as one JSON or CSV file containing all annotations or as PNG masks (however, there is one mask for every class and the user needs to figure out what to do with overlapping regions afterwards)
- Project management: Setting up a project is extremely easy, and there are many options for monitoring performance, including statistics on seconds needed to label an image. You can implement several quality control mechanisms, including activating automatic consensus between different labelers or setting gold standard benchmarks. You have the option to invite users (though permissions are not as granular) and to review the work of each one. The labeling interface is super user-friendly and supports hotkey shortcuts (although not customizable). One thing missing in the free version is the option to upload annotations so as to visualize or edit them.
Need something else? Here are some other platforms that you can consider:
- Diffgram — a really promising platform still in beta that optimizes image annotation by training RCNN, will be featured in the second article in this series!
- RectLabel — an awesome tool for bounding boxes and polygons for MacOS
- Prodigy — they offer a self-hosted back-end with different annotation interfaces, including image annotation with bounding boxes; pricing for their product starts at $390 for personal use (lifetime per user)
- DataTurks — a platform that offers many annotation capabilities; data annotated in the free version is publicly available, and enterprise pricing starts at $300 per month for small teams
- ImageTagger — an open source platform for collaborative image labeling
- Fast Annotation Tool — another open source tool, using OpenCV for bounding boxes in the RotatedRect format
- LabelMe — an industry classic, open source tool by MIT for polygonal annotation; precision is extremely low though
- PolygonRNN+ — available only as a demo, but still very promising; a tool that is trained on the Cityscapes dataset do generate automatic labels for self-driving cars with reinforcement learning
—
Dealing with large datasets and need to scale up your annotation efforts? Feel free to get in touch with us at hello@humansintheloop.org — our annotators are trained to use all of these platforms and we would love to contribute to your project.