×

Wildlife Insights Bulk Uploads

Wildlife Insights Getting Started Guide

Bulk upload your data

Have a lot of data ready to upload? Or have you already identified species in images and don't want to let all of your hard work go to waste? You can bring all of your new and old data together in Wildlife Insights. 

Note: Bulk upload services can be included with paid subscriptions or available as a paid add-on for non-subscribers, with pricing based on data volume. Visit our pricing page for more information, or contact the Wildlife Insights team for a quote.

1. Download the batch upload templates

To bulk upload your data into Wildlife Insights, you'll need to make sure your data is in the bulk upload template format. The files needed to complete a bulk upload for one project are: projects, cameras, deployments and images (if your data is already annotated) .csv files. 

2. Complete the templates with your data

Enter your information into each of the templates. Refer to the Batch Upload Data Dictionary for default values, accepted formats, and required fields. Below are a few important steps to complete and recommendations to consider as you complete the templates:

  • Map taxa or common names used in your project to the Wildlife Insights taxonomy. All taxa and common names must be mapped to and changed to align with the Wildlife Insights taxonomy before bulk uploading.  The wi_taxon_id is also required for each image in column F in the images.csv file. You can find the wi_taxon_id , taxa and common names for any species using the Wildlife Insights Taxonomy Search Tool. You can also visit the Wildlife Insights Taxonomy Github repository and download the WI_Taxonomy.R file to access the most current version of the Wildlife Insights taxonomy.
  • Please avoid spaces and any special characters (hyphens and underscores are OK) in your folder paths and file names. The images.csv includes fields for your image filenames and their location on the Google Cloud Platform or other publicly accessible cloud storage. Please follow best practices and avoid spaces and special characters for these fields.
  • Leave non-required fields blank if they are not applicable to your project. If a field is not required and is not relevant to your project, please leave the field blank as you complete the template. Do not enter "NA" or "None". For example, the deployments.csv template includes a field for importing subproject_name. If your project does not use subprojects, please leave the field blank.
  • Create separate CSV files for each project. If you are uploading multiple projects, please create separate CSV files for each project. For example, if you have two projects to be batch uploaded, you will have 8 files in total: 2 projects.csv files, 2 deployments.csv files, 2 cameras.csv files and 2 images.csv files.
  • Save templates as a CSV and not an Excel file. To preserve date/time formats and trailing zeros, please save your completed template as a CSV file.
  • Consider building off of existing data migration code. The Wildlife Insights team has written code to transform data exported from popular camera trap software into the Wildlife Insights format. Code exists to transform data exported from Wild.ID, the Sanderson method, Camelot, and a few custom transformations. If you're writing your own custom code, this code is a great starting point! 

3. Submit your completed templates to Wildlife Insights

Submit your completed templates to a the Wildlife Insights team at [email protected], who will validate that your templates are correctly completed.
 

4. Upload your images to Wildlife Insights

You can transfer images to Wildlife Insights by uploading them directly to a private storage bucket in Wildlife Insights' Google Cloud Platform. Alternatively, you can make your images available from a public URL (e.g., a Google Drive or AWS link); if you'd like to use this option, contact the Wildlife Insights team to coordinate.

Note: Please avoid spaces and special characters (hyphens and underscores are OK) in your folder paths and file names.

Uploading images to Google Cloud Platform

A. Share your Google account with the Wildlife Insights team 

This can be either a Gmail account or a business Google Cloud account. The Wildlife Insights team will associate your account with a private storage bucket or folder on Wildlife Insights' Google Cloud Platform and grant you access to it.

B. Install Google Cloud CLI

The Google Cloud CLI (gcloud) is a tool that lets you access and manage files in Cloud Storage, including via the gcloud storagecommand set. Install it following the instructions at https://cloud.google.com/sdk/docs/install.

C. Upload Your Images to Google Cloud Platform

The Wildlife Insights team will create a top-level "bucket" for your organization. Underneath it, create a project "folder" for each project, and within each project folder, a deployment subfolder for each deployment. Upload your images into the matching deployment subfolder. In this example, the project URL is gs://wildmon/LondonHogwatch.

To copy files from a local storage location (i.e., laptop, attached drive, SAN, etc.):

  1. Open a terminal (MacOS/Linux) or Command Prompt/PowerShell (Windows).

  2. Authenticate with the Google account that you shared with the Wildlife Insights admins, using the following command:

    gcloud auth login

  3. Find the local folder path for the images. This path should match the location field in your images.csv . Example:  /Desktop/LH_Project1/Deployment1 (MacOS/Linux) or C:\Desktop\LH_Project1\Deployment1 (Windows).

  4. Upload the images from this folder (replace the curly brackets { } values with the relevant values):

    gcloud storage cp -r {LOCAL_FOLDER_PATH}/* gs://{BUCKET_NAME}/{PROJECT_NAME}/{DEPLOYMENT_NAME}/

The above command means the following:

cp:copy;
-r: recursive, or copies all images and subfolders below your top-level local folder path

In this example, the command would be:

MacOS/Linux

gcloud storage cp -r /Desktop/LH_Project1/Deployment1/* gs://wildmon/LondonHogwatch/Project1/Deployment1/

Windows

gcloud storage cp -r C:\Desktop\LH_Project1\Deployment1\* gs://wildmon/LondonHogwatch/Project1/Deployment1/