×

Wildlife Insights Bulk Uploads

Wildlife Insights Getting Started Guide

Move your already catalogued camera trap data into Wildlife Insights

If you've already catalogued your data and identified species in images, you don't need to let all that hard work go to waste! You can bring all of your old and new data together in Wildlife Insights. Continue reading below to learn how you can bulk upload all of your catalogued data into Wildlife Insights.

Note: Bulk upload services are currently only available for catalogued data and will be processed depending on our team's capacity. If you're interested in bulk uploading uncatalogued data, contact our team for pricing.

1. Download the batch upload templates

To batch upload your annotated data into Wildlife Insights, you'll need to make sure your data is in the batch upload template format. There are four templates needed to complete a batch upload for one project: one template each for a Project, Camera, Deployment and Images. To get started migrating your data into this format:

You can also visit the Wildlife Insights Data Migration page on GitHub for more information about the batch upload process and additional resources.

 

2. Complete the templates with your data

Enter your data into each of the templates. As you enter your data, make sure to refer to the Batch Upload Data Dictionary for default values, accepted formats, and required fields. Here are a few important steps to complete and recommendations to consider as you complete the templates:

  • Map the scientific or common names used in your project to the Wildlife Insights taxonomy list. Species names and objects must be mapped to the Wildlife Insights taxonomy and converted to a WI GUID before bulk uploading. The WI GUID is required for each image and will be used to populate column F in the images.csv file. These two resources can be used to find the WI GUID for your species:
  • Please avoid spaces and any special characters (hyphens and underscores are OK) in your folder paths and file names. The images.csv includes fields for your image filenames and the location in GCP or other publicly accessible cloud storage. Please follow best practices and avoid spaces and special characters for these fields.
  • Leave non required fields blank if it is not applicable to your project. If a field is not required and is not relevant to your project, please leave the field blank as you complete the template. Do not enter "NA" or "None". For example, the deployments.csv template includes a field for importing subproject_name. If your project does not use subprojects, please leave the field blank.
  • Create separate files for each project. If you are uploading multiple projects, please create separate files for each template. For example, if you have two projects to be batch uploaded, you will have 8 files in total: 2 project.csv files, 2 deployment.csv files, 2 camera.csv files and 2 images.csv files. 
  • Save templates as a .csv and not an excel file. To preserve date/time formats and trailing zeros, please save your completed template as a .csv.
  • Consider building off of existing data migration code. The Wildlife Insights team has written code to transform data exported from popular camera trap software into the Wildlife Insights format. Code exists to transform data exported from Wild.ID, the Sanderson method, Camelot, and a few custom transformations. If you're writing your own custom code, this code is a great starting point! 

 

 

3. Submit completed templates to Wildlife Insights

Once you finalize your batch upload template, please contact [email protected] with a short description of your project and the completed templates. Once submitted, the Wildlife Insights team will validate that the templates are correctly completed and will contact you with next steps.

Note: Bulk upload services are currently only available for catalogued data and will be processed depending on our team's capacity. If you're interested in bulk uploading uncatalogued data, contact our team for pricing.

4. Upload images to Google Cloud Platform

You can transfer images to the Wildlife Insights uploading the images directly to a private storage bucket in the Google Cloud Platform.

NOTE: Please avoid spaces and any special characters (hyphens and underscores are OK) in your folder paths and file names.

 

Uploading images to Google Cloud Platform

A. Please share your Google account name with the Wildlife Insights admins. This can either be a Gmail or associated business Google Cloud account.

The WI Admin will associate your account to a private storage object in the Google Cloud Platform and assign you access to the GCP storage bucket. 

B. Install gsutil

  1. Please go to this GCP page, select your operating systems and install gsutil.
  2. Open a terminal in MAC or Windows Dos Prompt depending on your operating system.
  3. Enter the following at the command prompt: curl https://sdk.cloud.google.com | bash
  4. Choose a Directory to download the file to (example: Desktop)
  5. You’ll be asked if you want to help improve the Google Cloud SDK. Type “Y” or “N”
  6. Type “Y” to continue when prompted to modify profile
  7. Enter the following at the command prompt: exec -l $SHELL
  8. Enter the following at the command prompt: gcloud init
  9. If prompted to pick a configuration, enter 1 to reinitialize the default configuration 
  10. Choose the account you would like to use for your configuration 
  11. You’ll be asked if you would like to create a project. Type “N”

 

C. Uploading Batch Upload Templates and Images to GCP

Once you have gsutil installed the next step is to upload your images. WI Admins will create a top-level ‘bucket’ to house images for your organization. In this example, the bucket is called gs://LondonHogwatch

To copy files from your local storage location (i.e. laptop, attached drive, SAN, etc) you will need to follow these steps:

  1. Open a terminal in MAC or Windows Dos Prompt depending on your operating system.
  2. Run the commands below to ensure you’re logged in locally with the same username used to create the Google GCP account:

    gcloud auth login

    gsutil config

  3. Find the local path to the folder that you want to upload. Within this folder should be all the images in your project. In this example, the folder name is Desktop/LH_Project1
  4. Use the following command to upload images:

gsutil -m cp -r YOUR_TOP_LEVEL IMAGE_DIR gs://LondonHogwatch

The above command means the following:

-m: starts many computer threads to upload (i.e. this means it will upload more than one image at a time);
cp: copy;
-r: recursive, or that all images and directories below your’ top level image dir’ will be copied.

In this example, the command would be 

gsutil -m cp -r Desktop/LH_Project1 gs://LondonHogwatch

If you have questions about this process, please consult this gsutil documentation.