Upload your already catalogued data
If you've already catalogued your data and identified species in images, you don't need to let all that hard work go to waste! You can bring all of your old and new data together in Wildlife Insights. Continue reading below to learn how you can bulk upload all of your catalogued data into Wildlife Insights.
Note: Bulk upload services are currently only available as a paid service. If you are interested in bulk uploading your data, contact the Wildlife Insights team at [email protected] for pricing.
SKIP AHEAD TO
Download the batch upload templates
Complete the templates with your data
1. Download the batch upload templates
To batch upload your annotated data into Wildlife Insights, you'll need to make sure your data is in the bulk upload template format. The four files needed to complete a bulk upload for one project are: Project, Camera, Deployment and Image .csv files. To get started:
- Download a blank bulk upload template
- Review the Batch Upload Data Dictionary to learn more about required fields, format, and default values.
- You can also visit the Wildlife Insights Data Migration page on GitHub for more information about the batch upload process and additional resources.
2. Complete the templates with your data
Enter your information into each of the templates. Refer to the Batch Upload Data Dictionary for default values, accepted formats, and required fields. Below are a few important steps to complete and recommendations to consider as you complete the templates:
- Map taxa or common names used in your project to the Wildlife Insights taxonomy. All taxa and common names must be mapped to and changed to align with the Wildlife Insights taxonomy before bulk uploading. The
wi_taxon_id
is also required for each image in column F in theimages.csv
file. You can find thewi_taxon_id
, taxa and common names for any species using the Wildlife Insights Taxonomy Search Tool. You can also visit the Wildlife Insights Taxonomy Github repository and download theWI_Taxonomy.R
file to access the most current version of the Wildlife Insights taxonomy. - Please avoid spaces and any special characters (hyphens and underscores are OK) in your folder paths and file names. The images.csv includes fields for your image filenames and their location on the Google Cloud Platform or other publicly accessible cloud storage. Please follow best practices and avoid spaces and special characters for these fields.
- Leave non-required fields blank if it is not applicable to your project. If a field is not required and is not relevant to your project, please leave the field blank as you complete the template. Do not enter "NA" or "None". For example, the deployments.csv template includes a field for importing subproject_name. If your project does not use subprojects, please leave the field blank.
- Create separate CSV files for each project. If you are uploading multiple projects, please create separate CSV files for each project. For example, if you have two projects to be batch uploaded, you will have 8 files in total: 2
project.csv
files, 2deployment.csv
files, 2camera.csv
files and 2images.csv
files. - Save templates as a CSV and not an Excel file. To preserve date/time formats and trailing zeros, please save your completed template as a CSV file.
- Consider building off of existing data migration code. The Wildlife Insights team has written code to transform data exported from popular camera trap software into the Wildlife Insights format. Code exists to transform data exported from Wild.ID, the Sanderson method, Camelot, and a few custom transformations. If you're writing your own custom code, this code is a great starting point!
3. Submit completed templates to Wildlife Insights
Once you finalize your batch upload template, please contact the Wildlife Insights team at [email protected] with a short description of your project and the completed templates. The Wildlife Insights team will validate that your templates are correctly completed and contact you about the next steps.
4. Upload images to Google Cloud Platform
You can transfer images to Wildlife Insights by either making them available from a public URL (e.g., a Google Drive, AWS link, etc.) or uploading them directly to a private storage bucket in the Google Cloud Platform.
Note: Please avoid spaces and special characters (hyphens and underscores are OK) in your folder paths and file names.
Uploading images to Google Cloud Platform
A. Please share your Google account name with the Wildlife Insights admins. This can either be a Gmail or an associated business Google Cloud account. The Wildlife Insights Admin will associate your account with a private storage object in Wildlife Insight's Google Cloud Platform and assign you access to the GCP storage bucket.
B. Install gsutil
- Please go to this Google Cloud Platform (GCP) page, select your operating systems and install gsutil.
- Open a terminal in MAC or Windows DOS Prompt, depending on your operating system.
- Enter the following at the command prompt:
curl https://sdk.cloud.google.com | bash
- Choose a Directory to download the file to (example: Desktop)
- You’ll be asked if you want to help improve the Google Cloud SDK. Type “Y” or “N”
- Type “Y” to continue when prompted to modify profile
- Enter the following at the command prompt:
exec -l $SHELL
- Enter the following at the command prompt:
gcloud init
- If prompted to pick a configuration, enter 1 to reinitialize the default configuration
- Choose the account you would like to use for your configuration
- You’ll be asked if you would like to create a project. Type “N”
C. Uploading Batch Upload Templates and Images to GCP
Once you have gsutil installed, the next step is to upload your images. WI Admins will create a top-level ‘bucket’ to house images for your organization. In this example, the bucket is called gs://LondonHogwatch
To copy files from your local storage location (i.e., laptop, attached drive, SAN, etc.) you will need to follow these steps:
- Open a terminal in MAC or Windows DOS Prompt, depending on your operating system.
Run the commands below to ensure you’re logged in locally with the same username used to create the Google GCP account:
gcloud auth login
gsutil config
- Find the local path to the folder that you want to upload. Within this folder should be all the images in your project. In this example, the folder name is Desktop/LH_Project1
- Use the following command to upload images:
gsutil -m cp -r YOUR_TOP_LEVEL IMAGE_DIR gs://LondonHogwatch
The above command means the following:
-m: starts many computer threads to upload (i.e., this means it will upload more than one image at a time);
cp: copy;
-r: recursive, or that all images and directories below your’ top level image dir’ will be copied.
In this example, the command would be
gsutil -m cp -r Desktop/LH_Project1 gs://LondonHogwatch