DeepLabCut Notebook

Create a new DeepLabCut Project

Note: Open this jupyter notebook in your DeepLabCut environment.

Create DLC Project

Import python libraries needed for this notebook

try:
    import deeplabcut
    import tensorflow
    import tkinter
    from tkinter import filedialog

    print(f'Using DeepLabCut version: {deeplabcut. __version__}')
    print(f'Using TensorFlow version: {tensorflow. __version__}')   

except:
    print("Please run the notebook in in your local environment")

Start by selecting the list of videos to be included in the model. You could manually type the full path of each video in a python list as argument of the deeplabcut.create_new_project() function, like so:

['C:\\Users\\computername\\Videos\\reachingvideo1.avi', 
 'C:\\Users\\computername\\Videos\\reachingvideo2.avi', 
 'C:\\Users\\computername\\Videos\\reachingvideo3.avi']

Note

Windows users need to use the double backslash for path directories or a python raw filestring.

Instead, we use tkinter to open a file dialoge and save the file paths in a python list called videolist:

video_files = filedialog.askopenfilenames(title='Choose new video files to analyze in DeepLabCut:')
videolist = list(video_files)

print(f'{len(videolist)} videos selected:')
for i in range(len(videolist)): 
    print(videolist[i])

Now we create a new project using the video paths in videolist, give the project a name and set a few parameters:

config_path = deeplabcut.create_new_project('Name of project', 'Name of Experimenter', videolist, working_directory='path working directory', copy_videos=True, multianimal=False)

Note

You can load existing DeepLabCut projects by specifying the config_path as below:

config_path = filedialog.askopenfilename(title='Choose DeepLabCut config.yaml file:')

Configure Project

Now that a new project has been created with a specific directory structure and configuration file, we need to tweak some parameters to tailor the bodyparts we want to track:

import webbrowser
webbrowser.open(config_path)
print('Please edit bodyparts list to be tracked')

Once happy with all bodyparts, skeleton: and numframes2pick: settings, start extracting frames to label:

deeplabcut.extract_frames(config_path, mode='automatic', algo='kmeans', userfeedback=False, crop=False)

Label Frames

deeplabcut.label_frames(config_path)

You can plot your labeled frames to check your annotation accuracy.

deeplabcut.check_labels(config_path)

Train Model

deeplabcut.create_training_dataset(config_path, num_shuffles=1, net_type='resnet_50', augmenter_type='imgaug')
deeplabcut.train_network(config_path, shuffle=1, trainingsetindex=0, max_snapshots_to_keep=5, displayiters=200, saveiters=20000, maxiters=1030000, allow_growth=True)

Evaluate Model

deeplabcut.evaluate_network(config_path,Shuffles=[1], plotting=True)

Analyze Videos

deeplabcut.analyze_videos(config_path, videolist, shuffle=1, save_as_csv=True, videotype='mp4' )
deeplabcut.filterpredictions(config_path,videolist, videotype='.mp4',filtertype= 'arima',ARdegree=5,MAdegree=2)
deeplabcut.analyzeskeleton(config_path, videolist, videotype='.mp4', shuffle=1, trainingsetindex=0, save_as_csv=False, destfolder=None)

Create labeled videos

deeplabcut.create_labeled_video(config_path, videolist, videotype='.mp4', filtered=False, trailpoints=10, draw_skeleton = True)
deeplabcut.create_labeled_video(config_path, videolist, videotype='.mp4', filtered=True, trailpoints=10, draw_skeleton = True, keypoints_only=True)
deeplabcut.plot_trajectories(config_path, videolist)

Load Existing Model

config_path = filedialog.askopenfilename(title='Choose the config.yaml file of your DeepLabCut project:')

print(f'Using project path: {config_path}')

Load new videos to analyze and/or merge to the project:

video_files = filedialog.askopenfilenames(title='Choose new video files to analyze in DeepLabCut:')
new_videos = list(video_files)

print(f'{len(new_videos)} videos selected:')
for i in range(len(new_videos)): 
    print(new_videos[i])

Analyze new videos with the previously trained model

The analysis results in this step will most probably not look as desired, but this is to be expected. Bare in mind, that we are trying to analyze new, unseen videos with a model trained on different data (i.e., either other individuals, other sessions and maybe even different light conditions or camera angles). The goal of this step is to find out where the model fails to generalize and cover this shortcomings.

deeplabcut.analyze_videos(config_path, new_videos, shuffle=1, save_as_csv=True, videotype='mp4' )
deeplabcut.filterpredictions(config_path, new_videos, shuffle=1, save_as_csv=True, videotype='mp4')
deeplabcut.create_labeled_video(config_path, new_videos, filtered=True, videotype = 'mp4', save_frames=False)

Extract outlier frames

Now this is the interesting part. Instead of including more videos to the project directly, and extracting frames as usual with kmeans, we are taking advantage of the previous model to tell us what frames exactly to label. This active learning step helps us recognize the shortcomings of our model and correct it in a targeted manner.

deeplabcut.extract_outlier_frames(config_path, new_videos, automatic=True)

Now that we have extracted new frames, we need to go back and start labeling. Instead of starting from the beginning, though, we are provided the model predictions and have to drag and drop them in place. Note: Make sure to remove labels that are not visible, the model will often guess the expected position based on learned geometric constraints.

deeplabcut.refine_labels(config_path)

You can again plot your labeled frames to check annotation accuracy.

deeplabcut.check_labels(config_path)

At this point you could get an error message like this telling you that saving the video path failed. In this case, you need to add the new video paths manually for DLC to include these in the new training set. You can either add them by hand, writing in the config.yaml file in the same format as the first video paths (see here), or you can run the following command to add the list of videos to your config file:

deeplabcut.add_new_videos(config_path, new_videos, copy_videos=False)

If the permission error persists, try starting a new anaconda terminal as administrator (right click > run as administrator) and then starting jupyter notebook with elevated privileges.

Merge Datasets

After refining all outlier frames extracted above, merge the datasets to combine old and new labels in your project.

deeplabcut.merge_datasets(config_path)
deeplabcut.create_training_dataset(config_path, net_type='resnet_50', augmenter_type='imgaug')

Note: Make sure that the new videos have been included in the config.yaml file without permission issues (see above).

Re-Train Network

When training a new model with an expanded dataset, you could either choose to start fresh with new data, or use the previous model as pre-trained network for your next model. Although not yet extensively verified, lets belief that transfer learning at least won’t harm the new model.

pose_cfg =  filedialog.askopenfilename(title='Choose DeepLabCut pose_cfg.yaml file from dlc-models:')
import webbrowser
webbrowser.open(pose_cfg)
print('Please edit init_weights to include the path to the last pre-trained model...')
# Example of own pre-trained model
init_weights: D:\FacialExpression\old-DLC-Project\dlc-models\iteration-0\DLCApr14-trainset95shuffle1\train\snapshot-1030000

deeplabcut.train_network(config_path, shuffle=1, displayiters=100, saveiters=1000)

Start over again…