Notebook 2

Instructions

README

• Navigating this notebook on Google Colab: There will be text blocks and code blocks throughout the notebook. The text blocks, such as this one, will contain instructions and questions for you to consider. The code blocks, such as the one below, will contain executible code. Sometimes you will have to modify the code blocks following the instructions in the text blocks. You can run the code block by either pressing control/cmd + enter or by clicking the arrow on left-hand side as shown.

  

• Saving Work: If you wish to save your work in this .ipynb, we recommend downloading the compressed repository from GitHub, unzipping it, uploading it to Google Drive, opening this notebook from within Google Drive, and setting WITHIN_GDRIVE to True.

• This notebook doesn’t require any modifications inside the code_and_data directory for a base walkthrough, but if you want to work on the optional exercises throughout the notebook, you will need to modify dsl.ipynb which will require saving your work. However, as a workaround, you can directly make changes to dsl.py and near.py and restart the runtime using Runtime > Restart Runtime. The video below shows how to do this.

• Finally, this notebook requires a NVIDIA GPU for training. Google Colab allows access to one GPU instance for free (at a given time). Go to Runtime > Manage Sessions > Accelerator: GPU.

# (Optional) Are you loading data from within Google Drive?
WITHIN_GDRIVE = False # otherwise: True

# Setup repository and download toy CalMS21 data
if not WITHIN_GDRIVE:
  !git clone https://github.com/neurosymbolic-learning/Neurosymbolic_Tutorial.git /content/Neurosymbolic_Tutorial
  %cd /content/Neurosymbolic_Tutorial/code_and_data
  !gdown 1XPUF4n5iWhQw8v1ujAqDpFefJUVEoT4L && (unzip -o calms21_task1.zip; rm -rf calms21_task1.zip )
else:
  from google.colab import drive
  drive.mount('/content/drive')
  # Change this path to match the corect destination
  %cd /content/drive/MyDrive/Neurosymbolic_Tutorial/code_and_data/
  import os; assert os.path.exists("dsl.py"), f"Couldn't find `dsl.py` at this location {os.getcwd()}. HINT: Are you within `code_and_data`"
  !gdown 1XPUF4n5iWhQw8v1ujAqDpFefJUVEoT4L && (unzip -o calms21_task1.zip; rm -rf calms21_task1.zip )


import os
import matplotlib.pyplot as plt
import numpy as np

fatal: destination path '/content/Neurosymbolic_Tutorial' already exists and is not an empty directory.
/content/Neurosymbolic_Tutorial/code_and_data
Downloading...
From: https://drive.google.com/uc?id=1XPUF4n5iWhQw8v1ujAqDpFefJUVEoT4L
To: /content/Neurosymbolic_Tutorial/code_and_data/calms21_task1.zip
100% 18.2M/18.2M [00:00<00:00, 167MB/s]
Archive:  calms21_task1.zip
  inflating: data/calms21_task1/test_data.npy  
  inflating: data/calms21_task1/test_investigation_labels.npy  
  inflating: data/calms21_task1/test_mount_labels.npy  
  inflating: data/calms21_task1/test_trajectory_data.npy  
  inflating: data/calms21_task1/train_data.npy  
  inflating: data/calms21_task1/train_investigation_labels.npy  
  inflating: data/calms21_task1/train_mount_labels.npy  
  inflating: data/calms21_task1/train_trajectory_data.npy  
  inflating: data/calms21_task1/val_data.npy  
  inflating: data/calms21_task1/val_investigation_labels.npy  
  inflating: data/calms21_task1/val_mount_labels.npy  
  inflating: data/calms21_task1/val_trajectory_data.npy  

(Important!) Convert Notebooks to Python Files

If you update the DSL in dsl.ipynb or the search algorithm in near.ipynb, you need to run this cell again to update the code in this notebook.

!jupyter nbconvert --to python dsl.ipynb
!jupyter nbconvert --to python near.ipynb

[NbConvertApp] Converting notebook dsl.ipynb to python
[NbConvertApp] Writing 41254 bytes to dsl.py
[NbConvertApp] Converting notebook near.ipynb to python
[NbConvertApp] Writing 39669 bytes to near.py

Search Tree Visualization

The Domain-Specific Languages we use for behavior analysis are in dsl.ipynb. Through this notebook, you will be training enumeration algorithms to search through the space of programs defined by this DSL, and there will also be exercises to modify the DSL to improve program performance.

See the figure below for a visualization of the search tree using an example DSL.

Enumeration: Base DSL

The enumeration algorithm will enumerate through all symbolic programs until max_num_programs is reached. Optional: see ENUMERATION in near.ipynb if you are interested in the implementation! Since the number of programs at each depth may be large, the argument max_num_programs limits the total number of programs we train.

We start with enumerating 25 programs (which enables us to search up to depth 5) using a basic DSL. Looking at the results printed at the end, do you observe anything common or different across program structures for the best and worst programs?

Note that:

• `score` = 1 - F1 Score
  

• `struct_cost` = cost of program structure (more complex programs have higher structural costs)
  

• `path_cost` = score + struct_cost
  

!yes | python train.py \
--algorithm enumeration \
--exp_name investigation_base \
--trial 1 \
--seed 1 \
--dsl_str "default" \
--train_data "data/calms21_task1/train_data.npy" \
--test_data "data/calms21_task1/test_data.npy" \
--valid_data "data/calms21_task1/val_data.npy" \
--train_labels "data/calms21_task1/train_investigation_labels.npy" \
--test_labels "data/calms21_task1/test_investigation_labels.npy" \
--valid_labels "data/calms21_task1/val_investigation_labels.npy" \
--input_type "list" \
--output_type "atom" \
--input_size 18 \
--output_size 1 \
--num_labels 1 \
--lossfxn "bcelogits" \
--learning_rate 0.0001 \
--symbolic_epochs 12 \
--max_num_programs 25 \
--class_weights "2.0"

Seed is 1
Experiment log and results saved at: results/investigation_base_enumeration_sd_1_001

DEBUG: starting enumerative synthesis with depth 1
DEBUG: 0 programs found
DEBUG: starting enumerative synthesis with depth 2
DEBUG: 8 programs found
DEBUG: starting enumerative synthesis with depth 3
DEBUG: 8 programs found
DEBUG: starting enumerative synthesis with depth 4
DEBUG: 8 programs found
DEBUG: starting enumerative synthesis with depth 5
DEBUG: 648 programs found
Symbolic Synthesis: generated 648/25 symbolic programs from candidate program.
Training candidate program (1/25) Start(Window5Avg(ResAngleHeadBodySelect()))
/content/drive/MyDrive/neurosymbolic_summer_school_notebooks/code_and_data_4/near.py:222: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:210.)
  batch_input = [torch.tensor(traj) for traj in batch]
Structural cost is 0.01 with structural penalty 0.01
Time to train child 7.202
Total time elapsed is: 7.202
New tentative BEST program found:
Start(Window5Avg(ResAngleHeadBodySelect()))
struct_cost 0.0100 | score 0.6675 | path_cost 0.6775 | time 7.2025
Training candidate program (2/25) Start(Window5Avg(SpeedSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 7.109
Total time elapsed is: 14.314
Training candidate program (3/25) Start(Window5Avg(TangentialVelocitySelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 7.179
Total time elapsed is: 21.493
New tentative BEST program found:
Start(Window5Avg(TangentialVelocitySelect()))
struct_cost 0.0100 | score 0.5733 | path_cost 0.5833 | time 21.4933
Training candidate program (4/25) Start(Window5Avg(AccelerationSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 7.247
Total time elapsed is: 28.742
Training candidate program (5/25) Start(Window5Avg(RelativeSocialAngleSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 8.677
Total time elapsed is: 37.419
Training candidate program (6/25) Start(Window5Avg(AxisRatioSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 7.477
Total time elapsed is: 44.897
New tentative BEST program found:
Start(Window5Avg(AxisRatioSelect()))
struct_cost 0.0100 | score 0.5693 | path_cost 0.5793 | time 44.8968
Training candidate program (7/25) Start(Window5Avg(OverlapBboxesSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 8.189
Total time elapsed is: 53.088
New tentative BEST program found:
Start(Window5Avg(OverlapBboxesSelect()))
struct_cost 0.0100 | score 0.3217 | path_cost 0.3317 | time 53.0878
Training candidate program (8/25) Start(Window5Avg(MinResNoseKeypointDistanceSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 7.137
Total time elapsed is: 60.227
Training candidate program (9/25) Start(Or(Window5Avg(ResAngleHeadBodySelect()), Window5Avg(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 10.089
Total time elapsed is: 70.316
Training candidate program (10/25) Start(Or(Window5Avg(SpeedSelect()), Window5Avg(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 9.963
Total time elapsed is: 80.279
Training candidate program (11/25) Start(Or(Window5Avg(TangentialVelocitySelect()), Window5Avg(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 9.996
Total time elapsed is: 90.275
Training candidate program (12/25) Start(Or(Window5Avg(AccelerationSelect()), Window5Avg(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 10.008
Total time elapsed is: 100.284
Training candidate program (13/25) Start(Or(Window5Avg(RelativeSocialAngleSelect()), Window5Avg(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 9.969
Total time elapsed is: 110.253
Training candidate program (14/25) Start(Or(Window5Avg(AxisRatioSelect()), Window5Avg(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 9.817
Total time elapsed is: 120.071
Training candidate program (15/25) Start(Or(Window5Avg(OverlapBboxesSelect()), Window5Avg(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 9.857
Total time elapsed is: 129.928
Training candidate program (16/25) Start(Or(Window5Avg(MinResNoseKeypointDistanceSelect()), Window5Avg(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 9.993
Total time elapsed is: 139.921
Training candidate program (17/25) Start(Or(Window5Avg(ResAngleHeadBodySelect()), Window5Avg(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 11.442
Total time elapsed is: 151.364
Training candidate program (18/25) Start(Or(Window5Avg(SpeedSelect()), Window5Avg(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 10.528
Total time elapsed is: 161.893
Training candidate program (19/25) Start(Or(Window5Avg(TangentialVelocitySelect()), Window5Avg(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 10.662
Total time elapsed is: 172.555
Training candidate program (20/25) Start(Or(Window5Avg(AccelerationSelect()), Window5Avg(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 10.314
Total time elapsed is: 182.870
Training candidate program (21/25) Start(Or(Window5Avg(RelativeSocialAngleSelect()), Window5Avg(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 9.948
Total time elapsed is: 192.818
Training candidate program (22/25) Start(Or(Window5Avg(AxisRatioSelect()), Window5Avg(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 9.753
Total time elapsed is: 202.571
Training candidate program (23/25) Start(Or(Window5Avg(OverlapBboxesSelect()), Window5Avg(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 9.788
Total time elapsed is: 212.359
Training candidate program (24/25) Start(Or(Window5Avg(MinResNoseKeypointDistanceSelect()), Window5Avg(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 9.823
Total time elapsed is: 222.183
Training candidate program (25/25) Start(Or(Window5Avg(ResAngleHeadBodySelect()), Window5Avg(TangentialVelocitySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 9.782
Total time elapsed is: 231.965
Total time elapsed is 231.988


BEST programs found:
Start(Or(Window5Avg(MinResNoseKeypointDistanceSelect()), Window5Avg(SpeedSelect())))
struct_cost 0.0400 | score 0.9794 | path_cost 1.0194 | time 222.1832
Start(Window5Avg(AccelerationSelect()))
struct_cost 0.0100 | score 0.7714 | path_cost 0.7814 | time 28.7419
Start(Or(Window5Avg(RelativeSocialAngleSelect()), Window5Avg(SpeedSelect())))
struct_cost 0.0400 | score 0.6744 | path_cost 0.7144 | time 192.8180
Start(Window5Avg(SpeedSelect()))
struct_cost 0.0100 | score 0.6966 | path_cost 0.7066 | time 14.3140
Start(Or(Window5Avg(AxisRatioSelect()), Window5Avg(SpeedSelect())))
struct_cost 0.0400 | score 0.6422 | path_cost 0.6822 | time 202.5713
Start(Window5Avg(ResAngleHeadBodySelect()))
struct_cost 0.0100 | score 0.6675 | path_cost 0.6775 | time 7.2025
Start(Or(Window5Avg(SpeedSelect()), Window5Avg(SpeedSelect())))
struct_cost 0.0400 | score 0.6038 | path_cost 0.6438 | time 161.8927
Start(Or(Window5Avg(ResAngleHeadBodySelect()), Window5Avg(SpeedSelect())))
struct_cost 0.0400 | score 0.6017 | path_cost 0.6417 | time 151.3642
Start(Or(Window5Avg(AccelerationSelect()), Window5Avg(SpeedSelect())))
struct_cost 0.0400 | score 0.5969 | path_cost 0.6369 | time 182.8698
Start(Or(Window5Avg(ResAngleHeadBodySelect()), Window5Avg(TangentialVelocitySelect())))
struct_cost 0.0400 | score 0.5902 | path_cost 0.6302 | time 231.9654
Start(Or(Window5Avg(OverlapBboxesSelect()), Window5Avg(SpeedSelect())))
struct_cost 0.0400 | score 0.5848 | path_cost 0.6248 | time 212.3595
Start(Or(Window5Avg(TangentialVelocitySelect()), Window5Avg(SpeedSelect())))
struct_cost 0.0400 | score 0.5834 | path_cost 0.6234 | time 172.5555
Start(Or(Window5Avg(AccelerationSelect()), Window5Avg(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.5833 | path_cost 0.6233 | time 100.2838
Start(Or(Window5Avg(SpeedSelect()), Window5Avg(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.5827 | path_cost 0.6227 | time 80.2791
Start(Or(Window5Avg(OverlapBboxesSelect()), Window5Avg(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.5742 | path_cost 0.6142 | time 129.9283
Start(Or(Window5Avg(AxisRatioSelect()), Window5Avg(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.5723 | path_cost 0.6123 | time 120.0708
Start(Or(Window5Avg(TangentialVelocitySelect()), Window5Avg(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.5706 | path_cost 0.6106 | time 90.2751
Start(Or(Window5Avg(RelativeSocialAngleSelect()), Window5Avg(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.5704 | path_cost 0.6104 | time 110.2533
Start(Or(Window5Avg(MinResNoseKeypointDistanceSelect()), Window5Avg(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.5703 | path_cost 0.6103 | time 139.9214
Start(Or(Window5Avg(ResAngleHeadBodySelect()), Window5Avg(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.5656 | path_cost 0.6056 | time 70.3161
Start(Window5Avg(RelativeSocialAngleSelect()))
struct_cost 0.0100 | score 0.5932 | path_cost 0.6032 | time 37.4196
Start(Window5Avg(TangentialVelocitySelect()))
struct_cost 0.0100 | score 0.5733 | path_cost 0.5833 | time 21.4933
Start(Window5Avg(MinResNoseKeypointDistanceSelect()))
struct_cost 0.0100 | score 0.5729 | path_cost 0.5829 | time 60.2272
Start(Window5Avg(AxisRatioSelect()))
struct_cost 0.0100 | score 0.5693 | path_cost 0.5793 | time 44.8968
Start(Window5Avg(OverlapBboxesSelect()))
struct_cost 0.0100 | score 0.3217 | path_cost 0.3317 | time 53.0878


Evaluating program Start(Window5Avg(OverlapBboxesSelect())) on TEST SET
F1 score achieved is 0.5735
Additional performance parameters: {'hamming_accuracy': 0.6826666666666666, 'all_f1s': array([0.74734607, 0.5734767 ])}

ALGORITHM END 

Enumeration: Morlet DSL

We now switch to using a DSL with a more powerful temporal filter. Optional: see DSL_DICT_MOR and MorletFilterOp in dsl.ipynb for the implementation of the filter. Instead of only taking the running average across a set window, the Morlet filter is a learnable temporal filter parametrized by s and w, where s controls frequency and w controls filter width:

Do you observe anything common or different across program structures for the best and worst programs using the Morlet Filter? How does the performance compare to the base DSL?

!yes | python train.py \
--algorithm enumeration \
--exp_name investigation_morlet \
--trial 1 \
--seed 1 \
--dsl_str "morlet" \
--train_data "data/calms21_task1/train_data.npy" \
--test_data "data/calms21_task1/test_data.npy" \
--valid_data "data/calms21_task1/val_data.npy" \
--train_labels "data/calms21_task1/train_investigation_labels.npy" \
--test_labels "data/calms21_task1/test_investigation_labels.npy" \
--valid_labels "data/calms21_task1/val_investigation_labels.npy" \
--input_type "list" \
--output_type "atom" \
--input_size 18 \
--output_size 1 \
--num_labels 1 \
--lossfxn "bcelogits" \
--learning_rate 0.0001 \
--symbolic_epochs 12 \
--max_num_programs 25 \
--class_weights "2.0"

Seed is 1
Experiment log and results saved at: results/investigation_morlet_enumeration_sd_1_001

DEBUG: starting enumerative synthesis with depth 1
DEBUG: 0 programs found
DEBUG: starting enumerative synthesis with depth 2
DEBUG: 8 programs found
DEBUG: starting enumerative synthesis with depth 3
DEBUG: 8 programs found
DEBUG: starting enumerative synthesis with depth 4
DEBUG: 8 programs found
DEBUG: starting enumerative synthesis with depth 5
DEBUG: 648 programs found
Symbolic Synthesis: generated 648/25 symbolic programs from candidate program.
Training candidate program (1/25) Start(MorletFilterOp(ResAngleHeadBodySelect()))
/content/drive/MyDrive/neurosymbolic_summer_school_notebooks/code_and_data_4/near.py:222: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:210.)
  batch_input = [torch.tensor(traj) for traj in batch]
Structural cost is 0.01 with structural penalty 0.01
Time to train child 5.548
Total time elapsed is: 5.548
New tentative BEST program found:
Start(MorletFilterOp(ResAngleHeadBodySelect()))
struct_cost 0.0100 | score 0.6457 | path_cost 0.6557 | time 5.5482
Training candidate program (2/25) Start(MorletFilterOp(SpeedSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 5.548
Total time elapsed is: 11.099
Training candidate program (3/25) Start(MorletFilterOp(TangentialVelocitySelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 5.547
Total time elapsed is: 16.646
New tentative BEST program found:
Start(MorletFilterOp(TangentialVelocitySelect()))
struct_cost 0.0100 | score 0.6231 | path_cost 0.6331 | time 16.6464
Training candidate program (4/25) Start(MorletFilterOp(AccelerationSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 5.592
Total time elapsed is: 22.241
Training candidate program (5/25) Start(MorletFilterOp(RelativeSocialAngleSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 5.504
Total time elapsed is: 27.745
Training candidate program (6/25) Start(MorletFilterOp(AxisRatioSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 5.530
Total time elapsed is: 33.275
New tentative BEST program found:
Start(MorletFilterOp(AxisRatioSelect()))
struct_cost 0.0100 | score 0.5677 | path_cost 0.5777 | time 33.2754
Training candidate program (7/25) Start(MorletFilterOp(OverlapBboxesSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 5.508
Total time elapsed is: 38.787
New tentative BEST program found:
Start(MorletFilterOp(OverlapBboxesSelect()))
struct_cost 0.0100 | score 0.2782 | path_cost 0.2882 | time 38.7873
Training candidate program (8/25) Start(MorletFilterOp(MinResNoseKeypointDistanceSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 5.490
Total time elapsed is: 44.280
Training candidate program (9/25) Start(Or(MorletFilterOp(ResAngleHeadBodySelect()), MorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 9.105
Total time elapsed is: 53.385
Training candidate program (10/25) Start(Or(MorletFilterOp(SpeedSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.813
Total time elapsed is: 60.198
Training candidate program (11/25) Start(Or(MorletFilterOp(TangentialVelocitySelect()), MorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 7.735
Total time elapsed is: 67.934
Training candidate program (12/25) Start(Or(MorletFilterOp(AccelerationSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.895
Total time elapsed is: 74.829
Training candidate program (13/25) Start(Or(MorletFilterOp(RelativeSocialAngleSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.974
Total time elapsed is: 81.804
Training candidate program (14/25) Start(Or(MorletFilterOp(AxisRatioSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.958
Total time elapsed is: 88.763
Training candidate program (15/25) Start(Or(MorletFilterOp(OverlapBboxesSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.946
Total time elapsed is: 95.709
Training candidate program (16/25) Start(Or(MorletFilterOp(MinResNoseKeypointDistanceSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.850
Total time elapsed is: 102.559
Training candidate program (17/25) Start(Or(MorletFilterOp(ResAngleHeadBodySelect()), MorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 7.026
Total time elapsed is: 109.586
Training candidate program (18/25) Start(Or(MorletFilterOp(SpeedSelect()), MorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.930
Total time elapsed is: 116.517
Training candidate program (19/25) Start(Or(MorletFilterOp(TangentialVelocitySelect()), MorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.810
Total time elapsed is: 123.327
Training candidate program (20/25) Start(Or(MorletFilterOp(AccelerationSelect()), MorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.752
Total time elapsed is: 130.079
Training candidate program (21/25) Start(Or(MorletFilterOp(RelativeSocialAngleSelect()), MorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.787
Total time elapsed is: 136.866
Training candidate program (22/25) Start(Or(MorletFilterOp(AxisRatioSelect()), MorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.796
Total time elapsed is: 143.662
Training candidate program (23/25) Start(Or(MorletFilterOp(OverlapBboxesSelect()), MorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.795
Total time elapsed is: 150.457
Training candidate program (24/25) Start(Or(MorletFilterOp(MinResNoseKeypointDistanceSelect()), MorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.853
Total time elapsed is: 157.311
Training candidate program (25/25) Start(Or(MorletFilterOp(ResAngleHeadBodySelect()), MorletFilterOp(TangentialVelocitySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.835
Total time elapsed is: 164.147
Total time elapsed is 164.188


BEST programs found:
Start(Or(MorletFilterOp(MinResNoseKeypointDistanceSelect()), MorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.9971 | path_cost 1.0371 | time 157.3111
Start(MorletFilterOp(MinResNoseKeypointDistanceSelect()))
struct_cost 0.0100 | score 0.7699 | path_cost 0.7799 | time 44.2800
Start(Or(MorletFilterOp(AxisRatioSelect()), MorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.7066 | path_cost 0.7466 | time 143.6626
Start(MorletFilterOp(SpeedSelect()))
struct_cost 0.0100 | score 0.7092 | path_cost 0.7192 | time 11.0989
Start(Or(MorletFilterOp(OverlapBboxesSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.6600 | path_cost 0.7000 | time 95.7087
Start(Or(MorletFilterOp(RelativeSocialAngleSelect()), MorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.6369 | path_cost 0.6769 | time 136.8659
Start(MorletFilterOp(AccelerationSelect()))
struct_cost 0.0100 | score 0.6623 | path_cost 0.6723 | time 22.2411
Start(Or(MorletFilterOp(ResAngleHeadBodySelect()), MorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.6265 | path_cost 0.6665 | time 109.5861
Start(MorletFilterOp(ResAngleHeadBodySelect()))
struct_cost 0.0100 | score 0.6457 | path_cost 0.6557 | time 5.5482
Start(Or(MorletFilterOp(RelativeSocialAngleSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.6151 | path_cost 0.6551 | time 81.8042
Start(MorletFilterOp(RelativeSocialAngleSelect()))
struct_cost 0.0100 | score 0.6424 | path_cost 0.6524 | time 27.7451
Start(Or(MorletFilterOp(MinResNoseKeypointDistanceSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.6029 | path_cost 0.6429 | time 102.5594
Start(Or(MorletFilterOp(TangentialVelocitySelect()), MorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.6015 | path_cost 0.6415 | time 67.9343
Start(Or(MorletFilterOp(AccelerationSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.5971 | path_cost 0.6371 | time 74.8295
Start(MorletFilterOp(TangentialVelocitySelect()))
struct_cost 0.0100 | score 0.6231 | path_cost 0.6331 | time 16.6464
Start(Or(MorletFilterOp(AxisRatioSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.5916 | path_cost 0.6316 | time 88.7627
Start(Or(MorletFilterOp(SpeedSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.5863 | path_cost 0.6263 | time 60.1986
Start(Or(MorletFilterOp(ResAngleHeadBodySelect()), MorletFilterOp(TangentialVelocitySelect())))
struct_cost 0.0400 | score 0.5846 | path_cost 0.6246 | time 164.1466
Start(Or(MorletFilterOp(AccelerationSelect()), MorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.5827 | path_cost 0.6227 | time 130.0791
Start(Or(MorletFilterOp(SpeedSelect()), MorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.5817 | path_cost 0.6217 | time 116.5166
Start(Or(MorletFilterOp(TangentialVelocitySelect()), MorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.5799 | path_cost 0.6199 | time 123.3268
Start(Or(MorletFilterOp(OverlapBboxesSelect()), MorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.5639 | path_cost 0.6039 | time 150.4574
Start(Or(MorletFilterOp(ResAngleHeadBodySelect()), MorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.5482 | path_cost 0.5882 | time 53.3853
Start(MorletFilterOp(AxisRatioSelect()))
struct_cost 0.0100 | score 0.5677 | path_cost 0.5777 | time 33.2754
Start(MorletFilterOp(OverlapBboxesSelect()))
struct_cost 0.0100 | score 0.2782 | path_cost 0.2882 | time 38.7873


Evaluating program Start(MorletFilterOp(OverlapBboxesSelect())) on TEST SET
F1 score achieved is 0.6272
Additional performance parameters: {'hamming_accuracy': 0.74, 'all_f1s': array([0.80040942, 0.62715105])}

ALGORITHM END 

Enumeration with Neural Modules in DSL

So far, the base and Morlet filter DSLs we have used do not contain any neural modules. The code below trains neurosymbolic programs by using DSL_DICT_NEUROSYM from dsl.ipynb. This DSL include NeuralFeatureSelectionFunction, which trains a fully connected neural network over the features.

How does the program found by enumeration compare to the previous ones using DSLs with feature selects?

Optional Exercise: modify the architecture of NeuralFeatureSelectionFunction in dsl.ipynb to see if your can improve the model performance - it is currently a 2 layer fully connected neural network. Also you can modify DSL_DICT_NEUROSYM with different feature selects but note that you should not change the DSL dictionary name from DSL_DICT_NEUROSYM.

!yes | python train.py \
--algorithm enumeration \
--exp_name investigation_neurosym \
--trial 1 \
--seed 1 \
--dsl_str "neurosym" \
--train_data "data/calms21_task1/train_data.npy" \
--test_data "data/calms21_task1/test_data.npy" \
--valid_data "data/calms21_task1/val_data.npy" \
--train_labels "data/calms21_task1/train_investigation_labels.npy" \
--test_labels "data/calms21_task1/test_investigation_labels.npy" \
--valid_labels "data/calms21_task1/val_investigation_labels.npy" \
--input_type "list" \
--output_type "atom" \
--input_size 18 \
--output_size 1 \
--num_labels 1 \
--lossfxn "bcelogits" \
--learning_rate 0.0001 \
--symbolic_epochs 12 \
--max_num_programs 25 \
--class_weights "2.0"

Result program found in save path, overwrite log and program? [y/n]Seed is 1
Experiment log and results saved at: results/investigation_neurosym_enumeration_sd_1_001

DEBUG: starting enumerative synthesis with depth 1
DEBUG: 0 programs found
DEBUG: starting enumerative synthesis with depth 2
DEBUG: 9 programs found
DEBUG: starting enumerative synthesis with depth 3
DEBUG: 9 programs found
DEBUG: starting enumerative synthesis with depth 4
DEBUG: 9 programs found
DEBUG: starting enumerative synthesis with depth 5
DEBUG: 900 programs found
Symbolic Synthesis: generated 900/25 symbolic programs from candidate program.
Training candidate program (1/25) Start(MorletFilterOp(ResAngleHeadBodySelect()))
/content/drive/MyDrive/neurosymbolic_summer_school_notebooks/code_and_data_4/near.py:222: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:210.)
  batch_input = [torch.tensor(traj) for traj in batch]
Structural cost is 0.01 with structural penalty 0.01
Time to train child 5.433
Total time elapsed is: 5.433
New tentative BEST program found:
Start(MorletFilterOp(ResAngleHeadBodySelect()))
struct_cost 0.0100 | score 0.7202 | path_cost 0.7302 | time 5.4333
Training candidate program (2/25) Start(MorletFilterOp(SpeedSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 5.395
Total time elapsed is: 10.830
New tentative BEST program found:
Start(MorletFilterOp(SpeedSelect()))
struct_cost 0.0100 | score 0.5675 | path_cost 0.5775 | time 10.8305
Training candidate program (3/25) Start(MorletFilterOp(TangentialVelocitySelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 5.417
Total time elapsed is: 16.250
Training candidate program (4/25) Start(MorletFilterOp(AccelerationSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 5.386
Total time elapsed is: 21.637
Training candidate program (5/25) Start(MorletFilterOp(RelativeSocialAngleSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 5.445
Total time elapsed is: 27.082
Training candidate program (6/25) Start(MorletFilterOp(AxisRatioSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 5.425
Total time elapsed is: 32.508
Training candidate program (7/25) Start(MorletFilterOp(OverlapBboxesSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 5.458
Total time elapsed is: 37.967
Training candidate program (8/25) Start(MorletFilterOp(MinResNoseKeypointDistanceSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 5.479
Total time elapsed is: 43.445
New tentative BEST program found:
Start(MorletFilterOp(MinResNoseKeypointDistanceSelect()))
struct_cost 0.0100 | score 0.4039 | path_cost 0.4139 | time 43.4455
Training candidate program (9/25) Start(MorletFilterOp(NeuralFeatureSelection()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 5.782
Total time elapsed is: 49.230
New tentative BEST program found:
Start(MorletFilterOp(NeuralFeatureSelection()))
struct_cost 0.0100 | score 0.3000 | path_cost 0.3100 | time 49.2304
Training candidate program (10/25) Start(Or(MorletFilterOp(ResAngleHeadBodySelect()), MorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.856
Total time elapsed is: 56.089
Training candidate program (11/25) Start(Or(MorletFilterOp(SpeedSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.822
Total time elapsed is: 62.911
Training candidate program (12/25) Start(Or(MorletFilterOp(TangentialVelocitySelect()), MorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.928
Total time elapsed is: 69.839
Training candidate program (13/25) Start(Or(MorletFilterOp(AccelerationSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.728
Total time elapsed is: 76.568
Training candidate program (14/25) Start(Or(MorletFilterOp(RelativeSocialAngleSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.752
Total time elapsed is: 83.320
Training candidate program (15/25) Start(Or(MorletFilterOp(AxisRatioSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.799
Total time elapsed is: 90.119
Training candidate program (16/25) Start(Or(MorletFilterOp(OverlapBboxesSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.753
Total time elapsed is: 96.872
Training candidate program (17/25) Start(Or(MorletFilterOp(MinResNoseKeypointDistanceSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.697
Total time elapsed is: 103.569
Training candidate program (18/25) Start(Or(MorletFilterOp(NeuralFeatureSelection()), MorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 7.069
Total time elapsed is: 110.638
Training candidate program (19/25) Start(Or(MorletFilterOp(ResAngleHeadBodySelect()), MorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.847
Total time elapsed is: 117.485
Training candidate program (20/25) Start(Or(MorletFilterOp(SpeedSelect()), MorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.856
Total time elapsed is: 124.342
Training candidate program (21/25) Start(Or(MorletFilterOp(TangentialVelocitySelect()), MorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.780
Total time elapsed is: 131.123
Training candidate program (22/25) Start(Or(MorletFilterOp(AccelerationSelect()), MorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.807
Total time elapsed is: 137.930
Training candidate program (23/25) Start(Or(MorletFilterOp(RelativeSocialAngleSelect()), MorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.813
Total time elapsed is: 144.744
Training candidate program (24/25) Start(Or(MorletFilterOp(AxisRatioSelect()), MorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.761
Total time elapsed is: 151.505
Training candidate program (25/25) Start(Or(MorletFilterOp(OverlapBboxesSelect()), MorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 6.766
Total time elapsed is: 158.271
Total time elapsed is 158.313


BEST programs found:
Start(MorletFilterOp(AccelerationSelect()))
struct_cost 0.0100 | score 0.9856 | path_cost 0.9956 | time 21.6371
Start(Or(MorletFilterOp(OverlapBboxesSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.9218 | path_cost 0.9618 | time 96.8721
Start(Or(MorletFilterOp(MinResNoseKeypointDistanceSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.9114 | path_cost 0.9514 | time 103.5693
Start(Or(MorletFilterOp(ResAngleHeadBodySelect()), MorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.8473 | path_cost 0.8873 | time 56.0887
Start(MorletFilterOp(AxisRatioSelect()))
struct_cost 0.0100 | score 0.7663 | path_cost 0.7763 | time 32.5080
Start(MorletFilterOp(ResAngleHeadBodySelect()))
struct_cost 0.0100 | score 0.7202 | path_cost 0.7302 | time 5.4333
Start(MorletFilterOp(TangentialVelocitySelect()))
struct_cost 0.0100 | score 0.7050 | path_cost 0.7150 | time 16.2505
Start(MorletFilterOp(OverlapBboxesSelect()))
struct_cost 0.0100 | score 0.6866 | path_cost 0.6966 | time 37.9666
Start(Or(MorletFilterOp(AxisRatioSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.6280 | path_cost 0.6680 | time 90.1191
Start(Or(MorletFilterOp(AccelerationSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.6107 | path_cost 0.6507 | time 76.5678
Start(Or(MorletFilterOp(TangentialVelocitySelect()), MorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.6055 | path_cost 0.6455 | time 131.1229
Start(MorletFilterOp(RelativeSocialAngleSelect()))
struct_cost 0.0100 | score 0.6203 | path_cost 0.6303 | time 27.0823
Start(Or(MorletFilterOp(SpeedSelect()), MorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.5856 | path_cost 0.6256 | time 124.3422
Start(Or(MorletFilterOp(RelativeSocialAngleSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.5772 | path_cost 0.6172 | time 83.3199
Start(Or(MorletFilterOp(SpeedSelect()), MorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.5691 | path_cost 0.6091 | time 62.9110
Start(Or(MorletFilterOp(AccelerationSelect()), MorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.5575 | path_cost 0.5975 | time 137.9302
Start(Or(MorletFilterOp(RelativeSocialAngleSelect()), MorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.5573 | path_cost 0.5973 | time 144.7438
Start(Or(MorletFilterOp(TangentialVelocitySelect()), MorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.5507 | path_cost 0.5907 | time 69.8395
Start(Or(MorletFilterOp(AxisRatioSelect()), MorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.5416 | path_cost 0.5816 | time 151.5048
Start(MorletFilterOp(SpeedSelect()))
struct_cost 0.0100 | score 0.5675 | path_cost 0.5775 | time 10.8305
Start(Or(MorletFilterOp(ResAngleHeadBodySelect()), MorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.5236 | path_cost 0.5636 | time 117.4856
Start(Or(MorletFilterOp(OverlapBboxesSelect()), MorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.4358 | path_cost 0.4758 | time 158.2710
Start(MorletFilterOp(MinResNoseKeypointDistanceSelect()))
struct_cost 0.0100 | score 0.4039 | path_cost 0.4139 | time 43.4455
Start(Or(MorletFilterOp(NeuralFeatureSelection()), MorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.3171 | path_cost 0.3571 | time 110.6382
Start(MorletFilterOp(NeuralFeatureSelection()))
struct_cost 0.0100 | score 0.3000 | path_cost 0.3100 | time 49.2304


Evaluating program Start(MorletFilterOp(NeuralFeatureSelection())) on TEST SET
F1 score achieved is 0.6750
Additional performance parameters: {'hamming_accuracy': 0.853, 'all_f1s': array([0.90501831, 0.67501842])}

ALGORITHM END 

Visualizing Runtime vs. Classification Performance

To evaluate your enumeration runs, we can take the saved log files in code_and_data/results and plot the total runtime against F1 score.

Optional Exercise: Some hyperparameters that affect runtime and F1 score performance are listed below. Try changing a few of these and save a performance plot of your results!

• max_num_programs : maximum number of programs to optimize for enumeration

• symbolic_epochs : the number of training epochs for updating program parameters

• You can modify the search depth with min_enum_depth and max_depth. Note that if the depth is too large, enumerating through all the programs will take a long time.

Also remember to update exp_name and trial to prevent overwriting previous runs.

# Utility functions for visualizing run time vs accuracy

def parse_runtime_f1_from_logs(log_files):

  runtime = []
  f1 = []

  runtime_key = 'Total time elapsed is:'
  f1_key = 'F1 score achieved is'

  for item in log_files:
    # If there's a list of list of files corresponding to different random seeds,
    # we take the average
    if len(item[0]) > 1:
      seed_runtime = []
      seed_f1 = []
      for seed in item:
        with open(os.path.join('results', seed, 'log.txt')) as f:
            lines = f.readlines()

            curr_runtimes = []
            for line in lines:
              if runtime_key in line:
                curr_runtimes.append(float(line.split(runtime_key)[-1].strip()))
            
              if f1_key in line:
                seed_f1.append(float(line.split(f1_key)[-1].strip()))
            seed_runtime.append(curr_runtimes[-1])
      runtime.append(np.mean(seed_runtime))
      f1.append(np.mean(seed_f1))        
    else:
      # There's only 1 seed per run
      with open(os.path.join('results', item, 'log.txt')) as f:
        lines = f.readlines()

      curr_runtimes = []
      for line in lines:
        if runtime_key in line:
          curr_runtimes.append(float(line.split(runtime_key)[-1].strip()))
      
        if f1_key in line:
          f1.append(float(line.split(f1_key)[-1].strip()))
      runtime.append(curr_runtimes[-1])

  return runtime, f1

def plot_runtime_f1(runtime, f1, labels):
  assert(len(runtime) == len(f1) == len(labels))

  fig = plt.figure()
  for i, item in enumerate(labels):
    if len(item[0]) > 1:
      item = item[0]
    plt.scatter(runtime[i], f1[i], label = item.split('_sd')[0])

  plt.xlim([10, 400])
  plt.ylim([0.3, 0.75])  
  plt.xlabel("Runtime (s)")
  plt.ylabel("F1 score")  
  plt.legend(loc='upper center', bbox_to_anchor=(0.5, 1 + 0.1*len(labels)))

# Directory names to plot inside near_code/results
run_names_to_plot = ['investigation_base_enumeration_sd_1_001', 
                     'investigation_morlet_enumeration_sd_1_001',
                     'investigation_neurosym_enumeration_sd_1_001']

runtime, f1 = parse_runtime_f1_from_logs(run_names_to_plot)

plot_runtime_f1(runtime, f1, run_names_to_plot)

Note: The code below will not work without running with seed = 2 for enumeration using the Morlet filter.

We include an example of how to plot using multiple random seeds - the code will average over all entries in the list of lists.

# Example of plotting with multiple random seeds
# Directory names to plot inside near_code/results
run_names_to_plot = ['investigation_base_enumeration_sd_1_001', 
                     ['investigation_morlet_enumeration_sd_1_001',
                      'investigation_morlet_enumeration_sd_2_001']]

runtime, f1 = parse_runtime_f1_from_logs(run_names_to_plot)

plot_runtime_f1(runtime, f1, run_names_to_plot)

Printing saved programs

You can use the example below to print your saved programs in results.

import pickle
from near import print_program

# Change the program name here if you want to print other programs
saved_program = pickle.load(open('results/investigation_morlet_enumeration_sd_1_001/program.p', 'rb'))
print(print_program(saved_program, ignore_constants = False))

[Optional] Implement your own temporal filter

We provide an implementation for the symmetric Morlet filter (filter shape is symmetric around time 0, the current frame), which you used earlier for enumeration.

Implement get_filter_default_xvals for AsymmetricFilterOp in dsl.ipynb so that you can also use AsymMorletFilterOp in your DSL. You can use the get_filter_default_xvals in SymmetricFilterOp as a reference. The Asymmetric Morlet filter is similar to the symmetric one, except it has two sets of parameters for s and w, one for left, which corresponds to timestamps before the current time, and one for right, which corresponds to timestamps after the current time. The filter is thus asymmetrical with different filter shapes before and after the current time.

Hint: How should get_filter_default_xvals call get_filter to implement the asymmetry?

Important: after modifying dsl.ipynb, you will need to re-convert your new dsl to python files, included in the cell below.

Run enumeration with your new DSL with asymmetric Morlet filter in addition to the symmetric Morlet and record your results. What happened to the search space after adding your own filter? Was your filter found by enumeration and did it improve performance?

!jupyter nbconvert --to python dsl.ipynb
!jupyter nbconvert --to python near.ipynb

[NbConvertApp] Converting notebook dsl.ipynb to python
[NbConvertApp] Writing 41826 bytes to dsl.py
[NbConvertApp] Converting notebook near.ipynb to python
[NbConvertApp] Writing 39669 bytes to near.py

!yes | python train.py \
--algorithm enumeration \
--exp_name investigation_asym_morlet \
--trial 1 \
--seed 1 \
--dsl_str "asym_morlet" \
--train_data "data/calms21_task1/train_data.npy" \
--test_data "data/calms21_task1/test_data.npy" \
--valid_data "data/calms21_task1/val_data.npy" \
--train_labels "data/calms21_task1/train_investigation_labels.npy" \
--test_labels "data/calms21_task1/test_investigation_labels.npy" \
--valid_labels "data/calms21_task1/val_investigation_labels.npy" \
--input_type "list" \
--output_type "atom" \
--input_size 18 \
--output_size 1 \
--num_labels 1 \
--lossfxn "bcelogits" \
--learning_rate 0.0001 \
--symbolic_epochs 12 \
--max_num_programs 25 \
--class_weights "2.0"

Seed is 1
Experiment log and results saved at: results/investigation_asym_morlet_enumeration_sd_1_001

DEBUG: starting enumerative synthesis with depth 1
DEBUG: 0 programs found
DEBUG: starting enumerative synthesis with depth 2
DEBUG: 8 programs found
DEBUG: starting enumerative synthesis with depth 3
DEBUG: 8 programs found
DEBUG: starting enumerative synthesis with depth 4
DEBUG: 8 programs found
DEBUG: starting enumerative synthesis with depth 5
DEBUG: 648 programs found
Symbolic Synthesis: generated 648/25 symbolic programs from candidate program.
Training candidate program (1/25) Start(AsymMorletFilterOp(ResAngleHeadBodySelect()))
/content/drive/MyDrive/neurosymbolic_summer_school_notebooks/code_and_data_4/near.py:222: UserWarning: Creating a tensor from a list of numpy.ndarrays is extremely slow. Please consider converting the list to a single numpy.ndarray with numpy.array() before converting to a tensor. (Triggered internally at  ../torch/csrc/utils/tensor_new.cpp:210.)
  batch_input = [torch.tensor(traj) for traj in batch]
Structural cost is 0.01 with structural penalty 0.01
Time to train child 6.255
Total time elapsed is: 6.255
New tentative BEST program found:
Start(AsymMorletFilterOp(ResAngleHeadBodySelect()))
struct_cost 0.0100 | score 0.6471 | path_cost 0.6571 | time 6.2550
Training candidate program (2/25) Start(AsymMorletFilterOp(SpeedSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 6.153
Total time elapsed is: 12.412
Training candidate program (3/25) Start(AsymMorletFilterOp(TangentialVelocitySelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 6.123
Total time elapsed is: 18.535
New tentative BEST program found:
Start(AsymMorletFilterOp(TangentialVelocitySelect()))
struct_cost 0.0100 | score 0.6416 | path_cost 0.6516 | time 18.5352
Training candidate program (4/25) Start(AsymMorletFilterOp(AccelerationSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 6.174
Total time elapsed is: 24.713
Training candidate program (5/25) Start(AsymMorletFilterOp(RelativeSocialAngleSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 6.165
Total time elapsed is: 30.878
Training candidate program (6/25) Start(AsymMorletFilterOp(AxisRatioSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 6.209
Total time elapsed is: 37.087
New tentative BEST program found:
Start(AsymMorletFilterOp(AxisRatioSelect()))
struct_cost 0.0100 | score 0.5831 | path_cost 0.5931 | time 37.0872
Training candidate program (7/25) Start(AsymMorletFilterOp(OverlapBboxesSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 6.148
Total time elapsed is: 43.239
New tentative BEST program found:
Start(AsymMorletFilterOp(OverlapBboxesSelect()))
struct_cost 0.0100 | score 0.2869 | path_cost 0.2969 | time 43.2388
Training candidate program (8/25) Start(AsymMorletFilterOp(MinResNoseKeypointDistanceSelect()))
Structural cost is 0.01 with structural penalty 0.01
Time to train child 6.175
Total time elapsed is: 49.417
Training candidate program (9/25) Start(Or(AsymMorletFilterOp(ResAngleHeadBodySelect()), AsymMorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 8.163
Total time elapsed is: 57.581
Training candidate program (10/25) Start(Or(AsymMorletFilterOp(SpeedSelect()), AsymMorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 8.214
Total time elapsed is: 65.795
Training candidate program (11/25) Start(Or(AsymMorletFilterOp(TangentialVelocitySelect()), AsymMorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 8.106
Total time elapsed is: 73.901
Training candidate program (12/25) Start(Or(AsymMorletFilterOp(AccelerationSelect()), AsymMorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 8.101
Total time elapsed is: 82.003
Training candidate program (13/25) Start(Or(AsymMorletFilterOp(RelativeSocialAngleSelect()), AsymMorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 8.176
Total time elapsed is: 90.179
Training candidate program (14/25) Start(Or(AsymMorletFilterOp(AxisRatioSelect()), AsymMorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 7.984
Total time elapsed is: 98.163
Training candidate program (15/25) Start(Or(AsymMorletFilterOp(OverlapBboxesSelect()), AsymMorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 7.979
Total time elapsed is: 106.143
Training candidate program (16/25) Start(Or(AsymMorletFilterOp(MinResNoseKeypointDistanceSelect()), AsymMorletFilterOp(ResAngleHeadBodySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 7.975
Total time elapsed is: 114.119
Training candidate program (17/25) Start(Or(AsymMorletFilterOp(ResAngleHeadBodySelect()), AsymMorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 8.045
Total time elapsed is: 122.164
Training candidate program (18/25) Start(Or(AsymMorletFilterOp(SpeedSelect()), AsymMorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 8.055
Total time elapsed is: 130.219
Training candidate program (19/25) Start(Or(AsymMorletFilterOp(TangentialVelocitySelect()), AsymMorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 8.071
Total time elapsed is: 138.290
Training candidate program (20/25) Start(Or(AsymMorletFilterOp(AccelerationSelect()), AsymMorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 8.130
Total time elapsed is: 146.420
Training candidate program (21/25) Start(Or(AsymMorletFilterOp(RelativeSocialAngleSelect()), AsymMorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 8.016
Total time elapsed is: 154.437
Training candidate program (22/25) Start(Or(AsymMorletFilterOp(AxisRatioSelect()), AsymMorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 8.007
Total time elapsed is: 162.445
Training candidate program (23/25) Start(Or(AsymMorletFilterOp(OverlapBboxesSelect()), AsymMorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 7.999
Total time elapsed is: 170.444
Training candidate program (24/25) Start(Or(AsymMorletFilterOp(MinResNoseKeypointDistanceSelect()), AsymMorletFilterOp(SpeedSelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 7.989
Total time elapsed is: 178.434
Training candidate program (25/25) Start(Or(AsymMorletFilterOp(ResAngleHeadBodySelect()), AsymMorletFilterOp(TangentialVelocitySelect())))
Structural cost is 0.04 with structural penalty 0.01
Time to train child 8.025
Total time elapsed is: 186.459
Total time elapsed is 186.518


BEST programs found:
Start(Or(AsymMorletFilterOp(MinResNoseKeypointDistanceSelect()), AsymMorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.9971 | path_cost 1.0371 | time 178.4341
Start(AsymMorletFilterOp(MinResNoseKeypointDistanceSelect()))
struct_cost 0.0100 | score 0.9087 | path_cost 0.9187 | time 49.4175
Start(Or(AsymMorletFilterOp(AxisRatioSelect()), AsymMorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.7116 | path_cost 0.7516 | time 162.4449
Start(Or(AsymMorletFilterOp(OverlapBboxesSelect()), AsymMorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.6792 | path_cost 0.7192 | time 106.1430
Start(AsymMorletFilterOp(SpeedSelect()))
struct_cost 0.0100 | score 0.7024 | path_cost 0.7124 | time 12.4121
Start(Or(AsymMorletFilterOp(ResAngleHeadBodySelect()), AsymMorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.6369 | path_cost 0.6769 | time 122.1644
Start(Or(AsymMorletFilterOp(RelativeSocialAngleSelect()), AsymMorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.6359 | path_cost 0.6759 | time 154.4373
Start(AsymMorletFilterOp(RelativeSocialAngleSelect()))
struct_cost 0.0100 | score 0.6505 | path_cost 0.6605 | time 30.8779
Start(AsymMorletFilterOp(AccelerationSelect()))
struct_cost 0.0100 | score 0.6483 | path_cost 0.6583 | time 24.7127
Start(Or(AsymMorletFilterOp(RelativeSocialAngleSelect()), AsymMorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.6179 | path_cost 0.6579 | time 90.1791
Start(AsymMorletFilterOp(ResAngleHeadBodySelect()))
struct_cost 0.0100 | score 0.6471 | path_cost 0.6571 | time 6.2550
Start(AsymMorletFilterOp(TangentialVelocitySelect()))
struct_cost 0.0100 | score 0.6416 | path_cost 0.6516 | time 18.5352
Start(Or(AsymMorletFilterOp(MinResNoseKeypointDistanceSelect()), AsymMorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.6070 | path_cost 0.6470 | time 114.1187
Start(Or(AsymMorletFilterOp(TangentialVelocitySelect()), AsymMorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.6028 | path_cost 0.6428 | time 73.9014
Start(Or(AsymMorletFilterOp(AxisRatioSelect()), AsymMorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.6023 | path_cost 0.6423 | time 98.1633
Start(Or(AsymMorletFilterOp(AccelerationSelect()), AsymMorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.6015 | path_cost 0.6415 | time 82.0029
Start(Or(AsymMorletFilterOp(SpeedSelect()), AsymMorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.5934 | path_cost 0.6334 | time 65.7954
Start(Or(AsymMorletFilterOp(TangentialVelocitySelect()), AsymMorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.5840 | path_cost 0.6240 | time 138.2906
Start(Or(AsymMorletFilterOp(SpeedSelect()), AsymMorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.5838 | path_cost 0.6238 | time 130.2196
Start(Or(AsymMorletFilterOp(ResAngleHeadBodySelect()), AsymMorletFilterOp(TangentialVelocitySelect())))
struct_cost 0.0400 | score 0.5825 | path_cost 0.6225 | time 186.4593
Start(Or(AsymMorletFilterOp(AccelerationSelect()), AsymMorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.5801 | path_cost 0.6201 | time 146.4206
Start(Or(AsymMorletFilterOp(OverlapBboxesSelect()), AsymMorletFilterOp(SpeedSelect())))
struct_cost 0.0400 | score 0.5594 | path_cost 0.5994 | time 170.4443
Start(AsymMorletFilterOp(AxisRatioSelect()))
struct_cost 0.0100 | score 0.5831 | path_cost 0.5931 | time 37.0872
Start(Or(AsymMorletFilterOp(ResAngleHeadBodySelect()), AsymMorletFilterOp(ResAngleHeadBodySelect())))
struct_cost 0.0400 | score 0.5468 | path_cost 0.5868 | time 57.5811
Start(AsymMorletFilterOp(OverlapBboxesSelect()))
struct_cost 0.0100 | score 0.2869 | path_cost 0.2969 | time 43.2388


Evaluating program Start(AsymMorletFilterOp(OverlapBboxesSelect())) on TEST SET
F1 score achieved is 0.6250
Additional performance parameters: {'hamming_accuracy': 0.74, 'all_f1s': array([0.80102041, 0.625     ])}

ALGORITHM END 

[Optional] Open-Ended Exploration

Can you improve the performance of enumerative program search? If you feel you have a good algorithm, please run it using seed 1 (default) and submit your top program to https://forms.gle/peEi6vryaJQejTcn9. Feel free to make any changes to the code! Below are some suggestions:

• Modifying the DSL:

  • Are there DSL functions that are found to work better than others during enumeration? Maybe you also have ideas for new features to add to the DSL. Based on your observations, add your own dsl to dsl.ipynb, and note you need to update train.py to import your DSL.

• Modifying the Search Algorithm:

  • One way to improve runtime is to train each program with a small number of epochs, but keep track of the top N programs, then train only the top N programs to convergence at the very end. To implement this, you need to modify the ENUMERATION class in near.ipynb to sort the program list based on performance, then train the top N programs for an additional number of epochs.

Tomorrow we will go over neurally-guided search to further improve the efficiency of program search.

Acknowledgements: This notebook was developed by Jennifer J. Sun (Caltech) and Atharva Sengal (UT Austin) for the neurosymbolic summer school. The data subset is processed from CalMS21 and the DSL is developed by by Megan Tjandrasuwita (MIT) from her work on Interpreting Expert Annotation Differences in Animal Behavior. Megan’s work is partly based on NEAR by Ameesh Shah (Berkeley) and Eric Zhan (Argo).