How to run the code and replicate the results of an initial policy like PPO?

[zhaozj89]

Hi, thanks very much for sharing this inspiring work!

I have attempted to execute the code to replicate the results using the following scripts:

python train_rl_il.py --env CartPole --algo ppo --steps 1000000

python collect_demo.py --env CartPole --agent-path ./logs/CartPole/ppo/seed0_20240808225352

python train_clf.py --env CartPole --buffer ./demos/CartPole/size20000_std0.1_bias0.1_reward11.pkl

However, the rewards I achieved were significantly lower than those reported in the paper. For instance, here are the training logs from my CartPole environment:

python train_rl_il.py --env CartPole --algo ppo --steps 1000000

> Training with cuda
> No pre-training
> Training PPO...
Num steps: 20000 , Return: 27.9 , Min/Max Return: 20.4 /31.7                    
Num steps: 40000 , Return: 38.3 , Min/Max Return: 27.3 /52.4                    
Num steps: 60000 , Return: 30.4 , Min/Max Return: 25.3 /36.7                    
Num steps: 80000 , Return: 37.1 , Min/Max Return: 29.6 /45.1                    
Num steps: 100000, Return: 19.2 , Min/Max Return: 18.2 /20.8                    
Num steps: 120000, Return: 21.7 , Min/Max Return: 19.2 /26.5                    
Num steps: 140000, Return: 29.6 , Min/Max Return: 24.2 /39.9                    
Num steps: 160000, Return: 38.9 , Min/Max Return: 31.3 /48.5                    
Num steps: 180000, Return: 30.1 , Min/Max Return: 26.5 /34.3                    
Num steps: 200000, Return: 51.7 , Min/Max Return: 32.7 /94.9                    
Num steps: 220000, Return: 43.5 , Min/Max Return: 37.8 /54.0                    
Num steps: 240000, Return: 31.6 , Min/Max Return: 28.3 /33.9                    
Num steps: 260000, Return: 36.2 , Min/Max Return: 29.2 /42.1                    
Num steps: 280000, Return: 30.3 , Min/Max Return: 27.9 /35.6                    
Num steps: 300000, Return: 31.1 , Min/Max Return: 27.5 /36.1                    
Num steps: 320000, Return: 19.7 , Min/Max Return: 18.6 /20.9                    
Num steps: 340000, Return: 46.6 , Min/Max Return: 34.7 /64.5                    
Num steps: 360000, Return: 40.5 , Min/Max Return: 30.2 /53.2                    
Num steps: 380000, Return: 28.8 , Min/Max Return: 24.9 /32.1                    
Num steps: 400000, Return: 48.6 , Min/Max Return: 30.7 /76.2                    
Num steps: 420000, Return: 55.4 , Min/Max Return: 43.2 /78.2                    
Num steps: 440000, Return: 31.7 , Min/Max Return: 29.0 /35.4                    
Num steps: 460000, Return: 39.0 , Min/Max Return: 30.2 /43.9                    
Num steps: 480000, Return: 34.8 , Min/Max Return: 29.2 /41.2                    
Num steps: 500000, Return: 39.7 , Min/Max Return: 34.2 /44.9                    
Num steps: 520000, Return: 50.4 , Min/Max Return: 40.1 /64.2                    
Num steps: 540000, Return: 34.2 , Min/Max Return: 27.9 /43.6                    
Num steps: 560000, Return: 35.3 , Min/Max Return: 29.2 /43.6                    
Num steps: 580000, Return: 43.1 , Min/Max Return: 36.1 /46.3                    
Num steps: 600000, Return: 41.8 , Min/Max Return: 34.1 /49.6                    
Num steps: 620000, Return: 46.7 , Min/Max Return: 34.6 /58.7 
Num steps: 640000, Return: 38.3 , Min/Max Return: 30.2 /41.1                    
Num steps: 660000, Return: 46.7 , Min/Max Return: 39.5 /56.8                    
Num steps: 680000, Return: 41.1 , Min/Max Return: 32.2 /54.3                    
Num steps: 700000, Return: 40.6 , Min/Max Return: 31.7 /53.0                    
Num steps: 720000, Return: 41.5 , Min/Max Return: 30.8 /56.2                    
Num steps: 740000, Return: 35.5 , Min/Max Return: 30.2 /44.1                    
Num steps: 760000, Return: 49.5 , Min/Max Return: 38.3 /69.2                    
Num steps: 780000, Return: 55.8 , Min/Max Return: 42.3 /80.5                    
Num steps: 800000, Return: 59.2 , Min/Max Return: 34.4 /109.7                   
Num steps: 820000, Return: 40.1 , Min/Max Return: 34.6 /54.1                    
Num steps: 840000, Return: 41.6 , Min/Max Return: 28.8 /53.7                    
Num steps: 860000, Return: 40.9 , Min/Max Return: 31.1 /56.4                    
Num steps: 880000, Return: 45.9 , Min/Max Return: 38.9 /50.8                    
Num steps: 900000, Return: 50.6 , Min/Max Return: 33.9 /108.5                   
Num steps: 920000, Return: 56.2 , Min/Max Return: 35.7 /123.4                   
Num steps: 940000, Return: 31.3 , Min/Max Return: 27.7 /35.2                    
Num steps: 960000, Return: 50.9 , Min/Max Return: 35.4 /63.1                    
Num steps: 980000, Return: 38.2 , Min/Max Return: 31.3 /58.3                    
Num steps: 1000000, Return: 45.5 , Min/Max Return: 35.2 /76.3                   
100%|████████████████████████████████| 1000000/1000000 [42:34<00:00, 391.46it/s]
> Done in 2555s

python collect_demo.py --env CartPole --agent-path ./logs/CartPole/ppo/seed0_20240808225352

epi: 0, reward: 11, steps: 20                                                                                                                                                                                      
epi: 1, reward: 21, steps: 73                                                                                                                                                                                      
epi: 2, reward: 8, steps: 33                                                                                                                                                                                       
epi: 3, reward: 10, steps: 50                                                                                                                                                                                      
epi: 4, reward: 17, steps: 27                                                                                                                                                                                      
epi: 5, reward: 6, steps: 15                                                                                                                                                                                       
epi: 6, reward: 22, steps: 52                                                                                                                                                                                      
epi: 7, reward: 10, steps: 24                                                                                                                                                                                      
epi: 8, reward: 11, steps: 18                                                                                                                                                                                      
epi: 9, reward: 3, steps: 9                                                                                                                                                                                        
epi: 10, reward: 17, steps: 35                                                                                                                                                                                     
epi: 11, reward: 15, steps: 32                                                                                                                                                                                     
epi: 12, reward: 10, steps: 18                                                                                                                                                                                     
epi: 13, reward: 7, steps: 17                                                                                                                                                                                      
epi: 14, reward: 4, steps: 11                                                                                                                                                                                      
epi: 15, reward: 14, steps: 30                                                                                                                                                                                     
epi: 16, reward: 11, steps: 21                                                                                                                                                                                     
epi: 17, reward: 8, steps: 17                                                                                                                                                                                      
epi: 18, reward: 12, steps: 24                                                                                                                                                                                     
epi: 19, reward: 10, steps: 56                                                                                                                                                                                     
epi: 20, reward: 13, steps: 28                                                                                                                                                                                     
epi: 21, reward: 6, steps: 12                                                                                                                                                                                      
epi: 22, reward: 14, steps: 26                                                                                                                                                                                     
epi: 23, reward: 11, steps: 17                                                                                                                                                                                     
epi: 24, reward: 12, steps: 23                                                                                                                                                                                     
epi: 25, reward: 18, steps: 39                                                                                                                                                                                     
epi: 26, reward: 11, steps: 25                                                                                                                                                                                     
epi: 27, reward: 8, steps: 52                                                                                                                                                                                      
epi: 28, reward: 17, steps: 32             
...
epi: 666, reward: 19, steps: 73                                                                                                                                                                                    
epi: 667, reward: 9, steps: 19                                                                                                                                                                                     
epi: 668, reward: 6, steps: 12                                                                                                                                                                                     
epi: 669, reward: 4, steps: 9                                                                                                                                                                                      
epi: 670, reward: 11, steps: 65                                                                                                                                                                                    
epi: 671, reward: 9, steps: 19                                                                                                                                                                                     
epi: 672, reward: 11, steps: 29                                                                                                                                                                                    
epi: 673, reward: 7, steps: 15                                                                                                                                                                                     
epi: 674, reward: 13, steps: 25                                                                                                                                                                                    
epi: 675, reward: 6, steps: 13                                                                                                                                                                                     
epi: 676, reward: 12, steps: 37                                                                                                                                                                                    
epi: 677, reward: 14, steps: 24                                                                                                                                                                                    
epi: 678, reward: 9, steps: 30                                                                                                                                                                                     
epi: 679, reward: 17, steps: 35                                                                                                                                                                                    
epi: 680, reward: 11, steps: 22                                                                                                                                                                                    
epi: 681, reward: 15, steps: 46                                                                                                                                                                                    
epi: 682, reward: 6, steps: 34                                                                                                                                                                                     
epi: 683, reward: 24, steps: 45                                                                                                                                                                                    
epi: 684, reward: 18, steps: 52                                                                                                                                                                                    
epi: 685, reward: 14, steps: 27                                                                                                                                                                                    
100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 20000/20000 [01:24<00:00, 236.97it/s]
> buffer saved, mean reward: 11.35, std: 6.36
> video saved

python train_clf.py --env CartPole --buffer ./demos/CartPole/size20000_std0.1_bias0.1_reward11.pkl

> Training with cuda
> Using tuned hyper-parameters
> Training BC policy...
iter: 0, loss: 5.90e-01                                                         
iter: 500, loss: 4.12e-01                                                       
iter: 1000, loss: 3.98e-01                                                      
iter: 1500, loss: 3.48e-01                                                      
iter: 2000, loss: 3.62e-01                                                      
iter: 2500, loss: 4.65e-01                                                      
iter: 3000, loss: 3.31e-01                                                      
iter: 3500, loss: 3.60e-01                                                      
iter: 4000, loss: 3.94e-01                                                      
iter: 4500, loss: 3.55e-01                                                      
iter: 4999, loss: 3.23e-01                                                      
100%|██████████████████████████████████████| 5000/5000 [00:34<00:00, 146.67it/s]
> Done
> Evaluating BC policy
average reward: 20.88, average length: 28.40
> Done
> ----------- Outer iter: 1 -----------
> Training environment model...
iter: 0, loss: 1.11e+01                                                         
iter: 500, loss: 3.52e-01                                                       
iter: 1000, loss: 1.24e+00                                                      
iter: 1500, loss: 4.83e-01                        
...
> ----------- Outer iter: 2 -----------
> Training environment model...
iter: 0, loss: 2.45e-01                                                         
iter: 500, loss: 9.09e-02                                                       
iter: 1000, loss: 1.77e-01                                                      
iter: 1500, loss: 9.78e-02                                                      
iter: 2000, loss: 2.33e-01                                                      
iter: 2500, loss: 2.25e-01                                                      
iter: 3000, loss: 8.33e-02                                                      
iter: 3500, loss: 1.53e-01                                                      
iter: 4000, loss: 1.63e-01                                                      
iter: 4500, loss: 9.96e-02                                                      
iter: 4999, loss: 1.57e-01                                                      
100%|██████████████████████████████████████| 5000/5000 [00:28<00:00, 174.13it/s]
> Done
> Training CLF controller...
iter: 200, loss: 3.63e+01                                                       
iter: 400, loss: 2.67e+01                                                       
iter: 600, loss: 2.71e+01                                                       
iter: 800, loss: 2.68e+01                                                       
iter: 1000, loss: 2.09e+01                                                      
iter: 1200, loss: 1.99e+01                                                      
iter: 1400, loss: 1.89e+01                                                      
iter: 1600, loss: 1.81e+01                                                      
iter: 1800, loss: 2.69e+01                                                      
iter: 1999, loss: 2.06e+01                                                      
100%|███████████████████████████████████████| 1999/1999 [09:43<00:00,  3.42it/s]
> Evaluating policy...
average reward: 26.57, average length: 38.80
> Done
> Collecting demos...
mean reward: 17.36
> Done
> Final demo saved
> Iter time: 637s, total time: 1288s
...
> ----------- Outer iter: 48 -----------
> Training environment model...
iter: 0, loss: 5.08e-03                                                         
iter: 500, loss: 2.13e-03                                                       
iter: 1000, loss: 3.25e-01                                                      
iter: 1500, loss: 7.04e-03                                                      
iter: 2000, loss: 1.06e-02                                                      
iter: 2500, loss: 1.67e-03                                                      
iter: 3000, loss: 5.57e-02                                                      
iter: 3500, loss: 2.24e-03                                                      
iter: 4000, loss: 2.24e-03                                                      
iter: 4500, loss: 1.56e-01                                                      
iter: 4999, loss: 9.13e-03                                                      
100%|██████████████████████████████████████| 5000/5000 [00:23<00:00, 211.62it/s]
> Done
> Training CLF controller...
iter: 200, loss: 5.55e+00                                                       
iter: 400, loss: 5.22e+00                                                       
iter: 600, loss: 6.63e+00                                                       
iter: 800, loss: 5.17e+00                                                       
iter: 1000, loss: 5.09e+00                                                      
iter: 1200, loss: 4.63e+00                                                      
iter: 1400, loss: 8.60e+00                                                      
iter: 1600, loss: 5.35e+00                                                      
iter: 1800, loss: 4.95e+00                                                      
iter: 1999, loss: 5.37e+00                                                      
100%|███████████████████████████████████████| 1999/1999 [01:47<00:00, 18.52it/s]
> Evaluating policy...
average reward: 69.63, average length: 97.40
> Done
> Collecting demos...
mean reward: 34.59
> Done
> Final demo saved
> Iter time: 146s, total time: 10898s
> ----------- Outer iter: 49 -----------
> Training environment model...
iter: 0, loss: 7.01e-02                                                         
iter: 500, loss: 9.32e-02                                                       
iter: 1000, loss: 5.55e-02                                                      
iter: 1500, loss: 7.36e-03                                                      
iter: 2000, loss: 1.15e-02                                                      
iter: 2500, loss: 1.96e-03                                                      
iter: 3000, loss: 1.28e-02                                                      
iter: 3500, loss: 3.23e-03                                                      
iter: 4000, loss: 2.97e-03                                                      
iter: 4500, loss: 1.22e-03                                                      
iter: 4999, loss: 3.33e-03                                                      
100%|██████████████████████████████████████| 5000/5000 [00:24<00:00, 207.89it/s]
> Done
> Training CLF controller...
iter: 200, loss: 5.38e+00                                                       
iter: 400, loss: 6.41e+00                                                       
iter: 600, loss: 5.36e+00                                                       
iter: 800, loss: 5.71e+00                                                       
iter: 1000, loss: 5.38e+00                                                      
iter: 1200, loss: 4.79e+00                                                      
iter: 1400, loss: 5.99e+00                                                      
iter: 1600, loss: 5.13e+00                                                      
iter: 1800, loss: 4.94e+00                                                      
iter: 1999, loss: 5.89e+00                                                      
100%|███████████████████████████████████████| 1999/1999 [01:47<00:00, 18.66it/s]
> Evaluating policy...
average reward: 74.99, average length: 101.80
> Done
> Collecting demos...
mean reward: 34.93
> Done
> Final demo saved
> Iter time: 145s, total time: 11043s

For both the PPO and LYGE policies, the outcomes are substantially lower than those reported in the paper (refer to Figure 5 below).

For the first step of training a PPO initial policy, I have also experimented with other environments, such as InvertedPendulum, F16GCAS, and NeuralLander, each for 1 million steps. However, I observed that the rewards across all these environments were notably below the figures presented in Figure 5 of the paper (InvertedPendulum: approximately 100-200, F16GCAS: approximately 100, NeuralLander: approximately 2000).

I would like to inquire about any potential missteps I may have taken or if there might be some misinterpretation of the paper and code on my part. Thanks very much for your attention and time!