深度学习训练时间过长GPU占用率过低问题
配置环境问题描述调参结果修改num_workers结论 附录配置环境
操作系统：Ubuntu20.04
CUDA版本：10.2
Pytorch版本：1.6.0
TorchVision版本：0.7.0
mmdet版本：2.5.0
mmcv版本：1.1.5
IDE：PyCharm
硬件：RTX2070S*2
问题描述
在训练YOLOv4tiny时发现GPU占用率非常低，并且经常跳到0，导致训练速度很慢
为此博主对几个时间点就行设置，打印出来加载数据花费的时间和真正网络训练花费的时间，结果加载数据花费了20多秒，训练也只20多秒。加载数据出大问题，加载数据本文用的是Pytorch中自带的DataLoader
gen = DataLoader(train_dataset, shuffle=True, batch_size=Batch_size, num_workers=32, pin_memory=True,drop_last=True, collate_fn=yolo_dataset_collate)
调参结果
为了弄清楚要调什么参数来加快加速训练，做了以下实验
修改num_workers
截取日志中一部分进行日志说明：
num_workers=32, pin_memory=True############################################**************************************************0.00020313262939453125Epoch 1/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 23.64951467514038**************************************************every batch cost: 2.750419855117798Epoch 1/50: 5%|▌ | 2/39 [00:27<11:37, 18.84s/it, lr=0.001, step/s=1.21, ......Epoch 1/50: 100%|██████████| 39/39 [00:58<00:00, 1.51s/it, lr=0.001, step/s=0.623, total_loss=406]
############################################上的为此时的参数，这次实验便是num_workers=32, pin_memory=True
在每一次加载数据后都会将加载时间写在第一个Epoch下，此次实验便是23.64951467514038s
后面的every batch cost: 2.750419855117798为网络真正训练一步花费的时间
当出现100%|██████████| 39/39 [00:58<00:00,时，表示一个epoch训练结束，总花费时间为58s（包含加载数据时间）
结果见附录：
结论
随着num_workers的减少加载数据花费的时间反而少了，但是整个训练段时间却加长了
最佳的num_workers应为32
附录
num_workers=32, pin_memory=True############################################**************************************************0.00020313262939453125Epoch 1/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 23.64951467514038**************************************************every batch cost: 2.750419855117798Epoch 1/50: 5%|▌ | 2/39 [00:27<11:37, 18.84s/it, lr=0.001, step/s=1.21, total_loss=1.66e+3]**************************************************every batch cost: 1.2042901515960693Epoch 1/50: 8%|▊ | 3/39 [00:28<08:08, 13.57s/it, lr=0.001, step/s=1.26, total_loss=1.57e+3]**************************************************every batch cost: 1.2558915615081787Epoch 1/50: 10%|█ | 4/39 [00:34<06:32, 11.20s/it, lr=0.001, step/s=5.67, total_loss=1.48e+3]**************************************************every batch cost: 1.228856086730957Epoch 1/50: 13%|█▎ | 5/39 [00:35<04:37, 8.17s/it, lr=0.001, step/s=1.09, total_loss=1.4e+3]**************************************************every batch cost: 1.0908327102661133**************************************************every batch cost: 0.7014598846435547Epoch 1/50: 18%|█▊ | 7/39 [00:38<02:33, 4.79s/it, lr=0.001, step/s=1.05, total_loss=1.26e+3]**************************************************every batch cost: 1.0383291244506836Epoch 1/50: 21%|██ | 8/39 [00:39<01:53, 3.66s/it, lr=0.001, step/s=1.03, total_loss=1.19e+3]**************************************************every batch cost: 0.8855016231536865Epoch 1/50: 23%|██▎ | 9/39 [00:41<01:27, 2.90s/it, lr=0.001, step/s=1.11, total_loss=1.13e+3]**************************************************every batch cost: 1.1094863414764404Epoch 1/50: 26%|██▌ | 10/39 [00:42<01:07, 2.34s/it, lr=0.001, step/s=1.02, total_loss=1.07e+3]**************************************************every batch cost: 1.0229365825653076**************************************************every batch cost: 1.1253528594970703Epoch 1/50: 31%|███ | 12/39 [00:44<00:45, 1.70s/it, lr=0.001, step/s=1.03, total_loss=974]**************************************************every batch cost: 1.0269100666046143Epoch 1/50: 33%|███▎| 13/39 [00:44<00:34, 1.33s/it, lr=0.001, step/s=0.464, total_loss=930]**************************************************every batch cost: 0.46166372299194336**************************************************every batch cost: 0.442338228225708Epoch 1/50: 38%|███▊| 15/39 [00:45<00:21, 1.09it/s, lr=0.001, step/s=0.538, total_loss=851]**************************************************every batch cost: 0.5364859104156494Epoch 1/50: 41%|████| 16/39 [00:46<00:18, 1.27it/s, lr=0.001, step/s=0.486, total_loss=816]**************************************************every batch cost: 0.484661340713501Epoch 1/50: 44%|████▎| 17/39 [00:46<00:15, 1.45it/s, lr=0.001, step/s=0.455, total_loss=783]**************************************************every batch cost: 0.45422983169555664**************************************************every batch cost: 0.46462392807006836Epoch 1/50: 49%|████▊| 19/39 [00:47<00:12, 1.59it/s, lr=0.001, step/s=0.632, total_loss=724]**************************************************every batch cost: 0.6317436695098877**************************************************every batch cost: 0.4574618339538574Epoch 1/50: 54%|█████▍ | 21/39 [00:48<00:09, 1.81it/s, lr=0.001, step/s=0.48, total_loss=672]**************************************************every batch cost: 0.4788329601287842**************************************************every batch cost: 0.6374142169952393Epoch 1/50: 59%|█████▉ | 23/39 [00:49<00:08, 1.82it/s, lr=0.001, step/s=0.466, total_loss=627]**************************************************every batch cost: 0.4624333381652832Epoch 1/50: 62%|██████▏ | 24/39 [00:50<00:07, 1.88it/s, lr=0.001, step/s=0.478, total_loss=607]**************************************************every batch cost: 0.4764266014099121**************************************************every batch cost: 0.48482751846313477Epoch 1/50: 67%|██████▋ | 26/39 [00:51<00:07, 1.85it/s, lr=0.001, step/s=0.582, total_loss=569]**************************************************every batch cost: 0.5818459987640381Epoch 1/50: 69%|██████▉ | 27/39 [00:52<00:06, 1.79it/s, lr=0.001, step/s=0.595, total_loss=552]**************************************************every batch cost: 0.591942548751831**************************************************every batch cost: 0.45967578887939453Epoch 1/50: 74%|███████▍ | 29/39 [00:53<00:05, 1.99it/s, lr=0.001, step/s=0.421, total_loss=521]**************************************************every batch cost: 0.4171140193939209Epoch 1/50: 77%|███████▋ | 30/39 [00:53<00:04, 1.89it/s, lr=0.001, step/s=0.584, total_loss=507]**************************************************every batch cost: 0.5828967094421387Epoch 1/50: 79%|███████▉ | 31/39 [00:54<00:04, 1.93it/s, lr=0.001, step/s=0.48, total_loss=493]**************************************************every batch cost: 0.4693794250488281**************************************************every batch cost: 0.47809910774230957Epoch 1/50: 85%|████████▍ | 33/39 [00:55<00:02, 2.00it/s, lr=0.001, step/s=0.468, total_loss=468]**************************************************every batch cost: 0.4680147171020508**************************************************every batch cost: 0.47008824348449707Epoch 1/50: 90%|████████▉ | 35/39 [00:56<00:02, 1.88it/s, lr=0.001, step/s=0.613, total_loss=445]**************************************************every batch cost: 0.6110391616821289**************************************************every batch cost: 0.5674312114715576Epoch 1/50: 95%|█████████▍| 37/39 [00:57<00:01, 1.76it/s, lr=0.001, step/s=0.612, total_loss=425]**************************************************every batch cost: 0.6096856594085693**************************************************every batch cost: 0.5798914432525635Epoch 1/50: 100%|██████████| 39/39 [00:58<00:00, 1.69it/s, lr=0.001, step/s=0.623, total_loss=406]**************************************************every batch cost: 0.621854305267334Epoch 1/50: 100%|██████████| 39/39 [00:58<00:00, 1.51s/it, lr=0.001, step/s=0.623, total_loss=406]Epoch 1/50: 0%|| 0/4 [00:00<?, ?it/s<class 'dict'>]Start ValidationEpoch 1/50: 100%|██████████| 4/4 [00:09<00:00, 2.36s/it, total_loss=53.2]Finish ValidationEpoch:1/50Total Loss: 396.0543 || Val Loss: 42.5870 Saving state, iter: 1Epoch 2/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 39.29310464859009**************************************************every batch cost: 0.7208483219146729Epoch 2/50: 5%|▌ | 2/39 [00:42<17:40, 28.66s/it, lr=0.000905, step/s=2.17, total_loss=55.5]**************************************************every batch cost: 0.728407621383667Epoch 2/50: 8%|▊ | 3/39 [00:43<12:14, 20.39s/it, lr=0.000905, step/s=1.08, total_loss=54.6]**************************************************every batch cost: 1.0782456398010254Epoch 2/50: 10%|█ | 4/39 [00:44<08:30, 14.58s/it, lr=0.000905, step/s=1.01, total_loss=53.8]**************************************************every batch cost: 1.004713773727417**************************************************every batch cost: 0.9956800937652588Epoch 2/50: 15%|█▌ | 6/39 [00:45<04:08, 7.54s/it, lr=0.000905, step/s=0.622, total_loss=52]**************************************************every batch cost: 0.678076934814Epoch 2/50: 18%|█▊ | 7/39 [00:46<02:53, 5.42s/it, lr=0.000905, step/s=0.458, total_loss=51.3]**************************************************every batch cost: 0.46021366119384766**************************************************every batch cost: 0.5946834087371826Epoch 2/50: 23%|██▎ | 9/39 [00:47<01:28, 2.96s/it, lr=0.000905, step/s=0.565, total_loss=49.9]**************************************************every batch cost: 0.5631444454193115**************************************************every batch cost: 0.44608592987060547Epoch 2/50: 28%|██▊ | 11/39 [00:48<00:48, 1.73s/it, lr=0.000905, step/s=0.609, total_loss=48.7]**************************************************every batch cost: 0.608043909072876Epoch 2/50: 31%|███ | 12/39 [00:49<00:36, 1.36s/it, lr=0.000905, step/s=0.486, total_loss=48.1]**************************************************every batch cost: 0.4828364849090576Epoch 2/50: 33%|███▎| 13/39 [00:49<00:28, 1.09s/it, lr=0.000905, step/s=0.465, total_loss=47.6]**************************************************every batch cost: 0.4629390239715576**************************************************every batch cost: 0.5994398593902588Epoch 2/50: 38%|███▊| 15/39 [00:50<00:20, 1.18it/s, lr=0.000905, step/s=0.606, total_loss=46.6]**************************************************every batch cost: 0.6047122478485107Epoch 2/50: 41%|████| 16/39 [00:51<00:17, 1.32it/s, lr=0.000905, step/s=0.536, total_loss=46]**************************************************every batch cost: 0.5342566967010498**************************************************every batch cost: 0.5950450897216797Epoch 2/50: 46%|████▌| 18/39 [00:52<00:14, 1.45it/s, lr=0.000905, step/s=0.617, total_loss=45]**************************************************every batch cost: 0.6158950328826904**************************************************every batch cost: 0.4612059593836Epoch 2/50: 51%|█████▏ | 20/39 [00:53<00:10, 1.74it/s, lr=0.000905, step/s=0.454, total_loss=44.1]**************************************************every batch cost: 0.4514193534851074**************************************************every batch cost: 0.566706657409668Epoch 2/50: 56%|█████▋ | 22/39 [00:54<00:09, 1.82it/s, lr=0.000905, step/s=0.48, total_loss=43.2]**************************************************every batch cost: 0.47788500785827637**************************************************every batch cost: 0.4445042610168457Epoch 2/50: 62%|██████▏ | 24/39 [00:55<00:08, 1.82it/s, lr=0.000905, step/s=0.608, total_loss=42.4]**************************************************every batch cost: 0.6049983501434326Epoch 2/50: 64%|██████▍ | 25/39 [00:56<00:07, 1.91it/s, lr=0.000905, step/s=0.456, total_loss=42]**************************************************every batch cost: 0.4528319835662842Epoch 2/50: 67%|██████▋ | 26/39 [00:56<00:06, 2.01it/s, lr=0.000905, step/s=0.421, total_loss=41.6]**************************************************every batch cost: 0.4186275005340576Epoch 2/50: 69%|██████▉ | 27/39 [00:57<00:05, 2.10it/s, lr=0.000905, step/s=0.416, total_loss=41.2]**************************************************every batch cost: 0.4143967628479004**************************************************every batch cost: 0.6277880668640137Epoch 2/50: 74%|███████▍ | 29/39 [00:58<00:04, 2.07it/s, lr=0.000905, step/s=0.381, total_loss=40.5]**************************************************every batch cost: 0.37793684005737305**************************************************every batch cost: 0.5735199451446533Epoch 2/50: 79%|███████▉ | 31/39 [00:59<00:04, 1.82it/s, lr=0.000905, step/s=0.627, total_loss=39.8]**************************************************every batch cost: 0.6239285469055176Epoch 2/50: 82%|████████▏ | 32/39 [00:59<00:03, 1.91it/s, lr=0.000905, step/s=0.447, total_loss=39.4]**************************************************every batch cost: 0.4438314437866211Epoch 2/50: 85%|████████▍ | 33/39 [01:00<00:03, 1.98it/s, lr=0.000905, step/s=0.455, total_loss=39.1]**************************************************every batch cost: 0.45426464080810547**************************************************every batch cost: 0.4659461975097656Epoch 2/50: 90%|████████▉ | 35/39 [01:01<00:02, 1.90it/s, lr=0.000905, step/s=0.588, total_loss=38.5]**************************************************every batch cost: 0.5880696773529053**************************************************every batch cost: 0.560051441192627Epoch 2/50: 95%|█████████▍| 37/39 [01:02<00:01, 1.93it/s, lr=0.000905, step/s=0.459, total_loss=37.9]**************************************************every batch cost: 0.4551582336425781Epoch 2/50: 97%|█████████▋| 38/39 [01:02<00:00, 1.98it/s, lr=0.000905, step/s=0.464, total_loss=37.6]**************************************************every batch cost: 0.46265411376953125Epoch 2/50: 100%|██████████| 39/39 [01:03<00:00, 1.85it/s, lr=0.000905, step/s=0.619, total_loss=37.3]**************************************************every batch cost: 0.6277220249176025Epoch 2/50: 100%|██████████| 39/39 [01:03<00:00, 1.63s/it, lr=0.000905, step/s=0.619, total_loss=37.3]Epoch 2/50: 0%|| 0/4 [00:00<?, ?it/s<class 'dict'>]Start ValidationEpoch 2/50: 100%|██████████| 4/4 [00:09<00:00, 2.41s/it, total_loss=24.6]Finish ValidationEpoch:2/50Total Loss: 36.4114 || Val Loss: 19.6662 Saving state, iter: 2Epoch 3/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 35.0421621799469**************************************************every batch cost: 1.97382736206Epoch 3/50: 5%|▌ | 2/39 [00:43<16:55, 27.45s/it, lr=0.000658, step/s=6.93, total_loss=26.2]**************************************************every batch cost: 1.0342633724212646Epoch 3/50: 8%|▊ | 3/39 [00:44<11:43, 19.55s/it, lr=0.000658, step/s=1.11, total_loss=26]**************************************************every batch cost: 1.1082875728607178Epoch 3/50: 10%|█ | 4/39 [00:45<08:12, 14.07s/it, lr=0.000658, step/s=1.25, total_loss=25.9]**************************************************every batch cost: 1.2519311904907227Epoch 3/50: 13%|█▎ | 5/39 [00:46<05:45, 10.16s/it, lr=0.000658, step/s=1.03, total_loss=25.6]**************************************************every batch cost: 1.0359077453613281**************************************************every batch cost: 0.8948409557342529Epoch 3/50: 18%|█▊ | 7/39 [00:47<02:49, 5.30s/it, lr=0.000658, step/s=0.439, total_loss=25.4]**************************************************every batch cost: 0.4398157596588135Epoch 3/50: 21%|██ | 8/39 [00:48<02:01, 3.91s/it, lr=0.000658, step/s=0.647, total_loss=25.2]**************************************************every batch cost: 0.6475496292114258**************************************************every batch cost: 0.441436767578125Epoch 3/50: 26%|██▌ | 10/39 [00:49<01:02, 2.15s/it, lr=0.000658, step/s=0.45, total_loss=25]**************************************************every batch cost: 0.4504427909851074**************************************************every batch cost: 0.659254789352417Epoch 3/50: 31%|███ | 12/39 [00:50<00:35, 1.33s/it, lr=0.000658, step/s=0.448, total_loss=24.8]**************************************************every batch cost: 0.4467794895172119Epoch 3/50: 33%|███▎| 13/39 [00:51<00:27, 1.07s/it, lr=0.000658, step/s=0.451, total_loss=24.6]**************************************************every batch cost: 0.45061635971069336Epoch 3/50: 36%|███▌| 14/39 [00:51<00:22, 1.12it/s, lr=0.000658, step/s=0.463, total_loss=24.5]**************************************************every batch cost: 0.46184635162353516**************************************************every batch cost: 0.45751118659973145Epoch 3/50: 41%|████| 16/39 [00:52<00:15, 1.47it/s, lr=0.000658, step/s=0.484, total_loss=24.3]**************************************************every batch cost: 0.4834902286529541Epoch 3/50: 44%|████▎| 17/39 [00:53<00:13, 1.59it/s, lr=0.000658, step/s=0.493, total_loss=24.2]**************************************************every batch cost: 0.49303245544433594**************************************************every batch cost: 0.4651303291320801Epoch 3/50: 49%|████▊| 19/39 [00:54<00:11, 1.72it/s, lr=0.000658, step/s=0.571, total_loss=24]**************************************************every batch cost: 0.5711841583251953**************************************************every batch cost: 0.6247999668121338Epoch 3/50: 54%|█████▍ | 21/39 [00:55<00:10, 1.67it/s, lr=0.000658, step/s=0.595, total_loss=23.8]**************************************************every batch cost: 0.5951683521270752**************************************************every batch cost: 0.4792053699493408Epoch 3/50: 59%|█████▉ | 23/39 [00:56<00:08, 1.84it/s, lr=0.000658, step/s=0.482, total_loss=23.6]**************************************************every batch cost: 0.4827558994293213Epoch 3/50: 62%|██████▏ | 24/39 [00:56<00:07, 1.93it/s, lr=0.000658, step/s=0.446, total_loss=23.5]**************************************************every batch cost: 0.4450535774230957**************************************************every batch cost: 0.4633326530456543Epoch 3/50: 67%|██████▋ | 26/39 [00:57<00:06, 2.05it/s, lr=0.000658, step/s=0.439, total_loss=23.2]**************************************************every batch cost: 0.43566155433654785**************************************************every batch cost: 0.6175928115844727Epoch 3/50: 72%|███████▏ | 28/39 [00:58<00:05, 1.99it/s, lr=0.000658, step/s=0.431, total_loss=23]**************************************************every batch cost: 0.42892026901245117Epoch 3/50: 74%|███████▍ | 29/39 [00:59<00:05, 1.88it/s, lr=0.000658, step/s=0.594, total_loss=23]**************************************************every batch cost: 0.593457698825**************************************************every batch cost: 0.4818403720855713Epoch 3/50: 79%|███████▉ | 31/39 [01:00<00:03, 2.01it/s, lr=0.000658, step/s=0.438, total_loss=22.8]**************************************************every batch cost: 0.4389064311981201**************************************************every batch cost: 0.5992274284362793Epoch 3/50: 85%|████████▍ | 33/39 [01:01<00:03, 1.88it/s, lr=0.000658, step/s=0.522, total_loss=22.7]**************************************************every batch cost: 0.5205366611480713**************************************************every batch cost: 0.6633074283599854Epoch 3/50: 90%|████████▉ | 35/39 [01:02<00:02, 1.71it/s, lr=0.000658, step/s=0.608, total_loss=22.5]**************************************************every batch cost: 0.6174895763397217Epoch 3/50: 92%|█████████▏| 36/39 [01:03<00:01, 1.88it/s, lr=0.000658, step/s=0.395, total_loss=22.4]**************************************************every batch cost: 0.39395880699157715Epoch 3/50: 95%|█████████▍| 37/39 [01:03<00:01, 1.94it/s, lr=0.000658, step/s=0.465, total_loss=22.3]**************************************************every batch cost: 0.46547436714172363Epoch 3/50: 97%|█████████▋| 38/39 [01:04<00:00, 1.81it/s, lr=0.000658, step/s=0.628, total_loss=22.2]**************************************************every batch cost: 0.6294529438018799Epoch 3/50: 100%|██████████| 39/39 [01:04<00:00, 1.92it/s, lr=0.000658, step/s=0.436, total_loss=22.1]**************************************************every batch cost: 0.43511104583740234Epoch 3/50: 100%|██████████| 39/39 [01:05<00:00, 1.67s/it, lr=0.000658, step/s=0.436, total_loss=22.1]Epoch 3/50: 0%|| 0/4 [00:00<?, ?it/s<class 'dict'>]Start ValidationEpoch 3/50: 100%|██████████| 4/4 [00:09<00:00, 2.42s/it, total_loss=17]Finish ValidationEpoch:3/50Total Loss: 21.5840 || Val Loss: 13.6119 Saving state, iter: 3Epoch 4/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 36.35298943519592Epoch 4/50: 3%|▎ | 1/39 [00:37<23:48, 37.58s/it, lr=0.000352, step/s=37.6, total_loss=19.2]**************************************************every batch cost: 1.2303707599639893**************************************************every batch cost: 1.144068956375122Epoch 4/50: 8%|▊ | 3/39 [00:39<11:22, 18.96s/it, lr=0.000352, step/s=0.999, total_loss=18.6]**************************************************every batch cost: 0.9969470500946045**************************************************Epoch 4/50: 10%|█ | 4/39 [00:40<07:56, 13.61s/it, lr=0.000352, step/s=0.878, total_loss=18.5]every batch cost: 1.114208698272705**************************************************every batch cost: 0.7834813594818115Epoch 4/50: 15%|█▌ | 6/39 [00:43<03:58, 7.23s/it, lr=0.000352, step/s=1.17, total_loss=18.3]**************************************************every batch cost: 1.1689801216125488**************************************************every batch cost: 1.0269944667816162Epoch 4/50: 21%|██ | 8/39 [00:45<02:05, 4.06s/it, lr=0.000352, step/s=0.997, total_loss=18.3]**************************************************every batch cost: 0.8472988605499268**************************************************every batch cost: 1.1264011859893799Epoch 4/50: 26%|██▌ | 10/39 [00:47<01:14, 2.58s/it, lr=0.000352, step/s=1.03, total_loss=18.3]**************************************************every batch cost: 1.317326545715332**************************************************every batch cost: 0.6522595882415771Epoch 4/50: 31%|███ | 12/39 [00:48<00:44, 1.64s/it, lr=0.000352, step/s=0.676, total_loss=18.3]**************************************************every batch cost: 0.6746888160705566Epoch 4/50: 33%|███▎| 13/39 [00:49<00:35, 1.35s/it, lr=0.000352, step/s=0.644, total_loss=18.3]**************************************************every batch cost: 0.6441326141357422Epoch 4/50: 36%|███▌| 14/39 [00:50<00:27, 1.09s/it, lr=0.000352, step/s=0.466, total_loss=18.3]**************************************************every batch cost: 0.4652385711669922Epoch 4/50: 38%|███▊| 15/39 [00:50<00:21, 1.12it/s, lr=0.000352, step/s=0.444, total_loss=18.2]**************************************************every batch cost: 0.443436384498**************************************************every batch cost: 0.6525671482086182Epoch 4/50: 44%|████▎| 17/39 [00:51<00:15, 1.40it/s, lr=0.000352, step/s=0.448, total_loss=18.2]**************************************************every batch cost: 0.44656848907470703**************************************************every batch cost: 0.46964049339294434Epoch 4/50: 49%|████▊| 19/39 [00:52<00:12, 1.56it/s, lr=0.000352, step/s=0.623, total_loss=18.1]**************************************************every batch cost: 0.6219379901885986**************************************************every batch cost: 0.5789623260498047Epoch 4/50: 54%|█████▍ | 21/39 [00:53<00:10, 1.71it/s, lr=0.000352, step/s=0.482, total_loss=18]**************************************************every batch cost: 0.48101806640625Epoch 4/50: 56%|█████▋ | 22/39 [00:54<00:10, 1.68it/s, lr=0.000352, step/s=0.605, total_loss=18]**************************************************every batch cost: 0.6064701080322266**************************************************every batch cost: 0.5328128337860107Epoch 4/50: 62%|██████▏ | 24/39 [00:55<00:08, 1.82it/s, lr=0.000352, step/s=0.468, total_loss=17.9]**************************************************every batch cost: 0.46857547760009766Epoch 4/50: 64%|██████▍ | 25/39 [00:56<00:07, 1.77it/s, lr=0.000352, step/s=0.597, total_loss=17.8]**************************************************every batch cost: 0.59743332862854Epoch 4/50: 67%|██████▋ | 26/39 [00:56<00:07, 1.72it/s, lr=0.000352, step/s=0.611, total_loss=17.8]**************************************************every batch cost: 0.6083073616027832Epoch 4/50: 69%|██████▉ | 27/39 [00:57<00:07, 1.66it/s, lr=0.000352, step/s=0.641, total_loss=17.8]**************************************************every batch cost: 0.6388955116271973**************************************************every batch cost: 0.6160726547241211Epoch 4/50: 74%|███████▍ | 29/39 [00:58<00:05, 1.80it/s, lr=0.000352, step/s=0.418, total_loss=17.7]**************************************************every batch cost: 0.4156637191772461Epoch 4/50: 77%|███████▋ | 30/39 [00:58<00:04, 1.87it/s, lr=0.000352, step/s=0.479, total_loss=17.7]**************************************************every batch cost: 0.4801647663116455**************************************************every batch cost: 0.6667578220367432Epoch 4/50: 82%|████████▏ | 32/39 [01:00<00:04, 1.67it/s, lr=0.000352, step/s=0.638, total_loss=17.6]**************************************************every batch cost: 0.6348106861114502Epoch 4/50: 85%|████████▍ | 33/39 [01:00<00:03, 1.63it/s, lr=0.000352, step/s=0.632, total_loss=17.6]**************************************************every batch cost: 0.6294233798980713Epoch 4/50: 87%|████████▋ | 34/39 [01:01<00:03, 1.63it/s, lr=0.000352, step/s=0.601, total_loss=17.5]**************************************************every batch cost: 0.599083662033081Epoch 4/50: 90%|████████▉ | 35/39 [01:02<00:02, 1.62it/s, lr=0.000352, step/s=0.619, total_loss=17.5]**************************************************every batch cost: 0.6171004772186279Epoch 4/50: 92%|█████████▏| 36/39 [01:02<00:01, 1.61it/s, lr=0.000352, step/s=0.625, total_loss=17.5]**************************************************every batch cost: 0.6238992214202881Epoch 4/50: 95%|█████████▍| 37/39 [01:03<00:01, 1.56it/s, lr=0.000352, step/s=0.676, total_loss=17.5]**************************************************every batch cost: 0.6731662750244141Epoch 4/50: 97%|█████████▋| 38/39 [01:04<00:00, 1.54it/s, lr=0.000352, step/s=0.659, total_loss=17.5]**************************************************every batch cost: 0.6571424007415771Epoch 4/50: 100%|██████████| 39/39 [01:04<00:00, 1.57it/s, lr=0.000352, step/s=0.599, total_loss=17.4]**************************************************every batch cost: 0.5965077877044678Epoch 4/50: 100%|██████████| 39/39 [01:05<00:00, 1.67s/it, lr=0.000352, step/s=0.599, total_loss=17.4]Epoch 4/50: 0%|| 0/4 [00:00<?, ?it/s<class 'dict'>]Start ValidationEpoch 4/50: 100%|██████████| 4/4 [00:09<00:00, 2.45s/it, total_loss=14.4]Finish ValidationEpoch:4/50Total Loss: 17.0058 || Val Loss: 11.5347 Saving state, iter: 4Epoch 5/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 38.64342498779297Epoch 5/50: 3%|▎ | 1/39 [00:39<24:55, 39.35s/it, lr=0.000105, step/s=39.3, total_loss=16]**************************************************every batch cost: 0.709841251373291Epoch 5/50: 5%|▌ | 2/39 [00:40<17:10, 27.84s/it, lr=0.000105, step/s=0.973, total_loss=15.5]**************************************************every batch cost: 0.972780704498291num_workers=16, pin_memory=True############################################Epoch 1/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 14.460282325744629Epoch 1/50: 3%|▎ | 1/39 [00:16<10:45, 16.98s/it, lr=0.001, step/s=17, total_loss=1.86e+3]**************************************************every batch cost: 2.522810220718384Epoch 1/50: 5%|▌ | 2/39 [00:28<09:32, 15.47s/it, lr=0.001, step/s=11.9, total_loss=1.75e+3]**************************************************every batch cost: 0.7985043525695801**************************************************every batch cost: 1.024996042251587Epoch 1/50: 10%|█ | 4/39 [00:31<04:49, 8.28s/it, lr=0.001, step/s=1.62, total_loss=1.56e+3]**************************************************every batch cost: 1.607698331787Epoch 1/50: 13%|█▎ | 5/39 [00:32<03:29, 6.17s/it, lr=0.001, step/s=1.23, total_loss=1.48e+3]**************************************************every batch cost: 1.2147128582000732Epoch 1/50: 15%|█▌ | 6/39 [00:34<02:35, 4.72s/it, lr=0.001, step/s=1.33, total_loss=1.4e+3]**************************************************every batch cost: 1.313018798828**************************************************every batch cost: 1.310088872909546Epoch 1/50: 21%|██ | 8/39 [00:36<01:31, 2.96s/it, lr=0.001, step/s=1.23, total_loss=1.26e+3]**************************************************every batch cost: 1.2326617240905762**************************************************every batch cost: 1.2288479804992676Epoch 1/50: 26%|██▌ | 10/39 [00:39<01:01, 2.12s/it, lr=0.001, step/s=1.33, total_loss=1.14e+3]**************************************************every batch cost: 1.3325657844543457Epoch 1/50: 28%|██▊ | 11/39 [00:40<00:51, 1.85s/it, lr=0.001, step/s=1.21, total_loss=1.08e+3]**************************************************every batch cost: 1.2089898586273193Epoch 1/50: 31%|███ | 12/39 [00:42<00:46, 1.74s/it, lr=0.001, step/s=1.32, total_loss=1.03e+3]**************************************************every batch cost: 1.170703649520874Epoch 1/50: 33%|███▎| 13/39 [00:42<00:35, 1.36s/it, lr=0.001, step/s=0.475, total_loss=983]**************************************************every batch cost: 0.4666435718536377Epoch 1/50: 36%|███▌| 14/39 [00:43<00:29, 1.18s/it, lr=0.001, step/s=0.733, total_loss=940]**************************************************every batch cost: 0.7289862632751465**************************************************every batch cost: 0.454329252243042Epoch 1/50: 41%|████| 16/39 [00:44<00:18, 1.23it/s, lr=0.001, step/s=0.449, total_loss=862]**************************************************every batch cost: 0.44701433181762695**************************************************every batch cost: 0.46754884719848633Epoch 1/50: 46%|████▌| 18/39 [00:45<00:13, 1.60it/s, lr=0.001, step/s=0.415, total_loss=795]**************************************************every batch cost: 0.41454458236694336**************************************************every batch cost: 0.4329829216003418Epoch 1/50: 51%|█████▏ | 20/39 [00:45<00:10, 1.84it/s, lr=0.001, step/s=0.47, total_loss=737]**************************************************every batch cost: 0.4669842717383**************************************************every batch cost: 0.44852232933044434Epoch 1/50: 56%|█████▋ | 22/39 [00:47<00:09, 1.85it/s, lr=0.001, step/s=0.59, total_loss=686]**************************************************every batch cost: 0.5892877578735352**************************************************every batch cost: 0.5070881843566895Epoch 1/50: 62%|██████▏ | 24/39 [00:48<00:08, 1.83it/s, lr=0.001, step/s=0.565, total_loss=642]**************************************************every batch cost: 0.5660033226013184**************************************************every batch cost: 0.4133646488189697Epoch 1/50: 67%|██████▋ | 26/39 [00:49<00:06, 2.02it/s, lr=0.001, step/s=0.451, total_loss=602]**************************************************every batch cost: 0.4506559371948242**************************************************every batch cost: 0.625328779220581Epoch 1/50: 72%|███████▏ | 28/39 [00:50<00:05, 1.93it/s, lr=0.001, step/s=0.471, total_loss=567]**************************************************every batch cost: 0.46766138076782227**************************************************every batch cost: 0.4441709518432617Epoch 1/50: 77%|███████▋ | 30/39 [00:51<00:04, 2.07it/s, lr=0.001, step/s=0.439, total_loss=536]**************************************************every batch cost: 0.4360997676849365**************************************************every batch cost: 0.4722580909729004Epoch 1/50: 82%|████████▏ | 32/39 [00:51<00:03, 2.11it/s, lr=0.001, step/s=0.439, total_loss=508]**************************************************every batch cost: 0.43885302543640137Epoch 1/50: 85%|████████▍ | 33/39 [00:52<00:02, 2.08it/s, lr=0.001, step/s=0.486, total_loss=495]**************************************************every batch cost: 0.485029935836792Epoch 1/50: 87%|████████▋ | 34/39 [00:52<00:02, 2.18it/s, lr=0.001, step/s=0.402, total_loss=483]**************************************************every batch cost: 0.39949965476989746Epoch 1/50: 90%|████████▉ | 35/39 [00:53<00:01, 2.15it/s, lr=0.001, step/s=0.473, total_loss=471]**************************************************every batch cost: 0.471925078955Epoch 1/50: 92%|█████████▏| 36/39 [00:53<00:01, 2.21it/s, lr=0.001, step/s=0.415, total_loss=460]**************************************************every batch cost: 0.4150104522705078Epoch 1/50: 95%|█████████▍| 37/39 [00:54<00:00, 2.14it/s, lr=0.001, step/s=0.493, total_loss=449]**************************************************every batch cost: 0.4902806282043457Epoch 1/50: 97%|█████████▋| 38/39 [00:54<00:00, 2.16it/s, lr=0.001, step/s=0.439, total_loss=439]**************************************************every batch cost: 0.4379861354827881Epoch 1/50: 100%|██████████| 39/39 [00:55<00:00, 2.15it/s, lr=0.001, step/s=0.466, total_loss=430]**************************************************every batch cost: 0.46781086921691895Start ValidationEpoch 1/50: 100%|██████████| 39/39 [00:55<00:00, 1.42s/it, lr=0.001, step/s=0.466, total_loss=430]Epoch 1/50: 100%|██████████| 4/4 [00:08<00:00, 2.07s/it, total_loss=58.2]Finish ValidationEpoch:1/50Total Loss: 418.9609 || Val Loss: 46.5974 Saving state, iter: 1Epoch 2/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 15.21496319770813**************************************************every batch cost: 0.6695339679718018Epoch 2/50: 5%|▌ | 2/39 [00:24<08:31, 13.83s/it, lr=0.000905, step/s=9.04, total_loss=59]**************************************************every batch cost: 0.5902526378631592Epoch 2/50: 8%|▊ | 3/39 [00:26<06:02, 10.07s/it, lr=0.000905, step/s=1.28, total_loss=57.9]**************************************************every batch cost: 1.288323163986206Epoch 2/50: 10%|█ | 4/39 [00:31<05:04, 8.71s/it, lr=0.000905, step/s=5.53, total_loss=57.2]**************************************************every batch cost: 0.9619312286376953Epoch 2/50: 13%|█▎ | 5/39 [00:33<03:41, 6.50s/it, lr=0.000905, step/s=1.34, total_loss=56.1]**************************************************every batch cost: 1.33282470703125num_workers=8, pin_memory=True########################################################**************************************************0.0002262592315673828Epoch 1/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 14.393999099731445Epoch 1/50: 3%|▎ | 1/39 [00:16<10:28, 16.53s/it, lr=0.001, step/s=16.5, total_loss=1.66e+3]**************************************************every batch cost: 2.138085126876831Epoch 1/50: 5%|▌ | 2/39 [00:17<07:13, 11.73s/it, lr=0.001, step/s=0.51, total_loss=1.56e+3]**************************************************every batch cost: 0.4993324279785156Epoch 1/50: 8%|▊ | 3/39 [00:17<05:02, 8.39s/it, lr=0.001, step/s=0.6, total_loss=1.47e+3]**************************************************every batch cost: 0.5929081439971924Epoch 1/50: 10%|█ | 4/39 [00:18<03:31, 6.03s/it, lr=0.001, step/s=0.525, total_loss=1.39e+3]**************************************************every batch cost: 0.515371561050415Epoch 1/50: 13%|█▎ | 5/39 [00:18<02:28, 4.37s/it, lr=0.001, step/s=0.5, total_loss=1.31e+3]**************************************************every batch cost: 0.4886586666107178**************************************************every batch cost: 0.7947738170623779Epoch 1/50: 18%|█▊ | 7/39 [00:20<01:23, 2.61s/it, lr=0.001, step/s=0.991, total_loss=1.17e+3]**************************************************every batch cost: 0.9867894649505615Epoch 1/50: 21%|██ | 8/39 [00:21<01:09, 2.24s/it, lr=0.001, step/s=1.37, total_loss=1.11e+3]**************************************************every batch cost: 1.365293025970459Epoch 1/50: 23%|██▎ | 9/39 [00:25<01:17, 2.60s/it, lr=0.001, step/s=3.42, total_loss=1.05e+3]**************************************************every batch cost: 0.9516646862030029**************************************************every batch cost: 0.5585477352142334Epoch 1/50: 28%|██▊ | 11/39 [00:26<00:44, 1.57s/it, lr=0.001, step/s=0.592, total_loss=950]**************************************************every batch cost: 0.5857737064361572**************************************************every batch cost: 0.48770785331726074Epoch 1/50: 33%|███▎| 13/39 [00:27<00:26, 1.04s/it, lr=0.001, step/s=0.531, total_loss=863]**************************************************every batch cost: 0.5245354175567627Epoch 1/50: 36%|███▌| 14/39 [00:28<00:22, 1.10it/s, lr=0.001, step/s=0.603, total_loss=825]**************************************************every batch cost: 0.6007485389709473Epoch 1/50: 38%|███▊| 15/39 [00:28<00:19, 1.26it/s, lr=0.001, step/s=0.522, total_loss=789]**************************************************every batch cost: 0.5292763710021973**************************************************every batch cost: 1.456395149230957Epoch 1/50: 44%|████▎| 17/39 [00:34<00:41, 1.87s/it, lr=0.001, step/s=3.91, total_loss=726]**************************************************every batch cost: 0.8605103492736816**************************************************every batch cost: 0.9400577545166016Epoch 1/50: 49%|████▊| 19/39 [00:35<00:25, 1.28s/it, lr=0.001, step/s=0.539, total_loss=670]**************************************************every batch cost: 0.5312719345092773**************************************************every batch cost: 0.6191096305847168Epoch 1/50: 54%|█████▍ | 21/39 [00:36<00:16, 1.06it/s, lr=0.001, step/s=0.587, total_loss=623]**************************************************every batch cost: 0.5655641555786133Epoch 1/50: 56%|█████▋ | 22/39 [00:37<00:13, 1.26it/s, lr=0.001, step/s=0.451, total_loss=601]**************************************************every batch cost: 0.44422078132629395Epoch 1/50: 59%|█████▉ | 23/39 [00:38<00:13, 1.15it/s, lr=0.001, step/s=1.03, total_loss=581]**************************************************every batch cost: 1.0400919914245605**************************************************every batch cost: 1.245389461517334Epoch 1/50: 64%|██████▍ | 25/39 [00:42<00:21, 1.52s/it, lr=0.001, step/s=2.77, total_loss=544]**************************************************every batch cost: 0.874835729598999Epoch 1/50: 67%|██████▋ | 26/39 [00:43<00:17, 1.34s/it, lr=0.001, step/s=0.912, total_loss=527]**************************************************every batch cost: 0.9050905704498291**************************************************every batch cost: 0.6378293037414551Epoch 1/50: 72%|███████▏ | 28/39 [00:44<00:10, 1.04it/s, lr=0.001, step/s=0.556, total_loss=497]**************************************************every batch cost: 0.5520014762878418Epoch 1/50: 74%|███████▍ | 29/39 [00:44<00:08, 1.21it/s, lr=0.001, step/s=0.507, total_loss=483]**************************************************every batch cost: 0.5015432834625244**************************************************every batch cost: 0.5492942333221436Epoch 1/50: 79%|███████▉ | 31/39 [00:46<00:05, 1.45it/s, lr=0.001, step/s=0.546, total_loss=457]**************************************************every batch cost: 0.5536196231842041Epoch 1/50: 82%|████████▏ | 32/39 [00:46<00:04, 1.55it/s, lr=0.001, step/s=0.534, total_loss=445]**************************************************every batch cost: 0.5281355381011963Epoch 1/50: 85%|████████▍ | 33/39 [00:50<00:09, 1.65s/it, lr=0.001, step/s=3.97, total_loss=434]**************************************************every batch cost: 1.134850025177002**************************************************every batch cost: 0.41797971725463867Epoch 1/50: 90%|████████▉ | 35/39 [00:51<00:04, 1.05s/it, lr=0.001, step/s=0.497, total_loss=413]**************************************************every batch cost: 0.4928009510040283Epoch 1/50: 92%|█████████▏| 36/39 [00:51<00:02, 1.15it/s, lr=0.001, step/s=0.429, total_loss=403]**************************************************every batch cost: 0.4249401092529297**************************************************every batch cost: 0.44547200202941895Epoch 1/50: 97%|█████████▋| 38/39 [00:52<00:00, 1.54it/s, lr=0.001, step/s=0.418, total_loss=385]**************************************************every batch cost: 0.4139523506164551**************************************************every batch cost: 0.49946165084838867Epoch 1/50: 100%|██████████| 39/39 [00:53<00:00, 1.37s/it, lr=0.001, step/s=0.491, total_loss=377]Start ValidationEpoch 1/50: 100%|██████████| 4/4 [00:07<00:00, 1.87s/it, total_loss=51.2]Finish ValidationEpoch:1/50Total Loss: 367.2327 || Val Loss: 40.9737 Saving state, iter: 1Epoch 2/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 15.813380241394043Epoch 2/50: 3%|▎ | 1/39 [00:16<10:43, 16.92s/it, lr=0.000905, step/s=16.9, total_loss=53.5]**************************************************every batch cost: 1.1080009937286377**************************************************every batch cost: 0.898298978805542Epoch 2/50: 8%|▊ | 3/39 [00:18<05:10, 8.62s/it, lr=0.000905, step/s=0.466, total_loss=51.2]**************************************************every batch cost: 0.4567604064941406**************************************************every batch cost: 0.5346977710723877Epoch 2/50: 13%|█▎ | 5/39 [00:19<02:32, 4.49s/it, lr=0.000905, step/s=0.502, total_loss=50.1]**************************************************every batch cost: 0.494734525680542**************************************************every batch cost: 0.5638992786407471Epoch 2/50: 18%|█▊ | 7/39 [00:20<01:22, 2.57s/it, lr=0.000905, step/s=0.818, total_loss=48.6]**************************************************every batch cost: 0.812410831451416**************************************************every batch cost: 0.8895196914672852Epoch 2/50: 23%|██▎ | 9/39 [00:22<00:55, 1.83s/it, lr=0.000905, step/s=1.28, total_loss=47.1]**************************************************every batch cost: 0.803992748260498Saving state, iter: 2Epoch 3/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 13.237895488739014num_workers=4, pin_memory=True########################################################**************************************************0.00022721290588378906Epoch 1/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 8.96416425704956**************************************************every batch cost: 2.4007461071014404Epoch 1/50: 5%|▌ | 2/39 [00:11<04:59, 8.10s/it, lr=0.001, step/s=0.486, total_loss=1.55e+3]**************************************************every batch cost: 0.4803886413574219Epoch 1/50: 8%|▊ | 3/39 [00:12<03:29, 5.82s/it, lr=0.001, step/s=0.496, total_loss=1.46e+3]**************************************************every batch cost: 0.5041465759277344Epoch 1/50: 10%|█ | 4/39 [00:12<02:27, 4.22s/it, lr=0.001, step/s=0.458, total_loss=1.37e+3]**************************************************every batch cost: 0.45255351066589355**************************************************every batch cost: 0.48969101905822754Epoch 1/50: 15%|█▌ | 6/39 [00:19<01:55, 3.51s/it, lr=0.001, step/s=0.438, total_loss=1.23e+3]**************************************************every batch cost: 0.4459989070892334Epoch 1/50: 18%|█▊ | 7/39 [00:19<01:23, 2.60s/it, lr=0.001, step/s=0.472, total_loss=1.16e+3]**************************************************every batch cost: 0.4651622772216797**************************************************every batch cost: 0.48007702827453613Epoch 1/50: 23%|██▎ | 9/39 [00:26<01:38, 3.30s/it, lr=0.001, step/s=6.4, total_loss=1.04e+3]**************************************************every batch cost: 0.437375545501709**************************************************every batch cost: 0.4270005226135254Epoch 1/50: 28%|██▊ | 11/39 [00:27<00:52, 1.87s/it, lr=0.001, step/s=0.534, total_loss=943]**************************************************every batch cost: 0.5334141254425049**************************************************every batch cost: 0.432300329208374Epoch 1/50: 33%|███▎| 13/39 [00:34<01:13, 2.81s/it, lr=0.001, step/s=6, total_loss=857]**************************************************every batch cost: 0.40425610542297363Epoch 1/50: 36%|███▌| 14/39 [00:34<00:52, 2.11s/it, lr=0.001, step/s=0.47, total_loss=819]**************************************************every batch cost: 0.479872465133667Epoch 1/50: 38%|███▊| 15/39 [00:35<00:38, 1.61s/it, lr=0.001, step/s=0.418, total_loss=783]**************************************************every batch cost: 0.4144725799560547**************************************************every batch cost: 0.5075886249542236Epoch 1/50: 44%|████▎| 17/39 [00:42<01:01, 2.81s/it, lr=0.001, step/s=6.37, total_loss=720]**************************************************every batch cost: 0.8164219856262207**************************************************every batch cost: 0.4030587673187256Epoch 1/50: 49%|████▊| 19/39 [00:43<00:32, 1.62s/it, lr=0.001, step/s=0.492, total_loss=665]**************************************************every batch cost: 0.4989743232727051Epoch 1/50: 51%|█████▏ | 20/39 [00:43<00:24, 1.30s/it, lr=0.001, step/s=0.554, total_loss=641]**************************************************every batch cost: 0.5493001937866211**************************************************every batch cost: 0.8048973083496094Epoch 1/50: 56%|█████▋ | 22/39 [00:50<00:34, 2.06s/it, lr=0.001, step/s=0.461, total_loss=596]**************************************************every batch cost: 0.4469301700592041Epoch 1/50: 59%|█████▉ | 23/39 [00:50<00:25, 1.58s/it, lr=0.001, step/s=0.455, total_loss=576]**************************************************every batch cost: 0.46288108825683594Epoch 1/50: 62%|██████▏ | 24/39 [00:51<00:19, 1.31s/it, lr=0.001, step/s=0.689, total_loss=558]**************************************************every batch cost: 0.6864700317382812Epoch 1/50: 64%|██████▍ | 25/39 [00:57<00:37, 2.70s/it, lr=0.001, step/s=5.92, total_loss=540]**************************************************every batch cost: 0.7928135395050049Epoch 1/50: 67%|██████▋ | 26/39 [00:57<00:26, 2.02s/it, lr=0.001, step/s=0.445, total_loss=523]**************************************************every batch cost: 0.43711113929748535**************************************************every batch cost: 0.4797182083129883Epoch 1/50: 72%|███████▏ | 28/39 [00:58<00:13, 1.22s/it, lr=0.001, step/s=0.408, total_loss=493]**************************************************every batch cost: 0.40384721755981445**************************************************every batch cost: 0.7942979335784912Epoch 1/50: 77%|███████▋ | 30/39 [01:05<00:18, 2.09s/it, lr=0.001, step/s=0.441, total_loss=466]**************************************************every batch cost: 0.4379851818084717Epoch 1/50: 79%|███████▉ | 31/39 [01:06<00:12, 1.62s/it, lr=0.001, step/s=0.504, total_loss=453]**************************************************every batch cost: 0.510647583008Epoch 1/50: 82%|████████▏ | 32/39 [01:06<00:08, 1.28s/it, lr=0.001, step/s=0.484, total_loss=442]**************************************************every batch cost: 0.4807112216949463Epoch 1/50: 85%|████████▍ | 33/39 [01:12<00:16, 2.81s/it, lr=0.001, step/s=6.38, total_loss=430]**************************************************every batch cost: 0.9237005710601807Epoch 1/50: 87%|████████▋ | 34/39 [01:13<00:10, 2.14s/it, lr=0.001, step/s=0.548, total_loss=420]**************************************************every batch cost: 0.546823263168335Epoch 1/50: 90%|████████▉ | 35/39 [01:13<00:06, 1.63s/it, lr=0.001, step/s=0.446, total_loss=410]**************************************************every batch cost: 0.4558219909667969**************************************************every batch cost: 0.5195400714874268Epoch 1/50: 95%|█████████▍| 37/39 [01:19<00:05, 2.51s/it, lr=0.001, step/s=5.32, total_loss=391]**************************************************every batch cost: 0.7924432754516602**************************************************every batch cost: 0.4358391761779785Epoch 1/50: 100%|██████████| 39/39 [01:20<00:00, 1.45s/it, lr=0.001, step/s=0.394, total_loss=374]**************************************************every batch cost: 0.403045654296875Start ValidationEpoch 1/50: 100%|██████████| 39/39 [01:21<00:00, 2.08s/it, lr=0.001, step/s=0.394, total_loss=374]Epoch 1/50: 100%|██████████| 4/4 [00:07<00:00, 1.89s/it, total_loss=49.5]Finish ValidationEpoch:1/50Total Loss: 364.2570 || Val Loss: 39.5904 Saving state, iter: 1Epoch 2/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************8.968660593032837Epoch 2/50: 3%|▎ | 1/39 [00:09<06:12, 9.82s/it, lr=0.000905, step/s=9.81, total_loss=52.4]**************************************************every batch cost: 0.8473787307739258Epoch 2/50: 5%|▌ | 2/39 [00:12<04:39, 7.55s/it, lr=0.000905, step/s=2.24, total_loss=51.9]**************************************************every batch cost: 0.46636486053466797**************************************************every batch cost: 0.4795670509338379Epoch 2/50: 10%|█ | 4/39 [00:13<02:17, 3.94s/it, lr=0.000905, step/s=0.464, total_loss=50.6]**************************************************every batch cost: 0.46152782440185547Epoch 3/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 10.565217733383179num_workers=2, pin_memory=True########################################################**************************************************0.0007519721984863281Epoch 1/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 8.411113977432251**************************************************every batch cost: 1.3904120922088623Epoch 1/50: 5%|▌ | 2/39 [00:10<04:19, 7.02s/it, lr=0.001, step/s=0.51, total_loss=1.66e+3]**************************************************every batch cost: 0.5176153182983398Epoch 1/50: 8%|▊ | 3/39 [00:15<03:56, 6.57s/it, lr=0.001, step/s=5.53, total_loss=1.56e+3]**************************************************every batch cost: 0.441662073135376**************************************************every batch cost: 0.4310026168823242Epoch 1/50: 13%|█▎ | 5/39 [00:22<02:58, 5.25s/it, lr=0.001, step/s=6.47, total_loss=1.39e+3]**************************************************every batch cost: 0.4786345958709717**************************************************every batch cost: 0.5005531311035156Epoch 1/50: 18%|█▊ | 7/39 [00:29<02:29, 4.66s/it, lr=0.001, step/s=6.59, total_loss=1.25e+3]**************************************************every batch cost: 0.5762255191802979Epoch 1/50: 21%|██ | 8/39 [00:30<01:45, 3.41s/it, lr=0.001, step/s=0.506, total_loss=1.18e+3]**************************************************every batch cost: 0.5124270915985107Epoch 1/50: 23%|██▎ | 9/39 [00:36<02:09, 4.33s/it, lr=0.001, step/s=6.45, total_loss=1.12e+3]**************************************************every batch cost: 0.5291411876678467Epoch 1/50: 26%|██▌ | 10/39 [00:37<01:32, 3.19s/it, lr=0.001, step/s=0.537, total_loss=1.07e+3]**************************************************every batch cost: 0.5446579456329346**************************************************every batch cost: 0.40877699851989746Epoch 1/50: 31%|███ | 12/39 [00:44<01:20, 3.00s/it, lr=0.001, step/s=0.379, total_loss=968]**************************************************every batch cost: 0.38829779624938965**************************************************every batch cost: 0.4400508403778076Epoch 1/50: 36%|███▌| 14/39 [00:51<01:14, 2.98s/it, lr=0.001, step/s=0.459, total_loss=883]**************************************************every batch cost: 0.4682137966156006**************************************************every batch cost: 0.4381704330444336Epoch 1/50: 41%|████| 16/39 [00:57<01:07, 2.96s/it, lr=0.001, step/s=0.4, total_loss=810]**************************************************every batch cost: 0.4084131717681885**************************************************every batch cost: 0.44188928604125977Epoch 1/50: 46%|████▌| 18/39 [01:05<01:02, 2.99s/it, lr=0.001, step/s=0.476, total_loss=747]**************************************************every batch cost: 0.48462367057800293Epoch 1/50: 49%|████▊| 19/39 [01:11<01:21, 4.06s/it, lr=0.001, step/s=6.54, total_loss=719]**************************************************every batch cost: 0.4357719421386719Epoch 1/50: 51%|█████▏ | 20/39 [01:12<00:56, 2.96s/it, lr=0.001, step/s=0.41, total_loss=692]**************************************************every batch cost: 0.41820716857910156**************************************************every batch cost: 0.3943486213684082Epoch 1/50: 56%|█████▋ | 22/39 [01:19<00:50, 2.97s/it, lr=0.001, step/s=0.431, total_loss=645]**************************************************every batch cost: 0.43925046920776367**************************************************every batch cost: 0.4390239715576172Epoch 1/50: 62%|██████▏ | 24/39 [01:25<00:44, 2.94s/it, lr=0.001, step/s=0.382, total_loss=603]**************************************************every batch cost: 0.39272522926330566Epoch 1/50: 64%|██████▍ | 25/39 [01:32<00:56, 4.01s/it, lr=0.001, step/s=6.5, total_loss=584]**************************************************every batch cost: 0.4005303382873535Epoch 1/50: 67%|██████▋ | 26/39 [01:32<00:38, 2.93s/it, lr=0.001, step/s=0.413, total_loss=566]**************************************************every batch cost: 0.4216008186340332**************************************************every batch cost: 0.3795938491821289Epoch 1/50: 72%|███████▏ | 28/39 [01:40<00:34, 3.10s/it, lr=0.001, step/s=0.52, total_loss=533]**************************************************every batch cost: 0.5263004302978516Epoch 1/50: 74%|███████▍ | 29/39 [01:47<00:41, 4.11s/it, lr=0.001, step/s=6.44, total_loss=518]**************************************************every batch cost: 0.3753941059112549**************************************************every batch cost: 0.3916609287261963Epoch 1/50: 79%|███████▉ | 31/39 [01:54<00:34, 4.30s/it, lr=0.001, step/s=7.34, total_loss=490]**************************************************every batch cost: 0.4356727600097656**************************************************every batch cost: 0.42064833641052246Epoch 1/50: 85%|████████▍ | 33/39 [02:02<00:25, 4.23s/it, lr=0.001, step/s=6.79, total_loss=465]**************************************************every batch cost: 0.44506239891052246Epoch 1/50: 87%|████████▋ | 34/39 [02:02<00:15, 3.10s/it, lr=0.001, step/s=0.427, total_loss=454]**************************************************every batch cost: 0.44188857078552246**************************************************every batch cost: 0.4287247657775879Epoch 1/50: 92%|█████████▏| 36/39 [02:09<00:09, 3.11s/it, lr=0.001, step/s=0.408, total_loss=432]**************************************************every batch cost: 0.41641998291015625Epoch 1/50: 95%|█████████▍| 37/39 [02:16<00:08, 4.04s/it, lr=0.001, step/s=6.2, total_loss=422]**************************************************every batch cost: 0.4182100296020508Epoch 1/50: 97%|█████████▋| 38/39 [02:16<00:02, 2.96s/it, lr=0.001, step/s=0.414, total_loss=413]**************************************************every batch cost: 0.42250919342041016Epoch 1/50: 100%|██████████| 39/39 [02:22<00:00, 3.95s/it, lr=0.001, step/s=6.25, total_loss=404]**************************************************every batch cost: 0.3862333297729492Epoch 1/50: 100%|██████████| 39/39 [02:22<00:00, 3.67s/it, lr=0.001, step/s=6.25, total_loss=404]Epoch 1/50: 0%|| 0/4 [00:00<?, ?it/s<class 'dict'>]Start ValidationEpoch 1/50: 100%|██████████| 4/4 [00:09<00:00, 2.34s/it, total_loss=55]Finish ValidationEpoch:1/50Total Loss: 393.7135 || Val Loss: 43.9618 Saving state, iter: 1Epoch 2/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 9.354033470153809Epoch 2/50: 3%|▎ | 1/39 [00:09<06:12, 9.81s/it, lr=0.000905, step/s=9.81, total_loss=57.9]**************************************************every batch cost: 0.46102428436279297Epoch 2/50: 5%|▌ | 2/39 [00:10<04:19, 7.01s/it, lr=0.000905, step/s=0.443, total_loss=56.8]**************************************************every batch cost: 0.44997072219848633Total Loss: 37.1114 || Val Loss: 19.7967 Saving state, iter: 2Epoch 3/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 8.828293561935425Saving state, iter: 3Epoch 4/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************8.770357131958008Epoch 4/50: 3%|▎ num_workers=1, pin_memory=True###############################################Epoch 1/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 7.561102628707886Epoch 1/50: 3%|▎ | 1/39 [00:08<05:39, 8.93s/it, lr=0.001, step/s=8.93, total_loss=1.47e+3]**************************************************every batch cost: 1.3696913719177246**************************************************every batch cost: 0.4643228054046631Epoch 1/50: 8%|▊ | 3/39 [00:21<04:34, 7.62s/it, lr=0.001, step/s=6.52, total_loss=1.3e+3]**************************************************every batch cost: 0.41152310371398926Epoch 1/50: 10%|█ | 4/39 [00:28<04:17, 7.36s/it, lr=0.001, step/s=6.74, total_loss=1.22e+3]**************************************************every batch cost: 0.4380664825439453Epoch 1/50: 13%|█▎ | 5/39 [00:34<04:02, 7.13s/it, lr=0.001, step/s=6.57, total_loss=1.15e+3]**************************************************every batch cost: 0.39557337760925293Epoch 1/50: 15%|█▌ | 6/39 [00:41<03:51, 7.01s/it, lr=0.001, step/s=6.72, total_loss=1.09e+3]**************************************************every batch cost: 0.43558669090270996Epoch 1/50: 18%|█▊ | 7/39 [00:48<03:41, 6.93s/it, lr=0.001, step/s=6.73, total_loss=1.03e+3]**************************************************every batch cost: 0.47762060165405273Epoch 1/50: 21%|██ | 8/39 [00:54<03:30, 6.79s/it, lr=0.001, step/s=6.47, total_loss=972]**************************************************every batch cost: 0.450481653213501**************************************************every batch cost: 0.6359219551086426Epoch 1/50: 26%|██▌ | 10/39 [01:07<03:10, 6.58s/it, lr=0.001, step/s=6.19, total_loss=875]**************************************************every batch cost: 0.3963806629180908**************************************************every batch cost: 0.40274739265441895Epoch 1/50: 31%|███ | 12/39 [01:21<03:00, 6.67s/it, lr=0.001, step/s=6.77, total_loss=792]**************************************************every batch cost: 0.43964362144470215Epoch 1/50: 33%|███▎| 13/39 [01:27<02:54, 6.71s/it, lr=0.001, step/s=6.79, total_loss=755]**************************************************every batch cost: 0.42827844619750977**************************************************every batch cost: 0.502422571182251Epoch 1/50: 38%|███▊| 15/39 [01:41<02:40, 6.67s/it, lr=0.001, step/s=6.73, total_loss=690]**************************************************every batch cost: 0.5271012783050537**************************************************every batch cost: 0.5445053577423096Epoch 1/50: 44%|████▎| 17/39 [01:54<02:28, 6.73s/it, lr=0.001, step/s=6.84, total_loss=634]**************************************************every batch cost: 0.5539577007293701Epoch 1/50: 46%|████▌| 18/39 [02:01<02:22, 6.80s/it, lr=0.001, step/s=6.95, total_loss=609]**************************************************every batch cost: 0.5347979068756104Epoch 1/50: 49%|████▊| 19/39 [02:08<02:13, 6.69s/it, lr=0.001, step/s=6.42, total_loss=586]**************************************************every batch cost: 0.5063800811767578**************************************************every batch cost: 0.5031697750091553Epoch 1/50: 54%|█████▍ | 21/39 [02:21<02:00, 6.70s/it, lr=0.001, step/s=6.78, total_loss=544]**************************************************every batch cost: 0.6011106967926025Epoch 1/50: 56%|█████▋ | 22/39 [02:28<01:53, 6.70s/it, lr=0.001, step/s=6.69, total_loss=525]**************************************************every batch cost: 0.5622284412384033**************************************************every batch cost: 0.5850467681884766Epoch 1/50: 62%|██████▏ | 24/39 [02:41<01:41, 6.75s/it, lr=0.001, step/s=6.76, total_loss=491]**************************************************every batch cost: 0.498795747756958Epoch 1/50: 64%|██████▍ | 25/39 [02:48<01:34, 6.78s/it, lr=0.001, step/s=6.86, total_loss=475]**************************************************every batch cost: 0.5354490280151367Epoch 1/50: 67%|██████▋ | 26/39 [02:55<01:28, 6.82s/it, lr=0.001, step/s=6.89, total_loss=460]**************************************************every batch cost: 0.5751655101776123Epoch 1/50: 69%|██████▉ | 27/39 [03:02<01:23, 6.92s/it, lr=0.001, step/s=7.16, total_loss=447]**************************************************every batch cost: 0.615984312744Epoch 1/50: 72%|███████▏ | 28/39 [03:09<01:14, 6.74s/it, lr=0.001, step/s=6.31, total_loss=434]**************************************************every batch cost: 0.38555479049682617**************************************************every batch cost: 0.42298007011413574Epoch 1/50: 77%|███████▋ | 30/39 [03:22<01:00, 6.77s/it, lr=0.001, step/s=6.72, total_loss=410]**************************************************every batch cost: 0.3904249668121338Epoch 1/50: 79%|███████▉ | 31/39 [03:29<00:54, 6.76s/it, lr=0.001, step/s=6.73, total_loss=399]**************************************************every batch cost: 0.44513821601867676Epoch 1/50: 82%|████████▏ | 32/39 [03:36<00:47, 6.78s/it, lr=0.001, step/s=6.82, total_loss=389]**************************************************every batch cost: 0.3929564952850342**************************************************every batch cost: 0.4448988437652588Epoch 1/50: 87%|████████▋ | 34/39 [03:49<00:33, 6.72s/it, lr=0.001, step/s=6.61, total_loss=369]**************************************************every batch cost: 0.40523266792297363**************************************************every batch cost: 0.3851611614227295Epoch 1/50: 92%|█████████▏| 36/39 [04:03<00:20, 6.76s/it, lr=0.001, step/s=6.79, total_loss=352]**************************************************every batch cost: 0.43050265312194824**************************************************every batch cost: 0.41400766372680664Epoch 1/50: 97%|█████████▋| 38/39 [04:16<00:06, 6.67s/it, lr=0.001, step/s=6.65, total_loss=336]**************************************************every batch cost: 0.4372570514678955Epoch 1/50: 100%|██████████| 39/39 [04:22<00:00, 6.60s/it, lr=0.001, step/s=6.43, total_loss=329]**************************************************every batch cost: 0.37398743629455566Epoch 1/50: 100%|██████████| 39/39 [04:22<00:00, 6.74s/it, lr=0.001, step/s=6.43, total_loss=329]Epoch 1/50: 0%|| 0/4 [00:00<?, ?it/s<class 'dict'>]Start ValidationEpoch 1/50: 100%|██████████| 4/4 [00:14<00:00, 3.59s/it, total_loss=43.1]Finish ValidationEpoch:1/50Total Loss: 320.5249 || Val Loss: 34.4960 Saving state, iter: 1Epoch 2/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 8.003284931182861**************************************************every batch cost: 0.5475771427154541Epoch 2/50: 5%|▌ | 2/39 [00:15<04:59, 8.10s/it, lr=0.000905, step/s=7.03, total_loss=46.8]**************************************************every batch cost: 0.5343887805938721Epoch 2/50: 8%|▊ | 3/39 [00:22<04:37, 7.70s/it, lr=0.000905, step/s=6.78, total_loss=45.9]**************************************************every batch cost: 0.5383634567260742Epoch 2/50: 10%|█ | 4/39 [00:29<04:19, 7.40s/it, lr=0.000905, step/s=6.7, total_loss=45.1]**************************************************every batch cost: 0.5001688003540039**************************************************every batch cost: 0.571533203125Epoch 2/50: 15%|█▌ | 6/39 [00:42<03:55, 7.14s/it, lr=0.000905, step/s=6.8, total_loss=43.6]**************************************************every batch cost: 0.49525880813598633Epoch 2/50: 18%|█▊ | 7/39 [00:49<03:44, 7.02s/it, lr=0.000905, step/s=6.73, total_loss=43]**************************************************every batch cost: 0.40282368659973145**************************************************every batch cost: 0.40811753273010254Epoch 2/50: 23%|██▎ | 9/39 [01:03<03:30, 7.01s/it, lr=0.000905, step/s=7.09, total_loss=41.8]**************************************************every batch cost: 0.4412691593170166**************************************************every batch cost: 0.37858128547668457Epoch 2/50: 28%|██▊ | 11/39 [01:17<03:14, 6.96s/it, lr=0.000905, step/s=6.9, total_loss=40.7]**************************************************every batch cost: 0.42623257637023926Epoch 2/50: 31%|███ | 12/39 [01:24<03:09, 7.03s/it, lr=0.000905, step/s=7.19, total_loss=40.3]**************************************************every batch cost: 0.45459723472595215**************************************************every batch cost: 0.49076294898986816Epoch 2/50: 36%|███▌| 14/39 [01:38<02:51, 6.87s/it, lr=0.000905, step/s=6.7, total_loss=39.3]**************************************************every batch cost: 0.570073127746582**************************************************every batch cost: 0.4507887363433838Epoch 2/50: 41%|████| 16/39 [01:51<02:36, 6.80s/it, lr=0.000905, step/s=6.73, total_loss=38.4]**************************************************every batch cost: 0.41584324836730957Epoch 2/50: 44%|████▎| 17/39 [01:58<02:29, 6.80s/it, lr=0.000905, step/s=6.77, total_loss=37.9]**************************************************every batch cost: 0.3880279064178467Epoch 2/50: 46%|████▌| 18/39 [02:04<02:22, 6.76s/it, lr=0.000905, step/s=6.68, total_loss=37.6]**************************************************every batch cost: 0.4323999881744385Epoch 2/50: 49%|████▊| 19/39 [02:11<02:15, 6.78s/it, lr=0.000905, step/s=6.82, total_loss=37.2]**************************************************every batch cost: 0.4374668598175049**************************************************every batch cost: 0.38514113426208496Epoch 2/50: 54%|█████▍ | 21/39 [02:25<02:02, 6.81s/it, lr=0.000905, step/s=6.71, total_loss=36.5]**************************************************every batch cost: 0.46169018745422363**************************************************every batch cost: 0.442759613037Epoch 2/50: 59%|█████▉ | 23/39 [02:39<01:50, 6.88s/it, lr=0.000905, step/s=6.9, total_loss=35.8]**************************************************every batch cost: 0.42540407180786133**************************************************every batch cost: 0.44600915908813477Epoch 2/50: 64%|██████▍ | 25/39 [02:53<01:35, 6.85s/it, lr=0.000905, step/s=6.9, total_loss=35.2]**************************************************every batch cost: 0.3933718204498291**************************************************every batch cost: 0.4039616584777832Epoch 2/50: 69%|██████▉ | 27/39 [03:06<01:21, 6.77s/it, lr=0.000905, step/s=6.72, total_loss=34.6]**************************************************every batch cost: 0.40546417236328125**************************************************every batch cost: 0.3811759948730469Epoch 2/50: 74%|███████▍ | 29/39 [03:19<01:07, 6.75s/it, lr=0.000905, step/s=6.8, total_loss=33.9]**************************************************every batch cost: 0.3452949523925781**************************************************every batch cost: 0.39229869842529297Epoch 2/50: 79%|███████▉ | 31/39 [03:33<00:54, 6.81s/it, lr=0.000905, step/s=6.76, total_loss=33.4]**************************************************every batch cost: 0.3768949508666992Epoch 2/50: 82%|████████▏ | 32/39 [03:40<00:47, 6.84s/it, lr=0.000905, step/s=6.89, total_loss=33.1]**************************************************every batch cost: 0.39880943298339844**************************************************every batch cost: 0.45096421241760254Epoch 2/50: 87%|████████▋ | 34/39 [03:54<00:34, 6.84s/it, lr=0.000905, step/s=6.78, total_loss=32.6]**************************************************every batch cost: 0.4104328155517578Epoch 2/50: 90%|████████▉ | 35/39 [04:01<00:27, 6.84s/it, lr=0.000905, step/s=6.83, total_loss=32.3]**************************************************every batch cost: 0.4044673442840576Epoch 2/50: 92%|█████████▏| 36/39 [04:07<00:20, 6.75s/it, lr=0.000905, step/s=6.54, total_loss=32]**************************************************every batch cost: 0.3817863464355469Epoch 2/50: 95%|█████████▍| 37/39 [04:14<00:13, 6.70s/it, lr=0.000905, step/s=6.58, total_loss=31.8]**************************************************every batch cost: 0.3974142074584961Epoch 2/50: 97%|█████████▋| 38/39 [04:21<00:06, 6.73s/it, lr=0.000905, step/s=6.79, total_loss=31.6]**************************************************every batch cost: 0.3678750991821289**************************************************every batch cost: 0.40433621406555176Epoch 2/50: 100%|██████████| 39/39 [04:28<00:00, 6.87s/it, lr=0.000905, step/s=6.91, total_loss=31.3]Start ValidationEpoch 2/50: 100%|██████████| 4/4 [00:13<00:00, 3.50s/it, total_loss=20.4]Finish ValidationEpoch:2/50Total Loss: 30.5174 || Val Loss: 16.3263 Saving state, iter: 2Epoch 3/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 7.984367370605469**************************************************every batch cost: 0.63826584815979Epoch 3/50: 5%|▌ | 2/39 [00:15<04:56, 8.01s/it, lr=0.000658, step/s=6.56, total_loss=22]**************************************************every batch cost: 0.4317927360534668Epoch 3/50: 8%|▊ | 3/39 [00:22<04:36, 7.67s/it, lr=0.000658, step/s=6.88, total_loss=21.8]**************************************************every batch cost: 0.4045250415802002Epoch 3/50: 10%|█ | 4/39 [00:29<04:23, 7.53s/it, lr=0.000658, step/s=7.17, total_loss=21.9]**************************************************every batch cost: 0.5850446224212646**************************************************every batch cost: 0.45392608642578125Epoch 3/50: 15%|█▌ | 6/39 [00:42<03:52, 7.05s/it, lr=0.000658, step/s=6.54, total_loss=21.7]**************************************************every batch cost: 0.42398643493652344Epoch 3/50: 18%|█▊ | 7/39 [00:49<03:44, 7.00s/it, lr=0.000658, step/s=6.9, total_loss=21.5]**************************************************every batch cost: 0.5161738395690918**************************************************every batch cost: 0.4052703380584717Epoch 3/50: 23%|██▎ | 9/39 [01:02<03:24, 6.82s/it, lr=0.000658, step/s=6.72, total_loss=21.3]**************************************************every batch cost: 0.40622735023498535**************************************************every batch cost: 0.4583563804626465Epoch 3/50: 28%|██▊ | 11/39 [01:16<03:11, 6.84s/it, lr=0.000658, step/s=6.75, total_loss=21.3]**************************************************every batch cost: 0.45088720321655273**************************************************every batch cost: 0.41858887672424316Epoch 3/50: 33%|███▎| 13/39 [01:29<02:57, 6.81s/it, lr=0.000658, step/s=6.73, total_loss=20.9]**************************************************every batch cost: 0.5316376686096191Epoch 3/50: 36%|███▌| 14/39 [01:36<02:50, 6.81s/it, lr=0.000658, step/s=6.8, total_loss=20.8]**************************************************every batch cost: 0.43830227851867676Epoch 3/50: 38%|███▊| 15/39 [01:43<02:45, 6.92s/it, lr=0.000658, step/s=7.15, total_loss=20.7]**************************************************every batch cost: 0.5127363204956055**************************************************every batch cost: 0.5016570091247559Epoch 3/50: 44%|████▎| 17/39 [01:57<02:31, 6.88s/it, lr=0.000658, step/s=6.71, total_loss=20.6]**************************************************every batch cost: 0.541719012451**************************************************every batch cost: 0.48558616638183594Epoch 3/50: 49%|████▊| 19/39 [02:11<02:18, 6.94s/it, lr=0.000658, step/s=7.08, total_loss=20.4]**************************************************every batch cost: 0.5570189952850342**************************************************every batch cost: 0.5443787574768066Epoch 3/50: 54%|█████▍ | 21/39 [02:24<02:02, 6.80s/it, lr=0.000658, step/s=6.59, total_loss=20.2]**************************************************every batch cost: 0.5134713649749756**************************************************every batch cost: 0.5383806228637695Epoch 3/50: 59%|█████▉ | 23/39 [02:38<01:50, 6.89s/it, lr=0.000658, step/s=6.94, total_loss=20.2]**************************************************every batch cost: 0.600865364074707Epoch 3/50: 62%|██████▏ | 24/39 [02:46<01:44, 6.94s/it, lr=0.000658, step/s=7.03, total_loss=20.1]**************************************************every batch cost: 0.5180280208587646Epoch 3/50: 64%|██████▍ | 25/39 [02:52<01:36, 6.90s/it, lr=0.000658, step/s=6.82, total_loss=20]**************************************************every batch cost: 0.5188186168670654Epoch 3/50: 67%|██████▋ | 26/39 [02:59<01:29, 6.90s/it, lr=0.000658, step/s=6.88, total_loss=19.9]**************************************************every batch cost: 0.446591854095459**************************************************every batch cost: 0.614121675491333Epoch 3/50: 72%|███████▏ | 28/39 [03:13<01:16, 6.96s/it, lr=0.000658, step/s=6.91, total_loss=19.8]**************************************************every batch cost: 0.5028769969940186Epoch:3/50Total Loss: 18.5234 || Val Loss: 11.3431 Saving state, iter: 3Epoch 4/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 8.067720651626587Epoch 4/50: 3%|▎ | 1/39 [00:08<05:23, 8.51s/it, lr=0.000352, step/s=8.51, total_loss=16.7]**************************************************every batch cost: 0.4458169937133789num_workers=0, pin_memory=True,###############################################Loading weights into state dict...Finished!**************************************************0.00026345252990722656Epoch 1/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 6.736436128616333**************************************************every batch cost: 1.3526716232299805Epoch 1/50: 5%|▌ | 2/39 [00:14<04:44, 7.70s/it, lr=0.001, step/s=6.78, total_loss=1.64e+3]**************************************************every batch cost: 0.4258887767791748**************************************************every batch cost: 0.5254902839660645Epoch 1/50: 10%|█ | 4/39 [00:28<04:12, 7.21s/it, lr=0.001, step/s=6.68, total_loss=1.46e+3]**************************************************every batch cost: 0.4723479747772217Epoch 1/50: 13%|█▎ | 5/39 [00:34<03:57, 6.98s/it, lr=0.001, step/s=6.42, total_loss=1.38e+3]**************************************************every batch cost: 0.3637819290161133**************************************************every batch cost: 0.39740514755249023Epoch 1/50: 18%|█▊ | 7/39 [00:48<03:37, 6.81s/it, lr=0.001, step/s=6.63, total_loss=1.23e+3]**************************************************every batch cost: 0.3908064365386963**************************************************every batch cost: 0.4139695167541504Epoch 1/50: 23%|██▎ | 9/39 [01:01<03:24, 6.83s/it, lr=0.001, step/s=6.78, total_loss=1.11e+3]**************************************************every batch cost: 0.3817429542541504Epoch 1/50: 26%|██▌ | 10/39 [01:08<03:15, 6.75s/it, lr=0.001, step/s=6.58, total_loss=1.05e+3]**************************************************every batch cost: 0.35744786262512207**************************************************every batch cost: 0.45519328117370605Epoch 1/50: 31%|███ | 12/39 [01:22<03:03, 6.78s/it, lr=0.001, step/s=6.75, total_loss=955]**************************************************every batch cost: 0.389538049697876Epoch 1/50: 33%|███▎| 13/39 [01:28<02:56, 6.77s/it, lr=0.001, step/s=6.74, total_loss=912]**************************************************every batch cost: 0.3983135223388672**************************************************every batch cost: 0.4398193359375Epoch 1/50: 38%|███▊| 15/39 [01:42<02:42, 6.79s/it, lr=0.001, step/s=6.76, total_loss=834]**************************************************every batch cost: 0.42572808265686035**************************************************every batch cost: 0.4335670471191406Epoch 1/50: 44%|████▎| 17/39 [01:56<02:29, 6.81s/it, lr=0.001, step/s=6.8, total_loss=767]**************************************************every batch cost: 0.4214038848876953Epoch 1/50: 46%|████▌| 18/39 [02:02<02:21, 6.76s/it, lr=0.001, step/s=6.63, total_loss=737]**************************************************every batch cost: 0.3599708080291748**************************************************every batch cost: 0.4076557159423828Epoch 1/50: 51%|█████▏ | 20/39 [02:16<02:08, 6.78s/it, lr=0.001, step/s=6.9, total_loss=683]**************************************************every batch cost: 0.432664155960083Epoch 1/50: 54%|█████▍ | 21/39 [02:22<02:01, 6.75s/it, lr=0.001, step/s=6.66, total_loss=658]**************************************************every batch cost: 0.4187169075012207**************************************************every batch cost: 0.4376835823059082Epoch 1/50: 59%|█████▉ | 23/39 [02:36<01:48, 6.80s/it, lr=0.001, step/s=6.8, total_loss=614]**************************************************every batch cost: 0.41230297088623047Epoch 1/50: 62%|██████▏ | 24/39 [02:43<01:41, 6.78s/it, lr=0.001, step/s=6.71, total_loss=594]**************************************************every batch cost: 0.42777347564697266**************************************************every batch cost: 0.416337251663208Epoch 1/50: 67%|██████▋ | 26/39 [02:57<01:29, 6.85s/it, lr=0.001, step/s=6.91, total_loss=558]**************************************************every batch cost: 0.43065834045410156Epoch 1/50: 69%|██████▉ | 27/39 [03:04<01:22, 6.86s/it, lr=0.001, step/s=6.87, total_loss=541]**************************************************every batch cost: 0.4243924617767334Epoch 1/50: 72%|███████▏ | 28/39 [03:10<01:15, 6.84s/it, lr=0.001, step/s=6.79, total_loss=525]**************************************************every batch cost: 0.4390888214111328Epoch 1/50: 74%|███████▍ | 29/39 [03:17<01:08, 6.87s/it, lr=0.001, step/s=6.94, total_loss=510]**************************************************every batch cost: 0.44049930572509766**************************************************every batch cost: 0.4194369316101074Epoch 1/50: 79%|███████▉ | 31/39 [03:31<00:54, 6.85s/it, lr=0.001, step/s=6.79, total_loss=483]**************************************************every batch cost: 0.416440486907959Epoch 1/50: 82%|████████▏ | 32/39 [03:38<00:47, 6.85s/it, lr=0.001, step/s=6.83, total_loss=470]**************************************************every batch cost: 0.45289063453674316**************************************************every batch cost: 0.43844008445739746Epoch 1/50: 87%|████████▋ | 34/39 [03:52<00:34, 6.88s/it, lr=0.001, step/s=6.78, total_loss=447]**************************************************every batch cost: 0.4023940563043Epoch 1/50: 90%|████████▉ | 35/39 [03:59<00:27, 6.87s/it, lr=0.001, step/s=6.82, total_loss=436]**************************************************every batch cost: 0.3687868118286133**************************************************every batch cost: 0.43868470191955566Epoch 1/50: 95%|█████████▍| 37/39 [04:12<00:13, 6.82s/it, lr=0.001, step/s=6.65, total_loss=416]**************************************************every batch cost: 0.39241647720336914**************************************************every batch cost: 0.4395029544830322Epoch 1/50: 100%|██████████| 39/39 [04:26<00:00, 6.83s/it, lr=0.001, step/s=6.69, total_loss=398]Epoch 1/50: 0%|| 0/4 [00:00<?, ?it/s<class 'dict'>]**************************************************every batch cost: 0.4003739356994629Start ValidationEpoch 1/50: 100%|██████████| 4/4 [00:12<00:00, 3.01s/it, total_loss=53.4]Finish ValidationEpoch:1/50Total Loss: 387.7016 || Val Loss: 42.7095 Saving state, iter: 1Epoch 2/50: 0%|| 0/39 [00:00<?, ?it/s<class 'dict'>]**************************************************load data cost: 6.612175464630127**************************************************every batch cost: 0.4096841812133789Epoch 2/50: 5%|▌ | 2/39 [00:13<04:17, 6.97s/it, lr=0.000905, step/s=6.84, total_loss=53.3]**************************************************every batch cost: 0.4010329246520996Epoch 2/50: 8%|▊ | 3/39 [00:20<04:09, 6.92s/it, lr=0.000905, step/s=6.79, total_loss=52.8]**************************************************every batch cost: 0.40485644340515137**************************************************every batch cost: 0.3858015537261963
