The longer you train, the slower it gets | Voters | PerceptiLabs

Powered by Canny

The longer you train, the slower it gets

complete

P

PerceptiLabs

There seems to be a bunch of statistics stacking up and causing the training to go slower and slower, as state.pkl is growing.

Created by Robert Lundberg

September 13, 2021

R

Robert Lundberg

marked this post as

complete

R

Robert Lundberg

marked this post as

in progress

Moving this back to in-progress as it was discovered that the training now is faster but there still are issues with the training slowing down as it progresses.

R

Robert Lundberg

marked this post as

complete

R

Robert Lundberg

marked this post as

in progress

https://dev.azure.com/PerceptiLabs/PerceptiLabs/_workitems/edit/2309

R

Robert Lundberg

Merged in a post:

Training slowing down

P

PerceptiLabs

When training for a long time, it looks like the training speed is slowing down, together with GPU going down and RAM going up.

September 15, 2021

Powered by Canny