xg15 2 days ago (2021), still very interesting. Especially the "post-overfitting" training strategy is unexpected. dev_hugepages 2 days ago This is talking about the double descent phenomenon (https://en.wikipedia.org/wiki/Double_descent) luckystarr 2 days ago I remember vaguely that this was observed when training GPT-3 (probably?) as well. Just trained on and on, and the error went up and then down again. Like a phase transition in the model.
dev_hugepages 2 days ago This is talking about the double descent phenomenon (https://en.wikipedia.org/wiki/Double_descent)
luckystarr 2 days ago I remember vaguely that this was observed when training GPT-3 (probably?) as well. Just trained on and on, and the error went up and then down again. Like a phase transition in the model.
(2021), still very interesting. Especially the "post-overfitting" training strategy is unexpected.
This is talking about the double descent phenomenon (https://en.wikipedia.org/wiki/Double_descent)
I remember vaguely that this was observed when training GPT-3 (probably?) as well. Just trained on and on, and the error went up and then down again. Like a phase transition in the model.
The low sample efficiency of RL is well explained.