Skip to content
This repository has been archived by the owner on Oct 15, 2019. It is now read-only.

Wrong implementation of the xavier initializer #158

Open
wddabc opened this issue Mar 12, 2017 · 0 comments
Open

Wrong implementation of the xavier initializer #158

wddabc opened this issue Mar 12, 2017 · 0 comments

Comments

@wddabc
Copy link

wddabc commented Mar 12, 2017

The Xavier init in https://github.com/dmlc/minpy/blob/master/minpy/nn/init.py#L33
is wrong. According to Glorot et al (2010) Eq.(1) and Eq.(16), the weights are sampled from a uniform dist, not a normal dist (I assume the npr.randn is a normal dist as numpy.random does).

I tried to change L33 to:
ret = npr.uniform(low=-var, high=var, size=shape)
But it caused another exception in MXNet, this might be another issue

Updated: Even the above is fixed, it is still wrong as the Xavier initialization depends on different activation functions. See clab/dynet#295 and clab/dynet#348 for discussions.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant