Element-wise multiplication can out-perform dense layers

We ran 1,300 different training experiments to try a large range of different neural network architectures on basic arithmetic, logic and integer tasks. We tested on four different datasets. Surprisingly, the best performing network was one incorporating element-wise multiplication, an infrequently used component in today’s networks. We encourage further experimentation and usage of element-wise multiplication.

Read the full article

November 11, 2018