Abstract:
We study the expressivity of ReLU neural networks in the setting of a
binary classification problem from a topological perspective. Recently,
empirical studies showed that neural networks operate by changing
topology, transforming a topologically complicated data set into a
topologically simpler one as it passes through the layers. This
topological simplification has been measured by Betti numbers. We use
the same measure to establish lower and upper bounds on the topological
simplification a ReLU neural network can achieve with a given
architecture. We therefore contribute to a better understanding of the
expressivity of ReLU neural networks in the context of binary
classification problems by shedding light on their ability to capture
the underlying topological structure of the data. In particular the
results show that deep ReLU neural networks are exponentially more
powerful than shallow ones in terms of topological simplification.