Probabilistic neural architectures that explicitly represent and manipulate discrete probability mass functions across nodes or variables to enable tractable modeling, generation, and inference over categorical, ordinal, or count-valued data.
Discrete Distribution Networks occupy the intersection of classical discrete graphical models and modern deep generative modeling: they are architectures and design patterns for encoding, transforming and predicting distributions defined on discrete sample spaces (e.g., categorical variables, permutations, integer counts, graphs). Unlike continuous-flow or Gaussian-based models, DDNs address challenges of non-differentiability and combinatorial state spaces by combining structured factorization (autoregressive decompositions, factor graphs, and message-passing), discrete invertible transforms, and gradient-estimation or relaxation techniques (e.g., Gumbel-softmax relaxations, score-function estimators, or surrogate continuous surjections). Practical implementations include autoregressive DDNs for language and categorical sequence modeling, discrete normalizing-flow variants for exact-likelihood discrete generative models, architectures for permutation and matching problems using differentiable relaxations (Sinkhorn layers), and hybrid latent-variable models that couple discrete latents with neural decoders. In ML and AI applications, DDNs are significant for structured prediction, combinatorial optimization, molecular and graph generation, and reinforcement learning with large discrete action spaces, because they provide principled probabilistic outputs, support likelihood-based training or principled variational inference, and can exploit problem-specific combinatorial structure to improve sample efficiency and interpretability.
First appeared in the research literature in the mid-2010s and gained broader traction from roughly 2018–2022 as discrete generative methods, discrete normalizing flows, and improved gradient/relaxation techniques matured and were applied to language, graph, and combinatorial tasks.