Multivariate Bernoulli distribution

In this paper, we consider the multivariate Bernoulli distribution as a model to estimate the structure of graphs with binary nodes. This distribution is discussed in the framework of the exponential family, and its statistical properties regarding independence of the nodes are demonstrated. Importantly the model can estimate not only the main effects and pairwise interactions among the nodes but also is capable of modeling higher order interactions, allowing for the existence of complex clique effects. We compare the multivariate Bernoulli model with existing graphical inference models - the Ising model and the multivariate Gaussian model, where only the pairwise interactions are considered. On the other hand, the multivariate Bernoulli distribution has an interesting property in that independence and uncorrelatedness of the component random variables are equivalent. Both the marginal and conditional distributions of a subset of variables in the multivariate Bernoulli distribution still follow the multivariate Bernoulli distribution. Furthermore, the multivariate Bernoulli logistic model is developed under generalized linear model theory by utilizing the canonical link function in order to include covariate information on the nodes, edges and cliques. We also consider variable selection techniques such as LASSO in the logistic model to impose sparsity structure on the graph. Finally, we discuss extending the smoothing spline ANOVA approach to the multivariate Bernoulli logistic model to enable estimation of non-linear effects of the predictor variables.

Learning Undirected Graphical Models with Structure Penalty

In undirected graphical models, learning the graph structure and learning the functions that relate the predictive variables (features) to the responses given the structure are two topics that have been widely investigated in machine learning and statistics. Learning graphical models in two stages will have problems because graph structure may change after considering the features. The main contribution of this paper is the proposed method that learns the graph structure and functions on the graph at the same time. General graphical models with binary outcomes conditioned on predictive variables are proved to be equivalent to multivariate Bernoulli model. The reparameterization of the potential functions in graphical model by conditional log odds ratios in multivariate Bernoulli model offers advantage in the representation of the conditional independence structure in the model. Additionally, we impose a structure penalty on groups of conditional log odds ratios to learn the graph structure. These groups of functions are designed with overlaps to enforce hierarchical function selection. In this way, we are able to shrink higher order interactions to obtain a sparse graph structure. Simulation studies show that the method is able to recover the graph structure. The analysis of county data from Census Bureau gives interesting relations between unemployment rate, crime and others discovered by the model.