Semantic objective functions: A distribution-aware method for adding logical constraints in deep learning

Issues of safety, explainability, and efficiency are of increasing concern in learning systems deployed with hard and soft constraints. Loss-function based techniques have shown promising results in this area, by embedding logical constraints during neural network training. Through an integration of logic and information geometry, we provide a construction and theoretical framework for these tasks that generalize many approaches. We propose a loss-based method that embeds knowledge—enforces logical constraints—into a machine learning model that outputs probability distributions. This is done by constructing a distribution from the logical formula, and constructing a loss function as a linear combination of the original loss function with the Fisher-Rao distance or Kullback-Leibler divergence to the constraint distribution. This construction is primarily for logical constraints in the form of propositional formulas (Boolean variables), but can be extended to formulas of a first-order language with finite variables over a model with compact domain (categorical and continuous variables), and others statistical models that is to be trained with semantic information. We evaluate our method on a variety of learning tasks, including classification tasks with logic constraints, transferring knowledge from logic formulas, and knowledge distillation.