Maximum Entropy Distribution with Constraint

by jjet   Last Updated August 14, 2019 15:20 PM

I want to find the solution for the maximum entropy distribution with a cost constraint. The specific problem setup is as follows:

  1. Let $\bf{x}$ be a probability distribution.

  2. Let $\bf{c}$ be the cost vector associated with the distribution.

  3. Let $m$ be the maximum allowable cost of the distribution. In other words, $\textbf{c}^\top\textbf{x} = \sum_{i=1}^n c_i x_i \le m$.

  4. I want to maximize the entropy of $\bf{x}$ subject to this constraint. Mathematically, this is equivalent to minimizing, $\textbf{x}^\top \log(\textbf{x}) = \sum_{i=1}^n x_i \log(x_i)$.

I'm struggling to calculate the analytic solution using Lagrangian duality. I'm also unable to implement a numeric solution in Python. Solutions to either of these approaches would be much appreciated.

Related Questions

Constrained Optimization: Lagrange Multipliers

Updated February 24, 2018 23:20 PM

Lagrange multipliers don't seem to work

Updated April 12, 2019 05:20 AM