He also said the Justice Department is taking steps to identify practices for enhancing the use of drug treatment and community service programs as alternatives to jail.
Holder also said he has asked federal prosecutors to develop new guidelines for determining when federal charges should be filed and when they should not.
"I've also issued guidance to ensure that every case we bring serves a substantial federal interest and complements the work of our law enforcement partners," he said.
Holder additionally directed prosecutors to create comprehensive anti-violence strategies for badly afflicted areas.
The American Civil Liberties Union praised Holder's approach Monday, calling it an important step toward ending federal prison overcrowding and creating a "fairer criminal justice system."
Laura Murphy, director of the ACLU's Washington legislative office, said in a news release, however, that although Holder's announcement "is an important first step," Congress also must act to change laws that "lock up hundreds of thousands of Americans unfairly and unnecessarily."
Michael Mukasey, a former judge and an attorney general under President George W. Bush, said he is not a fan of mandatory minimums, but he does not support what Holder is doing.
"I generally agree with the goal of getting rid of mandatory minimums," said Mukasey. "But the way to do that is to pass a law," he said in an interview on CNN's "The Lead with Jake Tapper."