U.S. Sen. Cory Gardner said he’s lifting most of the holds he’s put on President Trump’s Justice Department nominees over Attorney General Jeff Sessions’ threats to crackdown on legal marijuana.

Gardner spoke by phone with Colorado Politics and a handful of other Colorado reporters Thursday morning to say he had made progress in negotiations with Assistant Attorney General Rod Rosenstein.

“Most importantly it’s about making sure we have a solution for Colorado that protects states’ rights,” the Republican senator from Yuma said. “And I think that’s where we’re heading, and I’m grateful for the Department of Justice moving in a direction I think will indeed protect those states’ rights.”

Sessions rescinded the Cole Memo in January. The Obama administration directive had allowed states such as Colorado, where voters legalized marijuana, to operate outside of federal oversight as long they adequately regulate marijuana.

Sessions, a former U.S. senator and Alabama attorney general, has long opposed marijuana legalization – an industry that made $1.5 billion in sales last year, according to the state Department of Revenue.

Sessions called out Gardner, though not by name, at a law enforcement conference on Monday for making the job of the Justice Department harder without key appointments. On Tuesday Gardner’s office said he would not give in without more assurances for those who have invested in the industry that generates hundreds of millions of dollars in taxes and employs thousands in Colorado.

Gardner said Thursday he would no longer block appointments of U.S. marshals, U.S. attorneys and the assistant attorney general for national security, which together comprise the bulk of the department’s vacancies.

He said the others are contingent on pot talks progressing.

Gardner said he hopes to work on legislation to give states where marijuana is already legal more solid ground to stand on apart from federal laws, which deem it illegal.

“We’ll lift the others as soon as they follow through with the actions that we have been talking about,’ Gardner said.

The Colorado Democratic Party, no fan of Gardner, said the senator from Yuma vacillated.

“This is exactly what Colorado voters have come to expect from Sen. Cory Gardner: all hat, no cattle,” said party spokesman Eric Walker in a statement.”The fact that Gardner surrendered his leverage to protect Colorado’s legal marijuana industry in exchange for vague promises from a proven liar shows that he’s not just a pushover but a fool as well.”

He said legislation would be similar to a provision Sen. Patrick Leahy, D-Vermont, has proposed to exempt medical marijuana from federal law.

Leahy has argued that the Department of Justice should be more focused on larger threats.

“We’d look at something very similar in other aspects of the industry,” Gardner said on the call with reporters. “That’s legislation I’m talking to my colleagues about, some of whom have already introduced such legislation that would protect states’ rights and entrust a process that the U.S. Constitutions prescribes.”

Gardner opposed marijuana legalization when it was before Colorado voters in 2012, but he said he respects the authority of voters to decide such issues in their states, as well as the industry that has grown up around government assurances.

On Thursday libertarian-leaning former Sen. Ron Paul took a similar view as his colleagues, the liberal from Vermont and the conservative from Colorado’s Eastern Plains.

“President Trump and the rest of the GOP would be foolish to stand with Sessions,” Paul wrote in regard to medical marijuana in an op-ed in Time Magazine posted Thursday.