Why Did Gary Ross Leave?

Why did Gary Ross leave Catching Fire? It wasn't because of money.

Fox’s January date led the bond company on Catching Fire to insist that Lionsgate commit to finish filming by Dec. 20, backing the studio into a production schedule with an August start date that Ross felt he could not accommodate. (Fox’s deal with Lawrence predates her contract for Hunger Games, putting X-Men in a priority position.)
According to knowledgeable sources, Lionsgate Motion Picture Group co-chair Rob Friedman pressed Fox to set a later start date for the X-Men movie, even offering allurements such as a chance to handle some of the overseas distribution on Catching Fire.

It was this short time frame that made Gary leave the movie.

 Ross, who has had Oscar nominations three times for screenplay (Big, Dave andSeabiscuit), felt that he had to focus on the script for at least two months, leaving him only six to eight weeks to prepare for shooting. Sources also say he feels the second novel is the most challenging in the trilogy to make into a compelling film. Novelist Suzanne Collins, who shares writing credit on The Hunger Games with Ross and Billy Ray, is said to want a screen version that is very faithful to her book.Though the key actors are locked into the Catching Fire sequel, Lions Gate didn’t kick off a negotiation with Ross for a second film until about three weeks before the first one opened on March 23.

So, Gary wanted to do the movie, but he didn't like the short time frame.

Source: The Hollywood Reporter