Thanking Mr Tillerson for his service, Mr Trump tweeted that his new top diplomat would do "a fantastic job".
Mr Tillerson, a former chief executive of ExxonMobil, was only appointed to the job just over a year ago.
The president also nominated Gina Haspel to become the first female director of the CIA.
Speaking to reporters outside the White House on Tuesday, Mr Trump said his differences with Mr Tillerson came down to personal "chemistry".
"We got along actually quite well, but we disagreed on things," said the president.
"When you look at the Iran deal, I think it's terrible.
I guess he thought it was OK.
"I wanted to either break it or do something and he felt a little bit differently, so we were not really thinking the same.
In October, Rex Tillerson responded to a report he called Mr Trump 'a moron' Image caption In October, Rex Tillerson responded to a report he called Mr Trump 'a moron' "With Mike, Mike Pompeo, we have a very similar thought process. I think it's going to go very well.
"Rex is a very good man, I like Rex a lot."
Mr Tillerson is the latest in a long list of senior officials who have either resigned or been fired since Mr Trump took office.
The Department of State said Mr Tillerson had not spoken to the president and was "unaware of the reason" for his firing.
Under Secretary Steve Goldstein said: "The Secretary had every intention of staying because of the critical progress made in national security."
Mr Goldstein himself was fired later on Tuesday by the White House.
According to the Associated Press, White House chief of staff John Kelly called Mr Tillerson on Friday and advised him to watch out for a presidential tweet about him.
Mr Kelly did not tell Mr Tillerson when the tweet would be posted or what it might say, reports the news agency.
The secretary of state was on an official tour of Africa last week when he was caught unawares by Mr Trump's agreement to meet North Korean leader Kim Jong-un.
The top US diplomat's team said he was feeling unwell on Saturday and later in the weekend the state department said he would cut short his tour by a day.
"I felt like, look, I just need to get back," Mr Tillerson told reporters on his plane home, reports the Washington Post.
The 65-year-old arrived back in Washington before dawn on Tuesday to learn he was out of a job a few hours later.
Washington Rex Tillerson, who as a long-time corporate executive was an unconventional pick for secretary of state, seemed like he was on thin ice from very early on in his administration tenure.
He was distrusted by veteran State Department employees, who viewed him as an outside interloper with little affinity for the organisation he headed.
The president, initially enamoured with the brash Texan, quickly appeared to sour on his pick, as he frequently broke with the White House line on foreign policy.
Reportedly calling the president a moron (intensified by a choice expletive) certainly didn't help, either. Mr Tillerson did manage to hang around through those earlier controversies, outliving his earlier expectations that he would be gone by year's end.
Instead he made it to March, but it was clear that this was a move some time in the making, given that Mr Trump already had successor Mike Pompeo lined up to take over the reins.
Just last week Mr Trump tweeted that he had a few more staff changes he wanted to make.
Now it's clear he had one move in particular in mind.
A US Senate report found that Ms Haspel ran a notorious CIA prison in Thailand, where prisoners were waterboarded in 2002.
The CIA deputy director was also accused of destroying dozens of videotapes of interrogations at the camp.
Gina Haspel - first female CIA director nominee Mr Pompeo is a former hardline conservative Republican lawmaker from Kansas and a Trump loyalist.
In 2014, he defended the CIA officers who waterboarded detainees as "patriots".
Reports had swirled since last year of a schism in the Trump administration between the commander-in-chief and his top diplomat.
Last October Mr Tillerson was forced to convene a news conference to deny claims he planned to quit, though he did not address a report that he had referred to his boss as a moron after a Pentagon meeting.
Last autumn, Mr Trump publicly undercut the former Texas oilman by tweeting that he was "wasting his time" trying to negotiate with North Korea.
In December, the secretary of state departed from administration talking points when he offered to begin direct talks with Pyongyang without preconditions.
The White House - which insisted North Korea must first accept any negotiations would be about giving up its nuclear arsenal - distanced itself from his remarks.
Mr Tillerson was reported to be astonished at how little Mr Trump grasped the basics of foreign policy.
The Republican president, meanwhile, was irritated by Mr Tillerson's body language during meetings, the New York Times reported.
The secretary of state was said to roll his eyes or slouch when he disagreed with Mr Trump's decisions.