Trump stunned the world last week when he agreed to meet with North Korean dictator Kim Jong-un to discuss the denuclearization of the peninsula after Kim made significant concessions.
While Trump has mentioned pulling out of the Iran nuclear deal, which he has called the "worst" ever negotiated, Tillerson was widely known as a proponent of remaining in the accord, which was negotiated under President Barack Obama.
Tillerson's insult to Trump's intelligence came after heated discussions about Afghanistan policy, in which Trump had pushed to end the US presence altogether and flirted with outsourcing the military counterterrorism campaign to private contractors.
"We got along actually quite well, but we disagreed on things", Trump said to the BBC.
Addressing reporters, Trump said the decision had been in the works for a while.
Trump said that he lacked "chemistry" with Tillerson but was expected to be on the same "America First" wavelength with Pompeo, a West Point graduate and former Republican representative from Kansas who was a Tea Party favorite in three terms in office.
A senior White House official also said Tuesday Tillerson's ouster occurred at this time because Trump wanted to "have his new team in place in advance of the upcoming talks with North Korea and various ongoing trade negotiations".
He pointedly declined to thank Trump personally or praise him, as he has done on previous occasions, but emphasided his strong relationship with Secretary of Defense Jim Mattis.More news: Sharad Pawar to host oppn meet by end of March: NCP
More news: Dzeko goal is enough to send Roma through
More news: Nicole Kidman to lead another HBO series The Undoing
Tillerson said he'd turn over all responsibilities to Deputy Secretary of State John Sullivan "effective at the end of the day". Goldstein, who had only been at his post since December, was sacked on Tuesday as well.
Concerned by Trump's disregard for diplomatic protocol, Tillerson worked during his tenure to soften the president's more hard-line stances, such as on the Iran deal and greater Middle East policy.
"I am confident he is the right person for the job at this critical juncture", Trump said of Pompeo in a statement distributed by the White House. Shortly before the president said he would meet with Kim, Tillerson had stressed that the US was a "long ways from negotiations" with North Korea.
"Mike Pompeo's impending move to secretary of state is sure to result in a much more hawkish and confrontational USA policy towards Iran", John Glaser, director of foreign policy studies at the Cato Institute, told FOX Business.
Tillerson, who had been traveling in Africa last week, cut short his trip to return to the U.S. Tillerson's firing came after he rightly denounced Russia for its alleged role in the nerve gas attack on a former Russian spy in England. Earlier, White House press secretary Sarah Sanders refrained from saying Russian Federation was responsible.
Although relatively new to the Trump inner circle, Pompeo has won the president's trust with tailored briefings on classified information. "He will do a fantastic job!" Tillerson is out as secretary of state.
After Haspel was named deputy Central Intelligence Agency director, the European Center for Constitutional and Human Rights asked German prosecutors to issue a warrant for her arrest over her role in the interrogations.
Another suggested Mr Tillerson may follow the path of former Trump aide Omarosa Manigault Newman and go for a stint on Celebrity Big Brother.
Tillerson, who was formerly the head of ExxonMobile, was appointed to the position of Secretary of State previous year, and he and the president have had numerous public disagreements.
The impact of this shift on the US-Russia relations and the US foreign policy in general is probably limited. "The question is, how much juice does she have in the White House?"