He admitted there had been “years of distrust” and said both sides needed to make a “sustained effort… to respect one another and seek common ground”.
Mr Obama said the US bond with Israel was unbreakable but described the Palestinians’ plight as “intolerable”.
The president made a number of references to the Koran and called on all faiths to live together in peace.
He received a standing ovation at the end of his speech at Cairo University.
White House officials had said the speech was intended to start a process to “re-energise the dialogue with the Muslim world”.
‘Not so unique’
Mr Obama said: “I have come here to seek a new beginning between the United States and Muslims around the world; one based upon mutual interest and mutual respect.”
Take these two sentences: “The United States does not accept the legitimacy of continued Israeli settlements” and “It is time for these settlements to stop”. It is the same language that Hillary Clinton has recently used. But what does it actually mean?
Does it mean that settlement expansion should stop? Or, in fact, for existing settlements to be disbanded? Could the first sentence even mean that the US is beginning to revisit its studied ambiguity, for the past 30 years, over whether it views all Israeli settlements on occupied territory as illegal?
Given the amount of time and effort that goes into working and re-working the text of a presidential speech, the apparent veiling of these two sentences is fascinating.
He said “violent extremists” had bred fear and that this “cycle of suspicion and discord must end”.
Mr Obama accepted that “no single speech can eradicate years of mistrust” but urged both sides to “say openly the things we hold in our hearts and that too often are said only behind closed doors”.
He cited the Koran as saying: “Be conscious of God and speak always the truth.”
Mr Obama said Islam had “always been a part of America’s story”.
He added that much had been made of the fact an African-American named Barack Hussein Obama had become US president, but he insisted his personal story was “not so unique”.
“The dream of opportunity for all people has not come true for everyone in America, but its promise exists for all who come to our shores – that includes nearly seven million American Muslims.”
‘Intolerable’
The president also said Muslim perceptions of the US must change.
“Just as Muslims do not fit a crude stereotype, America is not the crude stereotype of a self-interested empire.”
Mr Obama said America was not at war with Islam, but would confront violent extremists who threatened its security.
On the key issues of Iraq and Afghanistan, the president said the US sought no permanent bases in either country.
He said: “We would gladly bring every single one of our troops home if we could be confident that there were not violent extremists in Afghanistan and Pakistan determined to kill as many Americans as they possibly can. But that is not yet the case.”
On the Israeli-Palestinian issue, Mr Obama said the bond with Israel was “unbreakable”.
He said: “Palestinians must abandon violence. Resistance through violence and killing is wrong.”
But he also said the “situation for the Palestinian people is intolerable”.
“Israelis must acknowledge that just as Israel’s right to exist cannot be denied, neither can Palestine’s,” Mr Obama said.
On the key issue of Israeli settlements in the West Bank, Mr Obama said “there can be no progress towards peace without a halt to such construction”.
Israel is resisting calls to freeze building activity but Palestinian leaders have said there can be no progress towards peace without a halt.
After the speech, Israeli Prime Minister Benjamin Netanyahu summoned ministers to a special meeting and ordered aides and officials not to comment until a government statement was released.
The statement said the Israeli government hoped the speech “will indeed lead to a new era of reconciliation between the Arab and Muslim world and Israel”.
BBC diplomatic correspondent James Robbins says Mr Obama was tough on both sides – perhaps tougher on Israel than we are used to hearing from an American president.
Our correspondent says Mr Obama made it clear there was no justification for Holocaust denial but he seemed to associate the state of the Palestinians with that of slaves in America.
A spokesman for Palestinian leader Mahmoud Abbas said the speech was a “good start and an important step towards a new American policy”.
The AFP news agency quoted Hamas, the militant group that controls the Gaza Strip, as saying the speech showed “tangible change” but also contained contradictions.
