A photograph showed Obama and Castro, both wearing dark suits, chatting in a small group of leaders at the summit's opening ceremony. A White House official confirmed the two men shook hands and spoke briefly.
"This was an informal interaction and there was not a substantive conversation between the two leaders," the official said.
Obama and Castro are expected to meet again on April 11 and talk about their efforts to restore full diplomatic relations and boost trade and travel between the two countries.
Their rapprochement, first unveiled in a historic policy shift in December, is the central issue at the Summit of the Americas meeting in Panama.
"As we move towards the process of normalization, we'll have our differences government to government with Cuba on many issues. Just as we differ at times with other nations within the Americas, just as we differ with our closest allies," Obama said earlier on April 10.
Colombian President Juan Manuel Santos hailed Obama's push to improve relations with Cuba.
Obama, who met with activists from across Latin America, including two Cuban dissidents, appears to be close to removing communist-run Cuba from a US list of countries that it says sponsor terrorism.
Its inclusion on the list brings a series of automatic US sanctions and Cuba is insisting it be taken off as a condition of restoring diplomatic ties.
