At a historic news conference, Mr Castro said if he was given a list of political prisoners, he would "release them tonight".
The White House has said it has given Cuba lists of dissidents in the past.
Mr Castro does not view the prisoners as dissidents, US officials said.
That disagreement is central to the conflict between US and Cuban officials.
More needs to be done to lift the US embargo on trade with Cuba and that the Guantanamo Bay detention camp must close, Mr Castro said.
Mr Obama, the first sitting president to visit Cuba since 1959, said the trade embargo would be fully lifted.
"Cuba's destiny will not be decided by the United States or any other nation... The future of Cuba will be decided by Cubans not by anybody else," Mr Obama said.
Image copyrightGetty ImagesImage captionUS Secretary of State John Kerry was among White House officials at the dinnerImage copyrightGetty ImagesImage captionGuests enjoyed music by a Cuban band
Mr Obama and first lady Michelle Obama were later welcomed at a state dinner, along with some members of US Congress and senior White House staff.
The Failde Youth Band, a Cuban music, performed and guests ate shrimp mousse, cream soup flavoured with rum, and traditional pork with rice and plantain chips. Servers passed out Cuban cigars for guests.
Earlier on Monday, Mr Castro also defended Cuba's record on human rights and pointed to problems in the US.
"We defend human rights, in our view civil, political, economic, social and cultural rights are indivisible, inter-dependent and universal," Mr Castro said.
Mr Castro is not usually subject to any aggressive questioning from reporters and called the prisoners question "not polite", later ending the news conference by saying "I think this is enough".
"Actually we find it inconceivable that a government does not defend and ensure the right to healthcare, education, social security, food provision and development," he said.
Speaking with ABC News after the conference with Mr Castro, Mr Obama did not directly say he would be giving Mr Castro a list of political prisoners.
"We have given them a lists in the past and they have responded intermittently to our, uh, to our, uh engagement," he said. "And this I think is an example of why it was my belief that this would be a more successful mechanism for us to advance the values that we care about than an embargo and silence and no communications.
President Obama and his aides hoped that the Cuban president, Raul Castro, would approach the news conference with an open mind.
The White House officials also hoped that Mr Castro would be receptive to new ideas and to new ways of doing things. Specifically, the US officials hoped that he'd take questions from reporters. Surprisingly enough, he did, marking a departure from his past.
The two men joked around during the question-and-answer period, and they worked out some of their differences. When one of Mr Castro's aides interrupted his boss during the conference, for example, Mr Obama looked annoyed. But he and Mr Castro moved on, talking about the future of Cuba.
In a similar way, Mr Obama and his aides hope they can overcome other issues, including difficult ones such as human rights, as they forge a new relationship between the two countries.
Mr Obama could not say exactly when the trade embargo would be lifted, but recognised it was necessary.
"The reason is what we did for 50 years did not serve our interests or the interests of the Cuban people," he said.
His administration has done what it can on lifting trade restrictions, he said, but further action will require Congress which is "not as productive in an election year".
He also said further easing of the trade embargo will depend on actions Cuba takes on human rights.
(BBC News)