A recent terrorist attack in Mumbai (still really Bombay according to a friend from there) India focused on Americans, British, and Israelis. Terrorists seek out people from those countries as preferred victims for torture and killing. Well, chips built into American passports can make it easier for terrorists to identify a clustering of Americans abroad.
But RFID-chipped passports may present a terrible attack surface simply by existing. RFID chips don't actively announce their presence, but inexpensive and widely available readers can sense them -- and can sense when there are a number of them gathered together.
One security professional who travels internationally (and asked that he not be named) suggests that if terrorists wanted to pinpoint the location of large groups of Americans (a guided tour? a popular expat hangout?), the specific information on any one RFID chip would be far less useful than the simple ability to sense where a bunch of RFID chip carriers were grouped -- the very fact of their grouping may be information enough. Under those circumstances, grabbing the unique number(s) doesn't matter, since the specific ID data is unimportant; all that matters is the presence of the chips, and thus the targeted Americans.
Savvy owners of chipped passports or cards keep them in Faraday-cage wallets or sleeves. Faraday cages being what they are, not every kind of cage blocks every frequency, but the chips used in passports can be blocked fairly effectively...until you get to the TSA security checkpoint and your passport jacket sets off the metal detector.
Some security experts think these chips are too cheap and basically too easily hackable to provide a real increase in security.
My take: the chips could allow the automation of database look-up to find a relevant record for the person carrying the passport. Scan the card to get an ID number. Use the ID number to find the person's record. Then compare the record's information with results from a biometric scan (e.g. fingerprint or eye iris scan). In other words, we can't rely on a person to carry chip with a record that says anything accurate about them. But a chip, even if not reliable, could speed up the detection of whether the chip has been hacked or the person presenting the passport is trying to fake their identity.