Those with premium cards, defined as having an annual fee of at least $250, were an anomaly, Bank of America's research said.