A team of scientists searching for a rare species of whale instead found a species of whale they say has never been recorded.