A coalition of big banks filed its much-anticipated appellate brief Wednesday with the New York Court of Appeals, seeking to reinstate a $5 billion fraudulent conveyance suit against the bond insurer MBIA.