I do not know of any research personally but there is only a very small set of characters that are used when formatting numbers:
Decimal separator (. or , depending on locale).
Dash for negation (-).
If you don't have both of those, then there are numbers that you simply cannot enter. So you need those.
A backspace key is technically optional but without it correcting mistakes becomes difficult or potentially impossible.
A thousands separator (, or . depending on locale) is not necessary but given that including both . and , let you support all common locales without having to know which is in use, that's an option.
Everything else really depends on your specific situation:
Space: If the user can be editing text while in number pad mode, a space could be useful. If a user is entering a list of numbers, a space could be useful. Do you do things like this?
Enter: Same deal.
%, currency symbols, parentheses, E, etc. again, all optional, so you'd have to take typical context for your specific user base into account.
But you really ought to include a decimal separator and a hyphen, those are both important parts of numbers, and required to represent all numbers. The rest is icing on the cake.
If you get common independent requests from your users, that's basically your study right there.
Also if you are writing a generic keyboard app, you could track usage information. For example, if a user leaves number pad mode, enters only a small number of characters, then enters number pad mode again, send yourself some automated feedback about what those characters were. Same with characters entered just before entering or just after leaving number pad mode.
The other thing is, look at existing physical number pads (for example). They've all got about the same set of characters: Always digits, dash, and a decimal separator. Mathematical operators are common but only because the context is they're often used when performing calculations, which probably isn't a common use case on a generic mobile keyboard. Enter is also common. Other than that, there's not much else.
Bear in mind that if you start making things up or rearranging layouts in non-conventional ways, you risk creating a good amount of confusion. So I'd really just stick with the required characters: Decimal separator and dash, and at most include the common options: Thousands separator and enter, plus the convenience: Backspace.
