Edit: add N’Ko
I know of two modern living RTL mathematical tradition: Arabic and N’ko. I have no idea how other living RTL scripts (Hebrew, Dihevi, Syriac) deal with math.
For Arabic (including other languages (Farsi, Urdu), written in the Arabic alphabet), there are three distinct traditions (Morocco, Maghreb, Mashrek), so I think the fraction order might be ambiguous depending on the country you're in.
The W3C document on Arabic mathematical notations says, for this specific case :
Finally, although stacked fractions are rendered the same way in both European and Arabic, bevelled fractions in RTL Arabic will appear, as one would expect, with the terms in RTL order, i.e. A divided by B would appear as "B/A". In some locales, the preference is for the slash to also be mirrored, as "B\A".
This document also contains many other examples. You also have some examples on the corresponding Wikipedia page.
N’Ko is a modern (20th century) RTL script used in western Africa for Manding languages. I happen to have an introductory book on N’Ko at home, and it ends with some example of elementary arithmetic operations, which are written right to left, as expected. Furthermore, the numbers in N’Ko (contrary to Arabic) are written right-to-left: The current year (2017) is written ߂߀߁߇, with (߂=2 ;߀=0 ;߁=1 ;߇=7).
Remark on historical texts
And if you look at historical texts, then you have lots of different conventions to adapt maths to different typographical traditions, including vertical writing in East Asia. (See e.g. this Unicode proposal pdf about notations used to translate Western mathematical texts in Ming and Qing China).