The source of the first three numerals seems clear: they are collections of 1, 2, and 3 strokes, in Ashoka's era vertical I, II, III like Roman numerals, but soon becoming horizontal like the modern Chinese numerals. In the oldest inscriptions, 4 is a +, reminiscent of the X of neighboring Kharoṣṭhī, and perhaps a representation of 4 lines or 4 directions. However, the other unit numerals appear to be arbitrary symbols in even the oldest inscriptions. It is sometimes supposed that they may also have come from collections of strokes, run together in cursive writing in a way similar to that attested in the development of Egyptian hieratic and demotic numerals, but this is not supported by any direct evidence. Likewise, the units for the tens are not obviously related to each other or to the units, although 10, 20, 80, 90 might be based on a circle.
Brahmi numerals signs of the 2nd century CE.
The sometimes rather striking graphic similarity they have with the hieratic and demotic Egyptian numerals, while suggestive, is not prima facie evidence of an historical connection, as many cultures have independently recorded numbers as collections of strokes. With a similar writing instrument, the cursive forms of such groups of strokes could easily be broadly similar as well, and this is one of the primary hypotheses for the origin of Brahmi numerals.
Another possibility is that the numerals were acrophonic, like the Attic numerals, and based on the Kharoṣṭhī alphabet. For instance, chatur 4 early on took a ¥ shape much like the Kharosthi letter ch; panca 5 looks remarkably like Kharosthi p; and so on through shat 6, sapta 7, and nava 9 (Kharosthi sh, s, n). However, there are problems of timing and lack of records. The full set of numerals is not attested until the 1st-2nd century CE, 400 years after Ashoka. Assertions that either the numerals derive from tallies or that they are alphabetic are, at best, educated guesses.