I would say that a text file is one which, when sequentially read, has is a simple transformation from the bytes to a sequence of characters in one or more character repertoires (lists), fully consuming all bytes with none remaining, except any file-termination codes. This transformation may be direct mapping using the values of the bytes, or may involve mapping sequences of bytes to some other number (e.g. UTF-8), or may involve a simple state machine (e.g. ISO 2022), for example, (but surely nothing requiring a stack or random access.) The result and initial objective of parsing the file is a single sequence of characters.
I would say that a binary file, when used in distinction to "text file", is one which uses potentially more complex transformations, where the result and initial objective of parsing the file will be a data structure or event stream.
Something like that.
The members of the English Church had ingenuously imagined up
to that moment that it was possible to contain, in a frame of
words, the subtle essence of their complicated doctrinal system,
involving the mysteries of the Eternal and the Infinite on the
one hand, and the elaborate adjustments of temporal government
on the other. They did not understand that verbal definitions
in such a case will only perform their functions so long as
there is no dispute about the matters which they are intended
to define: that is to say, so long as there is no need for
them. For generations this had been the case with the Thirty-nine
Articles. Their drift was clear enough; and nobody bothered over
their exact meaning. But directly someone found it important to
give them a new and untraditional interpretation, it appeared
that they were a mass of ambiguity, and might be twisted into
meaning very nearly anything that anybody liked.
--Lytton Strachey, "Cardinal Manning"