In these cases it can be helpful to look for tell-tale file format signatures and inferring how the application is using them based on these signatures, as well as how these formats may be abused to provoke undefined behavior within the application. When assessing an application, one may run into files that have strange or unknown extensions or files not readily consumed by applications associated with those extensions.
