feat: Match fingerprints by instruction filters

[LisoUseInAIKyrios]

Adds instruction filters to support more flexible instruction fingerprinting.
 

Changes

Fingerprints can still use opcode patterns, or they can use instruction filters that allow more precise matching.

Basic support exists for matching instructions using:

• MethodFilter: Specifies defining class, method name, parameters, return type, etc. (Sort of mini method fingerprints themselves).
• FieldFilter: Class/static field usage, by defining class, field name, field type.
• LiteralFilter: Literal instruction of type long/double
• Logical OR filter: Match one of any filters. Useful when instructions differ between multiple app targets.

Projects can define their own custom instruction filters, such as ResourceMappingPatch with it's own kind of LiteralFilter that matches resource literal values (no more mucking about with using a ResourcePatches to first set a resource value a fingerprint then uses).
 

Variable space allowed between instruction filters

By default, all filters allow any amount of space between them. But if filters are always immediately after each other, or there is a rough estimate of the maximum number of indexes until the next instruction, then a maximum distance can be set. An example is using an opcode filter of MOVE_RESULT or MOVE_RESULT_OBJECT after a method call, where the max instruction spacing is always 0.
 

Breaking changes

Fuzzy pattern match is now obsolete, as it's functionality is now part of the filtering itself. Variable spacing is allowed between instruction filters, and non important instructions are now ignored by simply not defining instruction filters for them.

Fingerprints are now declared using by semantics.

Before: internal val shortsBottomBarContainerFingerprint = fingerprint {

After: internal val shortsBottomBarContainerFingerprint by fingerprint {

If a fingerprint fails to resolve, the stack traces now includes the fingerprint name.

 

Example fingerprint before and after this change

Before:

val bottomBarContainer = resourceMappings["id", "bottom_bar_container"]

internal val shortsBottomBarContainerFingerprint = fingerprint {
    accessFlags(AccessFlags.PUBLIC, AccessFlags.FINAL)
    returns("V")
    parameters("Landroid/view/View;", "Landroid/os/Bundle;")
    strings("r_pfvc")
    literal { bottomBarContainer }
}

shortsBottomBarContainerFingerprint.method.apply {
    // Search for indexes after the fact, after the fingerprint already resolved.

    // First instruction of interest.
    val resourceIndex = indexOfFirstLiteralInstruction(bottomBarContainer)
    
    // Second instruction of interest.
    val index = indexOfFirstInstructionOrThrow(resourceIndex) {
        getReference<MethodReference>()?.name == "getHeight"
    }
    
    // Third instruction of interest.
    val heightRegister = getInstruction<OneRegisterInstruction>(index + 1).registerA
    
    addInstructions(
          index + 2,
          """
              invoke-static { v$heightRegister }, $EXTENSION_CLASS_DESCRIPTOR->getNavigationBarHeight(I)I
              move-result v$heightRegister
          """
    )
}

Now the indexOfFirst() logic is in the fingerprint itself:

internal val shortsBottomBarContainerFingerprint by fingerprint {
    accessFlags(AccessFlags.PUBLIC, AccessFlags.FINAL)
    returns("V")
    parameters("Landroid/view/View;", "Landroid/os/Bundle;")
    instructions(
        // First instruction of interest.
        resourceLiteral("id", "bottom_bar_container"),

        // Here lies other unrelated instructions.

        // Second instruction of interest.
        methodCall(methodName = "getHeight"),

        // Third instruction of interest.
        opcode(Opcode.MOVE_RESULT)

        // Other unrelated instructions.

        // Fourth instruction of interest.
        string("r_pfvc")
    )
}

shortsBottomBarContainerFingerprint.let {
    it.method.apply {
        val index = it.filterMatches[2].index
        val heightRegister = getInstruction<OneRegisterInstruction>(index).registerA

        addInstructions(
            index + 1,
            """
                invoke-static { v$heightRegister }, $EXTENSION_CLASS_DESCRIPTOR->getNavigationBarHeight(I)I
                move-result v$heightRegister
            """
        )
    }
}

[LisoUseInAIKyrios]

Still a work in progress. But so far works well.

[oSumAtrIX]

The documentation should be updated to match the new usage.

[oSumAtrIX]

There's multiple issues I have with this file and still confused from past discussions.

• If InstructionFilter filters instructions as it implies, it is not supposed to have context about methods or classes, but instructions, regardless of the reason behind it. If context about classes or methods is needed, "InstructionFilter" needs to have a different name as it is something else than what the current name implies.

• There are currently some existing filters implemented in Patcher under assumptions for relevance that should not be taken. Today field references, method calls, consts, and object instantiations may be relevant, tomorrow class references, string values, or other things that would require changing this file to adjust to a new assumption. APIs shouldn't be offered based on assumptions. Based on the current assumptions, you will restrict someone to current existing filters to avoid implementing the interface for whatever filter they need. Instead, an universal filter API should be offered that does not make any assumptions of what might or might not be useful and leave this decision to the API consumer. That said, those filters can be implemented somewhere in a separate module, outside of the patcher module, but can't be part of the patcher module just based on assumptions of relevance.

• Comments should follow the current style. Constructor parameters should be commented as @param in an inline comment for the constructor. Constructors should start with a sentence explaining what the class is/does. Currently, some just jump to examples (such as in LiteralFilter)

• Currently no DSL api is present, even though the patches & fingerprint API is currently fully DSL. Something like this would be acceptable:

  fingerprint {
      instructions {
          field(...)
          Opcode.X()
          Opcode.Y()
          "string"()
          123()
      }
  }

  However with the filter API the current simple usage of opcode patterns now involves more boilerplate. Before:

  fingerprint {
      opcodes(Opcode.X, Opcode.Y, ...)
  }

  fingerprint {
      Opcode.X()
      Opcode.Y()
      ...()
  }

  Every filter is added via invocation which has to be done as many times as many opcodes exist, a linear amount of times.

  An alternative API would be

  fingerprint {
      instructions {
          opcodes(Opcode.X, Opcode.Y) // Also works for one
          field()
          string("")
          ....
  	}
  }

• Filters look to me more like something suitable in custom block rather than replacing the opcode pattern. As explained in another review comment, fingerprints image a method, filters are not a direct attribute of a method making them suitable at most in custom (where also context about class and method both exist furthermore showing evidence of being a suitable place). Not sure how you'd want to pull that off, but replacing a direct attribute a fingerprint can image is something to avoid.

[LisoUseInAIKyrios]

There will always be the usage of method filters, field, and literal constants.

It's up to the patches to declare new filters if desired. Such as ResourceMappingPatch declaring it's own filter that finds decoded resource literals.

Instruction filter no longer has a classDef parameter. The instruction method is passed as a parameter and most filters don't use it, but some require it to check how many indexes the method has and others to parse out the enclosing class.

It's important to note this is not a custom block replacement. It's checking the instruction on a more fine grained scale, and checking the ordering of the instructions, and it produces a list of indexes for those matches that is then used by the patch itself. There should be little to no usage of method.indexOfFirst(previousIndex) { /* do checking here */ }, as these checks are now part of the fingerprint itself.

With just opcodes you only get patternMatchResults which is the start/end. But with instruction filters you get indexes of each filter since there can be variable spacing between each filter.

This is an expansion of what opcodes previously did, which is why opcode filters still exist and can still be used.

Previously with only opcode method calls you could only declare invoke_interface, but there is no way to indicate what it's invoking, especially when it's a non obfuscated class such as List.add(). Now you can declare more specific usage of these opcodes and not just patterns which are fragile, can match completely unrelated stuff, and frequently break when just a single extra register move opcode is added by the compiler.

DSL style declarations can be added, that's not an issue.

[LisoUseInAIKyrios]

Here's a simple example, where all the index searching was previously in the patch execute:

Before:

internal val shortsBottomBarContainerFingerprint = fingerprint {
    accessFlags(AccessFlags.PUBLIC, AccessFlags.FINAL)
    returns("V")
    parameters("Landroid/view/View;", "Landroid/os/Bundle;")
    strings("r_pfvc")
    literal { bottomBarContainer }
}

shortsBottomBarContainerFingerprint.method.apply {
    // Search for indexes after the fact, after the fingerprint already resolved.

    // First instruction of interest.
    val resourceIndex = indexOfFirstLiteralInstruction(bottomBarContainer)
    
    // Second instruction of interest.
    val index = indexOfFirstInstructionOrThrow(resourceIndex) {
        getReference<MethodReference>()?.name == "getHeight"
    }
    
    // Third instruction of interest.
    val heightRegister = getInstruction<OneRegisterInstruction>(index + 1).registerA
    
    addInstructions(
          index + 2,
          """
              invoke-static { v$heightRegister }, $FILTER_CLASS_DESCRIPTOR->getNavigationBarHeight(I)I
              move-result v$heightRegister
          """
    )
}

Now the indexOfFirst() logic is in the fingerprint itself:

internal val shortsBottomBarContainerFingerprint by fingerprint {
    accessFlags(AccessFlags.PUBLIC, AccessFlags.FINAL)
    returns("V")
    parameters("Landroid/view/View;", "Landroid/os/Bundle;")
    strings("r_pfvc")
    instructions(
        // First instruction of interest.
        ResourceMappingFilter("id", "bottom_bar_container"),

        // Here lies other unrelated instructions.

        // Second instruction of interest.
        MethodFilter(methodName = "getHeight"),
        // Third instruction of interest.
        OpcodeFilter(Opcode.MOVE_RESULT)
    )
}

shortsBottomBarContainerFingerprint.let {
    it.method.apply {
        val index = it.filterMatches.last().index
        val heightRegister = getInstruction<OneRegisterInstruction>(targetIndex).registerA

        addInstructions(
            index + 1,
            """
                invoke-static { v$heightRegister }, $FILTER_CLASS_DESCRIPTOR->getNavigationBarHeight(I)I
                move-result v$heightRegister
            """
        )
    }
}

[LisoUseInAIKyrios]

InstructionFilter can be renamed, but I'm unsure what other name to use.

MethodFilter might be more appropriately named MethodCallFilter, since it matches method calls based on specifics of that call.

FieldFilter could be renamed to something like FieldAccessFilter since it matches iget, iput, sget, sput, etc.

Edit: Renamed to MethodCallFilter and FieldAccessFilter

[oSumAtrIX]

There will always be the usage of method filters, field, and literal constants.
It's up to the patches to declare new filters if desired. Such as ResourceMappingPatch declaring it's own filter that finds decoded resource literals.

While the current filters and functions like addInstructions are useful for certain cases, they can't cover every scenario. Assuming that filters will always be necessary is a flawed approach, as there may be situations where none of the existing filters are suitable. These utilities are based on assumptions of common usage, but this can limit flexibility.

The same applies to addInstructions. Its essentially just adding an item to a list, but introducing this specific functionality as an extension function is an overspecialization that conflicts with the goal of maintaining a generic library. Filters should follow the same principle: while the interface for filters is generic, providing a predefined set of filters creates unnecessary constraints. I’d prefer to move the actual filter implementations to an external module, ideally separate from the patcher repo.

Instruction filter no longer has a classDef parameter. The instruction method is passed as a parameter and most filters don't use it, but some require it to check how many indexes the method has and others to parse out the enclosing class.

An instruction filter should only have context about the instructions. Bringing the method into this context is problematic in terms of abstraction. A filter for instructions relying on a method does not sound right. If somehow it is necessary, it means you need to rethink what "instruction filters" actually are. Perhaps they are more than just that given that you need context about the method.

This is an expansion of what opcodes previously did, which is why opcode filters still exist and can still be used.

There should be one clear way to handle instruction fingerprinting. If we’re moving forward with filters, the old opcode filter approach should be replaced with the new approach and reimplemented in it if necessary.

Now you can declare more specific usage of these opcodes and not just patterns which are fragile, can match completely unrelated stuff, and frequently break when just a single extra register move opcode is added by the compiler.

I think here it also shows that there is a specialization in one direction that is assumed to be likely useful. However, it is nonetheless a specialization that shouldn't happen in a library context that is supposed to be abstract. An example is that you can now filter for method references, but how about filtering for the field type only in field references? You'd now ask to implement the interface to satisfy this situation and would have failed to provide an universal API via the existing filters implementations, because, albeit being likely useful, they are after all specialized for specific usecases.

    val index = it.filterMatches.last().index

Regarding the API, it can also be useful, if you can declare a filter so that you can reference it later on. This avoids having to rely on the index of filterMatches. In your example it could look like that:

val opcodeFilter = OpcodeFilter(Opcode.MOVE_RESULT)

internal val shortsBottomBarContainerFingerprint by fingerprint {
    accessFlags(AccessFlags.PUBLIC, AccessFlags.FINAL)
    returns("V")
    parameters("Landroid/view/View;", "Landroid/os/Bundle;")
    strings("r_pfvc")
    instructions(
        // First instruction of interest.
        ResourceMappingFilter("id", "bottom_bar_container"),

        // Here lies other unrelated instructions.

        // Second instruction of interest.
        MethodFilter(methodName = "getHeight"),
        // Third instruction of interest.
		opcodeFilter()
    )
}

shortsBottomBarContainerFingerprint.let {
    it.method.apply {
        val index = iopcodeFilter.index // Notice the reference to the val opcodeFilter 
        val heightRegister = getInstruction<OneRegisterInstruction>(targetIndex).registerA

        addInstructions(
            index + 1,
            """
                invoke-static { v$heightRegister }, $FILTER_CLASS_DESCRIPTOR->getNavigationBarHeight(I)I
                move-result v$heightRegister
            """
        )
    }
}

InstructionFilter can be renamed, but I'm unsure what other name to use.

After all you don't just filter based on the instructions. This is the reason I had initially assumed it to be similar to the custom block of fingerprints. For that reason, a different name is needed. Can you explain why you need anything else than instructions to filter instructions?

[oSumAtrIX]

You can already do exactly that with this PR. A field access can declare any part of the access, and leave out the parts that are obfuscated or it's desired to ignore. Such as: fieldAccess(type = "Ljava/lang/String;")

And how do i filter for GET vs PUT?

[oSumAtrIX]

I don't understand what you mean by "enclosing" or "this":

The enclosing method is needed for methodCall() andfieldAccess() to use the this keyword, since it's impossible to declare an obfuscated class for the method/field access but using this can be used to indicate it's a call to the enclosing class. The declaring class (which is part of the method object) is needed for support the functionality. The declaring method is also used by lastInstruction() since it needs to know how many instructions are in the enclosed method. I don't see any issues with passing along the enclosing method of the instruction as it allows more flexibility such as here.

[LisoUseInAIKyrios]

Right now all use cases of instruction filtering is covered. I can't think of or find any situations where a filter is not provided for a use case.

Because smali opcodes don't change and are pretty much set in stone these days, there won't be any new use cases if what is provided here covers everything.

The only situation I can see where custom filters could be created, is for specific resource filtering (such as looking up resources by name from an embedded JSON file in the app, or any other situation where resources are compiled into code). But even that could still be done with a helper method that returns a literal instruction filter.

[oSumAtrIX]

I deprecated the old opcodes() method, but unless someone wants to spend the possible 10+ hours updating all the old code (I definitely do not want to), then it's much easier and more reliably to deprecate but still support the old patches code.

I dont follow. What do you mean "updating all the old code"

[oSumAtrIX]

The example you list is exactly how this works right now, and the code shown is taken from the patches PR of this change.

I checked the API of the class "InstructionFilter" and could not find a public field index. How do you get it like I showed in the propsed API?

If you mean a FieldReference or some other object, there isn't one and there cannot be a concrete field because it's a filter that matches against a FieldReference. It cannot define an exact field type unless the filter declares everything (name, return type, parameters, access flags, etc). I don't see how this could be useful.

I meant retrieving the matching index. Look closely at the examplary code I provided:

[oSumAtrIX]

It may be better to match with "contains". Sometimes packages are minified, but classes not. E.g. Lp/SomeClass;.

I think for flexibility reasons it may be useful to make the matcher parametrized. Every possible place where "matching" can be done in different ways should be configurable. Make it possible to let the user choose the matching type. E.g.

return eq "Z"

or

return startsWith "Z"

As you can see the API is strongly typed and domain specific. At this point we have to decide. Does this level of DSL make sense?

If we have such fidelity, why not simply provide the return type and ask for a lambda? E.g:

fingerprint { method.returnType == "Z" }

What kind of abstraction do we want? What can we benefit of it? Performance? With the DSL we can have hashmaps, with a lambda, maybe? It is possible if you expose the maps:

fingerprint { "Z" in returnsTypes }

As of right now, we have strong limitations everywhere in the API. The parameters API is hardcoded to match with startsWith. Its a clear opinionated limitation and as such we have to think about the above.

[LisoUseInAIKyrios]

I've never seen proguard obfuscate a package but not the class name in a package.

So starts with always works because if the class name isn't obfuscated, then the package also isn't obfuscated.

What I think could be helpful, is to allow multiple values for access flags, return type, and especially parameters.

There are times where access flags change between app targets, and other times where parameters change slightly (commonly an additional parameter is added).

It would simplify some YT fingerprints where parameters could be multiple parameter declarations, and any parameter list can match.

I didn't add this because it's only useful for YT since most apps only care about patching the latest, and it adds more complexity.

[oSumAtrIX]

because it's only useful for YT since most apps

This can not be assumed. You cant force matching to just startsWith. My personal and private app is obfuscating only packages and already this should proof the need of e.g. endsWith. So if you just implement startsWith you have a conceptual limitation. With the proposed API this clear problem is solved precisely.

[LisoUseInAIKyrios]

But is it stripping packages, or obfuscating package names and keeping the class names?

If it's stripping packages (everything is always moved to default or a specific obfuscated package), then starts with still works.

But this could be changed to support both "ends with" and "start with". I'll try adding this.

[oSumAtrIX]

But is it stripping packages, or obfuscating package names and keeping the class names?

Of course it will be the one where startsWith doesnt work for demonstrative purposes.

But this could be changed to support both "ends with" and "start with". I'll try adding this.

What if my obfuscator obfuscates the end and the start?

Lp/Class->obj;LType becomes Lp/ABC->obj;LCDE

As you can see, as long as you dont provide full coverage of the api, and try to solve one special case with a special solution, another special case will still remain unsolved.

The API must somehow provide a way where you can let the user "choose the matcher".

E.g.

fun string(s: String, matcher: IMatcher)

IMatcher can be anything in this case and automatically covers any case you can imagine, not just cases where startsWith works. The example is not clean but it proofs theres a way to cover all cases. Now we just have to figure out a clean DSL/API.

E.g. the one i proposed

return eq "Z"

(eq would be a builtin prefix function which is emits an Instruction filter in the this receiver)

So it would basically translate to

instructions {
eq(return, "s") // emits a string filter with equals matching
}

if you generalize merely this api provides full coverage:

instructions {
instruction {
// "this" is the current instruction
}
}

This API provides FULL coverage because instruction becomes a lambda where you can define anything, e.g.

instructions {
instruction {
it.getReference().string == "A"
}
}

[oSumAtrIX]

The source files are not meant for navigation. As for such they should not conform a name that is supposed to enable or aid that. Removing the prefix entirely is even okay as there's no specific reason for the prefix to be in the name outside for the unwanted purpose of navigation through the source files. Instead, navigation will be enabled once the files are rendered by some parser into a user-facing document, e.g. the site, a pdf or similar.

[oSumAtrIX]

I don't see a purpose of the field name being visible in the stack trace when there's already the LoC. In this case I can see it occurred in line 63 which is unambiguous. This spares the necessity for non-intuitive-on-first-sight API which you have to look up to understand just for a dininishing return of now understanding its minor use.

[oSumAtrIX]

I think these apis should follow the opcode names, e.g.

GetObject
PutObject

Once again, with the other review comment above, these are limited apis. What if I want to match for iGet instead of Get. This flexibility issue needs to be fixed regardless of how likely or unlikely you will need it.

[LisoUseInAIKyrios]

You can match any specific get/put instruction by specifying one or more opcodes with the filter. Then you can define if it's a static get/put or it's accessing a primitive/object/array/etc from an object field. So the filter can be as wide as any kind of field access or as specific as matching an exact field type that is static or not static.

I agree it could be two different field access filters, one for get and another for put, because it's always one or the other and never do you want to match either or.

[oSumAtrIX]

specifying one or more opcodes with the filter

Then i cant match on the operand. How do I match "iput & operand = SomeField"

[LisoUseInAIKyrios]

I'm away from my computer, but that is possible right now.

Something like:
fieldAccess(opcode = Opcodes.IPUT, name = "someField")

[oSumAtrIX]

I think, "filter" is terminology we can get rid of before we introduce it. We also do not call the returns api a filter, so consistency must be retained somehow

[oSumAtrIX]

"clearMatch" is weird api. I recall there was something being said about good api that an api should not just be callable once, e.g. that there should not be a "state", unless it is explicit, and where it acts as a deconstructor for some class/state

Examples where it is allowed is "close" in streams, because it is a kind of deconstructor.

We can fix the issue with performance when multiple patches rely on a fingerprint by moving the match out of the fingerprint to the designated bytecode context.

E.g.

with(someBytecodeContext) {
fingerprint.match() // matches in relation to the context
}

with(otherBytecodeContext) {
fingerprint.match() // will match again
fingerprint.match() // wont match again
}

with(someBytecodeContext) {
fingerprint.match() // early returns since in the current context it already has matched
}

Code implementation wise we might not want to move the "match" field to the context. But keeping track of contexts via a map in the fingerprint class also is weird. So implementation wise this has to be looked into first. Right now I only have:

class BytecodeContext {
matches: Map<Fingerprint, Match>
}

or

class Fingerprint {
matches: Map<BytecodeContext, Match>
}

[oSumAtrIX]

Might wanna rename to "immutableClassDef". This would make the API naming consistent with dexlib (the type is called ImmutableClassDef)

[oSumAtrIX]

I think we should get rid of this abstraction. It is overspecialized and redundant

[oSumAtrIX]

Exposing "classes" is enough as you can then simply use classes.single

[oSumAtrIX]

Perhaps we can simplify this api by replacing recursion with iteration.

This would spare us the odd api "stop".

Instead we could have

fun to(vararg indices: Int, mutable: boolean)

Chaining can be allowed with another overload:

fun Method.to(vararg indices: Int, mutable: boolean)

this way it will work with:

navigate(someMethod).to(1,2,3).to(4, mutable=true)

We could maybe even get rid of more abstraction and just provide:

navigate(someMethod, 1,2,3, mutable)

and maybe even normalize the name for example with

method(from=someMethod, at=1, mutable=true) (or getMethod etc)

[oSumAtrIX]

Whats the motivation for the name?

[oSumAtrIX]

Not a full review, but advancing iteratively