Hello from CCCS! 🍁(Idea: Integration with Assemblyline)

[cccs-rs]

I stumbled across your project this morning and noticed you're supporting the CaRT format (nice!)

I was wondering if you've ever heard of our project called Assemblyline, an automated malware analysis and triage platform.

I think that MalwareDB could have some nice integration with the platform in terms of retaining files for an indefinite amount of time after analysis while providing the data in a way that could be beneficial for research & forensics.

I know the project is still in it's alpha phase but I figured it would be a good opportunity to gage if there's any interest in collaboration 😁

If you have any questions about Assemblyline or anything in general, feel free to ask! 🤓

[rjzak]

Hello! Thanks for reaching out and yes, I've heard of AssemblyLine but haven't used it personally. I agree that they would likely work nicely together. What sort of things do you think might be needed on the MalwareDB side to make this work?

And yes, thanks for the CaRT format, it's ingenious. I use it as-is plus some extra fields (more hashes, entropy).

[cccs-rs]

Based on what I've read for current and future goals of the project, one of the things would be to provide extra metadata to file that's stored in the DB in addition to the origin (ie. this could be the date the file was acquired from the origin source or some kind of identifier that could be tied to a certain incident).

In Assemblyline, we use a classification system like TLP to perform access control of information through our API. So I believe having a way of optionally configuring that for a MalwareDB deployment would be great to have as an alternative to group-based access control 😁. This is something we could package to make it more easier to integrate since we have our classification engine written in Python (and I believe in Rust as well).

An example of how it's used is let's say I have my account's classification set to TLP:AMBER//REL TO CCCS, then it would mean I should have access to any data that's TLP:CLEAR, TLP:GREEN, and TLP:AMBER and in addition anything that's related to CCCS (ie. TLP:CLEAR//REL TO CCCS, TLP:GREEN//REL TO CCCS, TLP:AMBER//REL TO CCCS) but I wouldn't have access to data that's TLP:AMBER+STRICT.

In terms of adding more features for files in the DB for ML training + big data, we do have a set of ontologies that we generate that capture the results from Assemblyline in a more database-acceptable format but ideally this could be transformed into any format that makes querying easier on your side. In that documentation, we have ontologies that are service-agnostic (services are what we call our plugins that perform analysis over a file) but capture common analysis like network interactions, malware configuration, and more.

In principle, sending these kinds of records should contain both the metadata that I've mentioned earlier as well as the features/analysis we were able to extract about the file, so having a way to ingest those records, in one form or another, would be the ultimate goal.

---

This is probably a lot of information that I'm throwing at you all at once, but if these kinds of requests align with your vision of the project, then great! I would love to hear what your thoughts are on it 🤓

[rjzak]

What are the possible values for the classification type? Would user accounts need to have this classification value as well? Initially I thought there would be an AssemblyLine service account but maybe that's not the case if the user accounts have to be aware of the classification type? I'm open to discussing virtually if that helps. richard.j.zak@gmail.com

[cccs-rs]

Our classification engine supports custom definitions, by default we followed TLP 2.0 found here but this can be tailored to the deployment's environment (ie. add more groups, assign different levels, etc.).

To perform access control (at least similar to how we do it), the user accounts would need to have a classification value that the API can use to check if a user has access to a piece of data (which would also have a classification value to check accessibility).

I'd be happy to discuss this virtually, maybe I can provide a brief demo of Assemblyline if that's of interest 😁

[rjzak]

Further thoughts:

• AssemblyLine would be the source of samples, but is it the only source? What if a user wants to add files?
• AssemblyLine would need its own API endpoint, and its own service account. How would this work, maybe check that the API key is for AssemblyLine user?
• Regarding the classification levels to see if the user has >= access, does the classification object support this type of comparison? I don't see that in the code.
• User accounts would need to have their classification level data, where does this come from? Does AssemblyLine create users?
• I would add this functionality in a PR, would you be willing to review it to ensure it makes sense?
• Database: person table gets a classification text field, file table gets a classification text field and an ontologies field, perhaps Json?
• I'm thinking this functionality should be gated behind a feature flag since it changes how permissions work, and adds an API endpoint.
• A virtual chat and a demo would be awesome.

Lots of questions and things to be figured out before I get started, but I really like this idea and should be able to get started in a week or two.

[cccs-rs]

AssemblyLine would be the source of samples, but is it the only source? What if a user wants to add files?

For our use-case, Assemblyline would be the entry point where samples are initially submitted to and then some decision making is made to see if we want to retain that data in MalwareDB (usually to see if the right metadata is in-place before retention) but I wouldn't discount the idea that there could be other sources where samples could come from (ie. for research from a user or analysis from another platform like VT)

AssemblyLine would need its own API endpoint, and its own service account. How would this work, maybe check that the API key is for AssemblyLine user?

That could be an option from an integration standpoint. If let's say there's a service account that Assemblyline can connect to MalwareDB using an API key, then I think that's sufficient. Having it's own endpoint for creating records would be useful if there's special processing logic needed as opposed to a general endpoint (kind of relates to your previous question about other data providers besides Assemblyline)

Regarding the classification levels to see if the user has >= access, does the classification object support this type of comparison?

Yup, that should be present here where it compares two classification strings to check accessibility.

User accounts would need to have their classification level data, where does this come from? Does AssemblyLine create users?

Yup, Assemblyline supports creating users and depending on if classification is enabled, it will assign a classification level to user accounts.

I would add this functionality in a PR, would you be willing to review it to ensure it makes sense?

Sure! I'm not as experienced with Rust myself but I have colleagues that will be able to help with the review process 😁

[rjzak]

I've got a PR for Assemblyline support: #773

I'm thinking initially:

• Separate API endpoint for AS to push a file.
• AS should probably be what creates the user so the classification part can be added (and disallow the admin CLI from creating a user)

Additionally:

• I'd added the ability for MDB to store files of unknown file types.
• MDB is now beta!