Rewrite repo_info_extractor

[peti2001]

The current solution has quite a few problems. But most importantly it is hard to use. It is written in Python and for developers who have no Python experience, it is not very convenient to use.

Problems with the current solution

• hard to install, requires Python
• too big output, JSON takes too much memory and CPU to process, GRPCGateway and filechanges use a lot of memory.
• it is slow, because of that we have to skip large repos

Requirements:

• It must have the same feature that the current solution has
• Binary executable (Windows, Linux, OSX)
• The output must be parsable line by line to avoid loading everything into the memory
• Make at least 10 times faster. Must be able to progress theses in less than 10 minutes:
  • https://github.com/aosp-mirror/platform_frameworks_base
  • https://github.com/eXpandFramework/eXpand
  • https://github.com/expand/eXpand
  • https://github.com/eXpandFramework/eXpand.lab
  • https://github.com/eXpandFramework/lab
  • https://github.com/laravel-enso/enso
  • https://github.com/fellipegpbotelho/odonto-uni
• Import existing trained Python model to Go to find similar emails. Being able to detect similar emails. For example, if I have a commit with test.peter@example.com peti2001@example.com, testpeter@codersrank.io it has to recognize it comes from the same user.

Nice to have

• Merge with the multi_repo_info_extractor, being able to extract multiple repos by passing tokens, credentials.
• Serverless compatibility (Go is available in Google Cloud)
• parse the code instead of using regex, to improve the accuracy of the import detection
• Minimalize disk IO, don't check out the code, do it in memory
• Support multiple outputs. Easily extendable by the community.
• Recognize squashed commits, not just merges
• GUI

If you have any suggestions, problems with the current implementation, please share.

[JJ]

I don't really have a big problem with Python. I'll look for ways to speed it up, if that's the blocker. You can only get so far doing that, however. WRT to current requirements.

1. Create a test suite that you will use to test that it's really got the same features. Create a set of model inputs (languages), and model outputs. Coverage should be 100%
2. If the reason is that it makes it faster, so be it. If it's that it should be easy to run anywhere, maybe a Docker container (with different architecture) will be a better solution. If it's that it should be easily installable anywhere, you should be looking at packaging options, maybe.
3. You seem to be looking at low memory footprint, but that might clash with the next requirement. Mind you, memory need not be "in the same place". It can be in some other VM, or node, and accessible via a socket. So instead of presenting a solution, it might be better to look at the actual problem and propose several solutions for it.
4. Again, the problem might be the overall duration of processing, not the individual duration. Maybe instead of individual limits, it would be better to go at overall limits in a machine with several cores. That might restrict the choice of languages, or frameworks to process stuff. To be clear, it doesnot really matter if those take 100 minutes (which I guess is what they take) as long as they can be processed at the same time in a 16-core machine. Anyway, making the overall process faster is never a bad thing, but by limiting ourselves to sequentially faster languages we might drawing ourselves into a corner that includes only C and maybe Java and Go.
5. Don't get this one. Similar emails?