Well, if you have access to the source code and can perform a whitebox test, you should first look at unsafe functions.
Google “Unsafe functions php/ruby/…” and you will find some for each language.
If I am looking at php, there are tons of functions that you can look for.
- exec
- passthru
- system
- shell_exec
- popen
- proc_open
You can just grep for those functions and then check where they are used. And what parameters are passed to them. Then follow the parameters. Where are they coming from? Is it maybe even user provided input?
If you’re lucky maybe some user input will be passed to those functions and you most likely will have a RCE vector.
You can also grep for mysql stuff and see if there are any queries which have unescaped input, where you could perform an SQLi.
File handling is also good to look at. Is the application reading or writing to the disk? Follow that stream too and see where the data is coming from.
Also check for unserialize as it might be vulnerable to object injection.
You can use for example https://github.com/tomnomnom/gf and write some patterns for the relevant unsafe functions, so you do not need to memorize all of them.
And thats basically it. Try to find the functions, follow the input, try to understand whats happening here and where the input is coming from. And try to inject payloads.
What you can do is also just check the OWASP Top X vulnerbilities and see what triggers them and then look for the code in the application.
This is like the baseline check you can perform. However, as you are looking at the code you slowly will understand the application better and might even find logical errors which can be exploited.
Another approach would be to use a Fuzzer. Which is also great, but requires some experience in using the fuzzer and the application. It does not make sense to fuzz every possible input of an application, because it would take forever. If you want to fuzz, you need to set your scope properly.