Hi @digitalXmage,
When I got started with ELF I was already familiar with binary files… mostly image formats. Those are way simpler, but you get use to deal with fields of specific sizes, flags and so on. So, for ELF I just read a couple of tutorials and then jumped into the spec… It is smaller/simpler, than you may think (compared to other specifications out there) . I do not think that would help you.
I’d say you start with simpler tutorials (compared to the book you mentioned that is great by the way). Have you read this? It may help. There are a lot of information about ELF in the site. Use the search box 
Also, use the available tools (readelf, elfsh,…) … don’t look at hex directly… that can burn your eyes 
… jk. Use the tools that is easier and eventually those hex values will just got stuck in your head.
I believe that what you are doing (a simple parser and looking into injection) is perfect. Just keep that path, and whenever you find something that you do not understand… try to find out what is going on first by yourself (you learn a lot more that way) and if that doesn’t work then ask here. In no time you will manage to solve your doubts by yourself (as far as you want to put the required effort in)
Also @Danus message gives a very good advice . There is no magic recipe, it just takes time and effort.