SHELL-AFFECT - New Examination Model

Cry0l1t3 · March 11, 2020, 2:51pm

Hi, everybody,

It’s been a while since I posted something, so I’d like to pick it up a bit now.

Now that we will update the course next April and have already added almost 100 new pages of material, we have also revised and refined the certification process.
In this topic, I would like to introduce you to the new examination model we have developed for SHELL-AFFECT certification process. This topic is structured as follows:

1.0 The necessity and goals of the examinations.
2.0 The problems
3.0 New Examination Model
3.1 Goals of the Examination
3.2 Examination Phases
3.2.1 Course Material
3.2.2 Lab Environment
3.2.3 Live Presentation
3.2.4 HackTheBox Rank (Bonus)
4.0 Requirements for Passing

1.0 The necessity and goals of the examinations

Before we go into the model, we should take a look at what such an examination is necessary for.
We know that such exams serve to determine the skills and the required level of knowledge.
The skills and the required knowledge should be acquired through a course or workshop.

The learning material provided by the training provider should teach a particular repertoire so that the student can allow his/her creativity to solve problems that may arise.

Almost all exams have a time limit. Some give the student more time and others less. This is not only necessary for organizational reasons but also puts the student in a hard position to be able to quickly retrieve his or her knowledge when time pressure and effort are high. This way, the actual internalized processes, and skills are used, which the student should be able to apply intuitively.

Above all, the certificates obtained after such a passed examination should prove that the student has made a certain effort to pass it. Accordingly, this student has been able to prove to the certification body that he or she has acquired the required skills and knowledge in the examination environment provided by the training provider.

Especially in the area of penetration testing, the training providers try to check whether the student has developed a structured methodology for the given material.

So let us summarize what we have now:

possesses the skills taught in the course or workshop
Has internalized the knowledge taught
The student can combine his knowledge with his creativity to solve problems under time pressure and in a given environment.
The student has achieved a high level of effort and performance
Material for the development of the methodology

2.0 The problems

Regarding the required knowledge that a student is expected to achieve in a course or workshop, we found the problem that the exams too often focus on the performance of the short-term memory and therefore fail to test the actual skills, knowledge, and especially the understanding of the subject being studied.

Often such courses or workshops offer temporary learning environments or better said too short learning conditions for studying and practicing. Otherwise, students have to make a higher financial investment to “be allowed” to continue to practice. This fact puts the student in a situation where he or she has the material but cannot continue to practice to internalize the knowledge offered.

If a training provider offers learning materials and provides the student with the exam environment, they will dictate the student exactly how to proceed. The exam environment is based on the topics covered in the learning material. The student may be able to prove that he/she has solved one scenario or another, but only the problems of the specific topics are solved. If the environment differs from the known structure, the student is usually no longer able to solve the problem alone.

Obtaining a certificate may serve as a confirmation, but more than just the sheet of paper, the student cannot present anything more to the potential employer.

For the methodology, the examiners want to see that the student knows what he is doing and how. The development of such a methodology is not covered in the courses, which should help the student to develop its approach and methodology adapted to his strengths and weaknesses. This requires something that is not taught and is not even mentioned what it is to pay attention to. If we relate this to the point of creativity and say that the student must be able to do it himself, it is like an example when in an exam, the student is asked to build an engine he has never seen before.

Especially here, it is essential to learn not only to use tools for the identification of vulnerabilities but also to understand how these vulnerabilities can arise during configuration and administration. Instead, as penetration testers, we have to understand the tested environment better in a very short time than as administrators who took days, weeks, or even months for development.

In the end, the art is not to exploit the application, machine, or network but to find the way into it.

In summary, we see the following problems:

No.1 mostly the performance of the short-term memory is queried
No.2 limited and cost-intensive practice environments to internalize the learning material
No.3 Based on the learning material, the student is given the path
No.4 The orientation for problem-solving is predefined
No.5 Passing the test only leaves only a certificate as proof of completion
No.6 No material or explanations are provided for the development of methodology but are required for the exam
No.7 Due to the limited and predetermined working environments, the creativity of the student is limited
No.8 The individual strengths and weaknesses of the student are not considered

Some points have been omitted here, but these are the most important ones that should be clarified here.

3.0 New Examination Model

3.1 Goals of the Examination

The goal of the exam is to verify that the student has the theoretical knowledge and intermediate practical skills to test different types of systems for various vulnerabilities under high time pressure. Also, the student will be tested if he/she has developed a structured and logical methodology that allows him/her to creatively find ways to exploit the systems and their configurations for unauthorized access.

It should also be verified that the student has developed an understanding of how such vulnerabilities can occur in administration and configuration processes.
The certification should not only be a proof of completion of the course, but rather a confirmation of long and intensive theoretical and practical study in the field of Penetration Testing / Ethical Hacking, which also makes it possible for the student to present his work to potential employers in the future.

Summary:

Verify theoretical knowledge
Verify intermediate practical skills
Verify structured methodology
Verify understanding of how vulnerabilities can occur

3.2 Examination Phases

The certification process is divided into 4 phases. A maximum of 100 points can be collected. To pass the exam, the student has to collect at least 80 points .

Examination phases:

Course Material - Exercises
Lab Development
Live Presentation
Rank (Bonus)

3.2.1 Course Material

In the first phase, the student deals with the course material. Here the student gets 0.5 points for each solved exercise.

Actually, there are 77 evaluated exercises in total. This means that the student can earn a maximum of 38.5 points, but the point limit for this part is fixed at a maximum of 30 points . Exercises marked as “not required” are not evaluated.

This gives the student the flexibility to fulfill its abilities according to his own preferences and needs. Also, this solution provides the student with the possibility to skip 17 tasks and still achieve the full score.

Each exercise that takes the activity of the student in the form of testing or configuration requires at least one screenshot.
Students ID/Username must be visible in all screenshots.

This can be the username or a comment in the working environment. If no screenshot was taken for an exercise or the ID/Username is not visible in the screenshot, the exercise will be evaluated with 0 points. Each exercise must be clearly and unambiguously documented.

Solution for problem No. 3:

The learning material and the exercises do not prescribe a way for the student to use.

Solution for problem No.4:

When working through the course material, the exercises do not tell the student how to solve them. The student will come across problems and situations that can be individual due to the different approaches and require the student to review and analyze his/her own approach to find his/her mistakes.

Solution for problem No.6:

The course itself contains over 20 pages of material that deals only with the learning process, strengths, weaknesses, ways of learning, and the development of a methodology with many assisting elements.

Solution for problem No.8:

The student himself decides, according to his/her own strengths and weaknesses, how to solve the exercises. The student can skip some of the exercises and still get the full score for this phase. After all, we do not have to and cannot know everything. Therefore it is advantageous to use our strengths and minimize our weaknesses.

3.2.2 Lab Environment

During the course, the student starts to create a locally hosted vulnerable network that simulates a realistic scenario where the student performs a full penetration test.

This phase requires the creation of configuration documentation and a penetration testing report that explains step-by-step all changes and activities performed by the student.

These two documentations, in addition to the documentation for the exercises, must be made accessible to the examiner in a password-protected ZIP archive at least 5 days before the presentation date.

For the exam, he/she needs to set up and configure 5 vulnerable VMs, which are vulnerable in different ways. This means that only one vulnerability of e.g., RCE/LFI/RFI/XSS/SQLi or vulnerable application may be present in the entire configuration of the vulnerable Lab environment. The Lab must also contain at least one Linux and Windows-based operating system.

The point allocation depends on the proximity to reality and complexity. The student can achieve a maximum of 7 points per VM. This results in a total score of 35 points. For this category, a point limit of 25 points has been set.

Solution for problem no. 5:

In this phase, the student not only deals with each service and its configuration but also gets insight from an administrator’s perspective, which gives the student an understanding of the respective services, what information they work with and how vulnerabilities in the configurations can occur. The student will create two documentations that describe the configuration and its exploitation in detail. These documentations can later be presented to the potential employers as already performed performance in addition to the certificate, which in turn increases the chances for a commitment.

3.2.3 Live Presentation

The most important aspect of certification is its appreciation, which comes with the work done by the student. It is intended to confirm that the student has the appropriate knowledge and skills required for this. This is technical knowledge and practical skills, especially the confirmation of the development and use of their methodology.

Besides that, an essential skill that has to be verified is that the student has a structured methodology to approach scenarios and solve problems and find the way into the machine.

After the live session is established, the “Creativity Of Problem Solving” - scripts (COPS-Scripts) scripts come into play.
The COPS Scripts are small scripts created by the examiners based on the documentations provided by the students, which modify the Lab environment. These changes force the student to take specific steps necessary to find the path to the vulnerability. Additionally, these COPS scripts generate the user.txt and root.txt files with the appropriate hash values that the student must get. Also, these scripts change the passwords of the User and root/Administrator.

Such a lab will often be different, and the COPS scripts reduce the chances that the same environment will be used for the exams. This, in turn, prevents the exam from being leaked.

The student gets informed just during the presentation, which VMs will be changed by the COPS-Scripts and should be tested again.

In this phase, the practical skills, as well as the methodology and approach, are explicitly tested, as the student operates in a well-known environment to find the vulnerabilities.

3.2.4 HTB Rank (Bonus)

Since the student has the opportunity to practice for 1 year as a VIP on the HackTheBox Platform with over 140 VMs, he also can earn bonus points based on his rank.
A maximum of 10 points can be earned here.
With the rank “Script Kiddie”, the student receives 5 points.
If he reaches the rank “Hacker”, another 5 points will be awarded to the student for the exam.

A total of 10 points can be earned in this phase.

Solution for problem no. 1, 2, and 7:

Since the course is designed based on the HackTheBox Platform and the student can also get VIP status for cheap to get access to over 140 VMs with the course, it gives the student 1 year time to train and internalize his/her knowledge and skills.

4.0 Requirements for Passing

To be able to check the theoretical part, the student has to take an exam in which he/she creates services, configures them, and understands how they work and learns to understand how already found vulnerabilities could occur.

Another critical factor is that the student should not be given a path to use. It should be possible for the student to encourage their creativity and to extend their strengths in the best possible way and to improve their weaknesses as far as possible. These areas are covered by the first (1. Course Material - Exercises) and the second (2. Lab Development) phase.

The practical part is about the realization of theoretical knowledge and a logical and structured methodology and approach. To verify these skills and to further enhance the strengths and weaknesses of the student, the skills are tested in his/her local Lab and under high time pressure. An examiner must supervise and evaluate the student’s approach through a 60 minutes screen sharing session (3. Presentation).

This process uses the so-called “Creativity Of Problem Solving” - Scripts (COPS-Scripts), which change the student’s local Lab and make him/her take specific steps to find the way to the configurations or weaknesses.

As a bonus, the student can earn additional points with the practical area if he/she reaches specific ranks (4. Rank).

Summary:

Development and configuration experience
Creative thinking
Fast and smart learning methodology
Practical realization of the theory
Working under high time pressure
Attention to detail to solve problems

More information about this Penetration Testing course you can find at:
https://shell-affect.com

Let me hear your opinions and if you have any questions feel free to ask and I will try to answer them all.

Best regards,
Cry0l1t3