Creating custom questions

Select an item from the list for more information on custom question creation:

Create a custom question in the question editor

To create your own custom question, click Questions in the left navigation menu.  

If this is the first question you have created, click Create your first question.

To view your existing list of custom questions, select the Custom tab on the top right of the screen.

To add a new custom question, click New question in the top right corner.

In the New Question pop up, select a Question type. There are four primary options:

• Multiple choice question
• Free text question
• Project exercise
• Coding exercise

If you select Coding exercises, you’ll be able choose from Function-based or Language-specific questions.

Once you select the question type and any type-specific configurations, you’ll then need to select the question language(s) and click Create question.

For programming exercise questions, you’ll also have the option of selecting a template that will pre-fill the question editor with a CoderPad-validated programming exercise.

Create multiple choice questions

• Input the Title (the candidate does not see the title).
• Write out the question in the Statement box on the left.
• Text options include common styling options, code inputs, LaTeX formulas, and adding images and attachments.
• On the right, input the answer options.
• Indicate the correct answer(s) with the switch.
• Select Multiple answers possible if there is more than one answer.
• Select Randomize order during the test to randomize the answer list order if required.
• The candidate must select ALL valid answers, and no other answers, to receive the maximum points available, otherwise marks are proportional.

Settings

Add the following details:

• The domain (i.e. technology or programming language) of your question where you can select an existing domain or create a new domain. 
• The skill to test.
• The question difficulty.
• Timer options.
• Total points available.
• Whether to add the question to automatically generated tests.
• Whether to authorize Screen to add this question to the Screen question library.

Click SAVE to save the question.

Create free text questions

Input the Title and write out the question in the Statement box below.

Add the Settings as before.

Select (1) Automatic or (2) Manual validation in the Validation section.

1. Automatic validation

Select the skill and input the answer to the question. Candidate answers must match exactly, but are case insensitive.

✅ Check the Regular expression box and input a regex code that allows a more flexible range of answers. Check the Oracle Java Regex description for more information.

2. Manual validation

Input a Label for the report. Select the Skill and Weight. The Points value auto-populates.

After a candidate completes a test with a question that needs manual validation, you will receive an email inviting you to manually validate the answer to the question. The system then calculates a final score.

Create language-specific exercise

Once you’ve selected Language-specific from the Coding exercise question type menu, you’ll be taken to the language selection window.

Once you select a language, you’ll be taken to additional settings. Here you can select the language for the question. Additionally, the Template dropdown contains existing Screen questions that you can use as a basis for your own.

You can edit the details of the existing question, including adding a zip file, before choosing the settings.

In Settings, input the points for the question.

✅ The domain field may have more options depending on the language selected.

If you want, you can add your question to the Screen library by checking the authorization box. Hover over the information circle to see the details:

You’ll also have the option to add custom files in the External dependency section. By clicking the Add external dependency button, you can add CSV files, text files, JSON files, or any other kind of files you’d like the candidate to work with.

⚠️ Your dependency files must be uploaded as `.zip` format.

Scroll down to see the Initial candidate answer code and the Initial candidate test code input boxes. This is the editable code your candidate sees at the start of the test.

The Code validator section stores validation code which runs against the candidate’s solution in order to assess it.

Underneath the Code validator is a Validator Mapping section where you can set the criteria for validation.

In all language-specific coding exercises, there is a parent validator and possibly one or more child validators. The child validators are only evaluated if the parent validator passes — if a candidate fails the parent validator, they automatically get 0 points for that question.

ℹ️ The reason for utilizing this parent/child logic is that you may want to test specific edge cases (e.g. input is null), but you don’t want to award points if the candidate has only implemented the edge cases (if input is null then x).

If you really want to disable this logic, you can put a test that always evaluates to true in the parent validator.

✅ The method field in the Validator Mapping must match the method name in the Code validator.

You can also try out a Possible solution and test your code. Click Preview to test the question:

You will see the question as the candidate sees it:

After testing and submitting, click Save to save the question.

Create project exercise

✅ Project exercises are usually longer than other kinds of questions. This type of question enables candidates to download a starter project and complete it in their own environment. They can then upload the finalized project for evaluation. Your team must then manually review the project based on the evaluation criteria

After selecting Project exercise in the question menu, you’ll be taken to the Additional settings window. Select the question language, and click then Create question to proceed.

Input a title and a statement for your project question.

In Settings, choose difficulty, timer, and points:

Then, click + Add your project to upload the project information that serves as a starting point for the candidate. All file types are acceptable.

Add the Evaluation Criteria you want to use, then Preview and/or Save.

✅Project exercises will need to be manually validated.

Function-based question

Function-based questions allow you to create language-agnostic exercises where the candidate can answer in the programming language they are most comfortable with.

Once you’ve selected Function-based from the Coding exercise question type menu and selected the questions language, you’ll be taken to the configuration page.

First, in the Instructions section, you’ll need to describe the goal of this question using the input box. You can use text, images, links, formulas — whatever you need — to describe this question to your candidates.

✅ You can change the language at the top right of the screen to see what the question would look like in other languages.

Also in the Instructions section you’ll see the Implementation of the functions. This section will be filled out automatically as you complete the rest of the steps in the question creation process.

Next, you’ll fill out the inputs in the Exercise section.

You’ll add the name of the function you want the candidate to write (1), as well as the name (2), type (3), and description (4) of both the input and output parameters.

You can add more parameters by clicking Add parameter (8). Additionally, certain input data types will allow you to add constraints (9), including min values, max values, and pattern matching.

On the right side of the screen, the initial code will automatically fill out as you add the different parts of the function description.

Next, in the validation section, you can add test cases (visible to the candidate) and validators (hidden from the candidate) that will run against the code to let you know if the candidate was able to solve the question.

For tests, you simply need to add a label (test name), the input, and the expected output. If you already have a possible solution, you can click the Generate from solution button to have the correct output automatically generated from your code. To add another test, simply click Add test.

For validators, you’ll need a label (validator name), a skill that the validator is testing, and the weight of the question — the higher the weight, the more points will be awarded for passing that particular validator. If you already have a possible solution, you can click the Generate from solution button to have the correct output automatically generated from your code. To add another validator, simply click Add validator.

Once your possible solution is written, simply click the Validate the solution button below the test case/validator section to see the results.

Next you’ll fill out the question settings in the Settings section. Here you can edit:

• Question title
• Question domain
• Difficulty level
• Duration
• Point value
• Language
• Allow use of the question for test generation
• Allow CoderPad to use the question for our question library

Lastly, you’ll be able to view your question from the candidate’s perspective in the Preview section at the bottom.

Add translations

Whenever you create a new question, if you select more than one language, you will find tabbed areas for inputting the text in other languages.

Custom question video tutorial

For more information on creating custom questions, check out this Creating custom question with CoderPad Screen tutorial video: