Whether you’re just starting to look, or have been looking for a while, you’ve probably discovered that choosing a testing software product can be a confusing and time-consuming process.
This is especially true for mid- to large-size organizations where there are lots of variables to consider and implementation involves IT, HR, management and a cast of characters.
Software-what are you looking for?
You may refer to it as a test generator, test maker, quiz maker, exam builder -- call it what you will -- it's an application that enables you to create, deliver and administer tests to your testing audience.
You want the software to store test data and enable you to analyze the results in some meaningful way. You may also want to print and/or export the data.
Depending on your frame of reference what we refer to as a “test” may be known to you as an assessment, paper, exam or quiz. Whatever you choose to call it, it’s a method used to measure the achievement or proficiency of your testing audience.
Who are You Testing?
In addition to the traditional view of students taking paper-and-pencil tests in a classroom, there’s the even broader category of test takers in non-academic settings. 
Your testing audience may be fortunate enough to have access to the company intranet or the internet. They may have a computer on their work desktop and another at home, or they may be working “
In your
Your Role 
Decision Maker. You’re the person who is initiating the search and will authorize the purchase. You’re the fact-finder, the evangelist and you have a pretty good idea what you want.
IT Person. You’re the technical services manager, or the IT person tasked with finding a testing solution requested by management.
Committee: You represent a group tasked by management with the responsibility of finding a testing solution that satisfies the group’s testing requirements.
Researcher: You’re an administrative assistant or intern who has been tasked with going out and gathering information and presenting your findings to upper management.
Looking for, Need, Want…
- You're looking for a feature-rich application
- You need to test several test takers at the same time
- You want to store, analyze, print and export your test results
- You're not looking for the cheapest thing you can find
Any test authoring application worth looking at has probably involved a lot of time and effort in its development and support.
Our advice: look for product that will continue to evolve as your testing needs grow, and one that is backed by great support.
If you're working on a limited budget, you might take a look at some of the open-source testing solutions out there. Implementing these types of solutions may require technical resources to support it.
Why Are You Looking?
We do Paper-and-Pencil Testing and...
- It’s time-consuming and inefficient
- You must grade your tests manually or
- You must run them through a scanner
- You can’t easily randomize your tests
- Your test data is not readily available or easily analyzed
- Your paper tests “walk”, there's not test security
You Are Experiencing Software “Issues”
You’re using an “in-house” solution and…
- The person who created it left the company or
- The software isn’t supported in Windows 10 or
- The software is “kludgy” and doesn’t work half the time
Your Current Testing Software
- Technical support is non-existent
- There’s no training and/or the application is difficult to use
- The application does not support the latest Windows OS
- Does not support AD/LDAP, SSO or SCORM
- Does not support installations in a Unix environment
- Customization services are not available
- Can’t be installed on our own servers
- Does not offer a secure browser option
You Provide Industry-Specific Online Training…
- You want to publish tests to your online customers.
- You want to allow your customers to self-register for online testing
- You also want to provide an online, e-commerce registration form.
You have an LMS? Yes, but...
- Doesn't support the question types I want to use.
- Minimal question banking options
- Randomization options are limited
- Reporting options are pretty basic
- They won't customize the testing
- The testing features I need are not supported
Deployment: Your Current Testing Software…
- Is subscription only and you want to own it
- Will only allow you to host on their servers, not yours
- Charges you for each registered user
- Charges you for additional administrative-level users
- Charges you per test created and/or test given
- Charges you to store your test data
About Part 2
We divided the blog into two parts so that you’d have an easier time of it. If you haven't already read Part 1, take a look at your convenience. For those of you who prefer one long read, we also stitched it together that way in our white paper <link>.
We'll cover
| Features | Your technical requirements |
| Budgets and Pricing | Customization |
| Your Testing Requirements | Consultation |
Features
List your top 3 to 5 must-have test generator features.
If you’re just getting started and aren’t quite sure what to ask for, we’ve included a short Features list below. If you need more assistance, we’d be happy to help, contact us.
In one column, jot down your “must have” requirements” and in the other column jot down a list of any “nice-to-have-but-not-absolutely-necessary” items that you’d like to have in your perfect world of testing.
Stay focused on the program features and test behaviors that you need now, but make sure the software is robust enough to accommodate future growth. You don’t want to start looking for (yet) another program because you’ve outgrown the current one.
If you already have an Excel file that includes a prioritized list of features and behaviors—great! Email the list to the vendor in advance of a call or demo. This should save both of you time and allow the vendor to determine whether their product matches your testing requirements.
Features List
- Question Banking
- Randomization
- Automatic Scoring
- Question types – list what you require
- Security – list what you require
- Media support
- Feedback
- Email notification
- Multiple language test content delivery
- …Add your own requirements to the list
If you are doing the initial research and reporting back to a committee or decision maker, it’s best to find out if there’s a budget or price range that you can work with.
There are lots of choices ranging from free to expensive. I’m assuming you’re looking for a test authoring application that is feature-rich, includes support, training
Most testing solutions worthy of your attention will have various pricing schedules depending on your requirements.
We don’t recommend starting a conversation with a vendor without having some idea about what you want and some idea as to how much you are willing to spend. Giving the vendor details is always helpful—saves time and signals that you are a serious buyer.
If your budgets are a bit tight, don’t be shy about asking the vendor if they offer monthly, quarterly or bi-annual payment options.
If you are
If you are new to online testing and have never gotten a testing system up and running, we recommend working with prospective vendors who are responsive and eager to help. For your part, explicitly defining your requirements and expectations will make it easier for these vendors to determine whether their products match what you’re looking for.
The better you prepare, the more efficient the implementation. If you are rushed you are likely to forget something and make mistakes—like choosing the wrong test maker software. It’s never a good idea to rush through the selection and deployment of a new testing system, especially if you are a mid- to
Right now, we are...
- Not testing, but we need to get something going very soon
- Not currently testing but plan on testing, beginning <date, time-frame>
- Thinking about it, doing research, no specific start date
- Doing paper-and-pencil testing and don’t like it.
- Doing paper-and-pencil testing, like it and want to continue doing it
- Looking for another testing app—our current one is no longer available
- Using a "home-grown", testing application that is outdated
- Wanting an application that we can own and host on our servers
Test Authoring: We want to…
- Create and print tests
- Create question banks and then...
- Print and distribute paper tests or…
- Publish and deliver tests over our local network or…
- Publish and deliver tests over our company’s intranet or…
- Publish and deliver tests over the internet or…
- Publish and deliver tests over the intra-internet and print tests
The following applies to those of you who are giving tests on a computer or tablet.
When we say “create” we mean that you are either typing your questions
This information is stored in a database on your personal computer, on a server, or the vendor’s server “somewhere” on the internet.
When we say “publish” we mean that you make a test available to members of your test-taking audience. The details regarding how and when that is done will vary based on the features and behaviors that the exam software supports.
In most cases, your test-taking audience’s eligibility to take a test depends on whether they have a user profile in the testing software’s database. How this is done and how this is managed will vary from test maker to test generator to exam builder.
Hosting your test generator
By “hosting” we mean, “where do you want your exam creation software installed?”. Your test generator software can sit on your internal equipment or it can be deployed on the vendor’s servers.
Where you place your test making software may influence the cost and any technical resources you may need to get your system up and running.
Which of these options represents you?
- We don’t know where we want to deploy our test generator software
- We want to host the software in our operations center
- We want the vendor to host the software on their servers
Our testing audience consists of…
- Myself
- A group of <X> students, employees, trainees, pre-hires, clients
- I don’t know how many we’ll have, we’re just getting this program off the ground.
Testing frequency...
- Daily, monthly, quarterly, annually
- Other
- Don’t know yet
Number of individuals to be tested
- Less than 5 individuals at the same time
- More than [X] at the same time
- Don’t know yet
We are interested in purchasing,,,
- A yearly subscription
- A “pay-for-it-once” (perpetual) license
Does the vendor charge for…
- Tests Created
- Tests Taken
- Number of Users
- Test Administrators
- Test Data:
dbsize and/or bandwidth - Support
- Training
- Consultation
If we need to modify the product …
- Does the vendor offer custom programming services?
- Can they migrate our test data to other 3rd party applications?
- Does the vendor offer a non-compete source code license option?
What are Your Technical Requirements
If you are the test administrator, author or trainer-person who is “not real technical”, we recommend involving IT. It's never a good idea to buy something and then "surprise" your IT department with your purchase.
If you need assistance discussing technical matters, the vendor should be more than willing to provide whatever technical assistance you need, by phone, email and/or online documentation.
- What versions of SQL Server are supported?
- Is AD/LDAP supported?
- What
web servers are supported? - Windows 10 support?... and so on.
If you need assistance, contact us we’d be happy to help,
If the vendor’s technical support team is slow to
Personalized Demo
As mentioned earlier, but bears repeating, before launching into an online demo—assuming the vendor offers this service—I suggest that you first find out if the software meets your testing requirements and budget. This can all be figured out through an email exchange or phone calls. If you don’t like to chat on the phone, email the vendor your requirements list and asked for some preliminary pricing. Keep in mind that the price may change based on taking a deeper dive into your requirements.
If the vendor tells you they can fulfill your requirements and work within your budget, ask to schedule an online demo. Live, online demos are a great way of making sure that there is no disconnect between what the vendor said, the product performance and the price.
If there are other members of your team that you would like to involve in the online session, see if you can coordinate schedules and give the vendor some dates/times that work for you. You may also
Customization
To avoid disappointments later, we recommend asking the vendor if they offer customization services before committing to a purchase. If you have a specific customization request in mind, it never hurts to ask for some pricing guidelines. If it’s a “must-have” feature requiring customization, ask the vendor if they can provide you with an estimate. If it’s a “must-have” feature can you afford the customization, if not, you may need to keep
Here are some possible options for your consideration.
If your customization is deemed minor by the vendor, they may be willing to include it in the package if you commit to a purchase.
If your customization request is more complex, we recommend providing the vendor with a written specification. It doesn’t have to be fancy or technical, it just needs to be concise with enough detail—pictures are also acceptable—to enable the vendor to provide a quote that they can honor.
It’s always a good idea to live with your purchase for a while, get comfortable with it. You may find that you don’t really need the customization you thought you needed, and discover that you need some other
Consultation
When choosing a vendor, look for someone who is going to be responsive and genuinely interested in helping you succeed. You want someone who will help guide you through the implementation process.
If you’re the “evangelist” who’s herding cats toward a more efficient method of testing, make sure the vendor has your back.
Finding Your Best Test Generator: Part 2 Coming Soon!
Here are some related links that you might find helpful.
How to Choose Business Software
How to Choose the Right Software Vendor
Ten Steps for Evaluating and Selecting Software and Service Providers
