How do you test knowledge in the age of AI?

What if every class started with an essay assignment: 

Why you shouldn’t use AI to write this paper?

Over Thanksgiving I had an interesting conversation about The AI Problem with a university professor and a senior software engineer.

How do you test someone’s knowledge in the age of AI?

The professor, who relies on essays to test students’ mastery of the material, uses online AI checkers and hand-written exams.

(Remember blue books? They still exist!)

The engineer, who relies on technical puzzles to test problem-solving skills, asks candidates to close their eyes and describe how they would solve the problem. 

The consensus: AI use is fine... when you know what you’re doing. 

If you don't know what you're doing, you don't know enough to catch its errors. 

But if they got the question right, how do you know if it was because of AI or their knowledge?

Don’t forget, this is happening in the midst of every company pushing hard for AI adoption. No matter what happens with OpenAI, LLMs in the workplace are here to stay. 

The most successful workers of tomorrow will be AI native. They'll know how to work AI tools seamlessly into their workflows to enhance productivity.

But how do you test human knowledge when AI stupidity looks so much like human ignorance? 

Is the use of AI the problem, or just how it baffles the evaluators?

How do you create a question that will test someone’s knowledge and their ability to use AI to get the correct result?

What if you just asked them? 

What if, instead of asking a student to write about the ethics of the trolley problem or asking a candidate to digitally shuffle a deck of cards you asked,

“Why SHOULDN'T someone use AI to solve the trolley problem or digitally shuffle a deck of cards?”

This question will not only show the person’s command of AI in their area of expertise, it’ll also show their critical thinking, analytical skills, and knowledge of common logical fallacies in that field. 

Or... you know... you could keep handing out blue books and asking people to answer questions with their eyes closed. That works, too.

Join the conversation