Process Log 11

Date

Process Log 11

11/7-11/13

Auto-ethnography session on O'Reilly Media, Miro AI

CONTEXT

O'Reilly and Miro AI

PROCESS
I first asked both users to complete the pre-conception survey. I made sure each participant matched the prototype's target user to better follow their answers and observe their behavior in action.

I used a Figma prototype to test the AI nudge interface. I observed their behavior and noted any errors they encountered during user testing.

After user testing, I conducted the post-conception survey.

The surveys were designed to test their pre-conception of ethical concerns in using AI and how they use AI. The post-conception survey tested whether the interface and content were helpful to them and changed their perception of ethical principles in AI usage.

The surveys were completed on each participant's laptop, but Figma user testing was done on my laptop.

REFLECTIONS

Pre-User Testing > User Testing > Post-User Testing

Pre-User testing.
Duration: 5minutes
Data Collected: Survey responses of 10 questions from pre-testing and post-testing.

Some survey questions were hard to answer, and some overlapped and hard to understand what the questions was asking. Participant A quoted, "I do not understand what the question means, - "Have you ever had any time/moment that you were using AI too much and decided you do not want to use Generative AI tools?"

Participant A mentioned, "I don't think I use AI heavily so I don't think I have ethical concerns that much. I usually control my usage"

Changing in the format of the content was also needed to be changed. Participant B quoted*, "I wish this was multiple answers for the question, "When do you usually use Generative AI tools?"*

2. User Testing

Duration : 5 minutes
Data collected: During the User Testing, I collected observation notes, direct quotes during user testing,

It was hard for users to imagine their real experience of using an AI model. Because the Figma prototype is limited in its simulation, users struggled with the fake ChatGPT screenshot. They had to click to write a prompt but couldn't actually type anything. Participant A mentioned, "How can I type?"

The participant read the screenshot of ChatGPT 5 that was already filled with the question that was not relevant to them such as "What is Turing Machine?" The question used in the scenarios weren't relevant to them, so they had trouble understanding the scenarios themselves.

The scenarios might not have been engaging enough—users didn't spend much time reading or connecting with them.

The participant kept clicking the "next" button (which goes to the next scenario) without trying any interaction with the AI model screen.

3. Post-Usertesting.
Duration: 5minutes
Data Collected: Survey responses of 10 questions from pre-testing and post-testing.

There were too many scenarios to remember from user testing, making it hard for participants to recall them with the ethical concerns listed in the survey. Participant’s quoted, "There were so many, I cannot remember all scenarios. which were which..?"

"I chose 5, but will answer this anyway. I think I found that the prompts such as the time spent on the webpage might be a little hard to track since I may have had the page open on my browser without actually looking at it. But prompts such as reminders when you add images to prompt you about privacy would be helpful for people who are susceptible and very reliant on AI. I think an additional prompt could be the addition of prompts when they detect real names, phone numbers or any sensitive information."

"I think some of the prompts were interesting and would prompt me to think more, such as the fact-checking — this would be helpful as a reminder to check sources in case AI is hallucinating."

One participant didn't list sustainability as an ethical concern at first. But after seeing the prompt with the content, she thought her perspective on energy use might have been influenced by the prompt.

"Not really an answer to the question, but just explaining why I put the energy usage as the most useful for me even though I put sustainability as a low priority for me — this was because it was not something that I really thought of before? But seeing a number did put things in perspective for me? Just that how would you know how much energy the prompt is using? Is it arbitrary at the moment? Is there a possibility that you might start seeing the nudges as something to be taken for granted once they start popping up more frequently? What is the way to close these prompts?"

Both participants thought the feature would be useful and would likely use it if available.

It was interesting to see users learn new perspectives on the participant’s specific ethical concerns using generative AI.

Participant A quoted, "I would like to see prompt on copyright"

Overall

I liked the survey answers—they were really helpful and insightful.

I don't like how my user testing was designed. I didn't take any videos or transcribe the audio.

It was hard for user to empathize their time and moment of using the real AI model. Because Figma prototype is limited to certain simulation, the users had difficulties encountering the fake ChatGPT screenshot. They had to click to write prompt but was not able to write anything to prompt.

The questions that was used in the scenario was not relevant topic to them so they had hard time understanding the scenario itself.

The way of presenting the scenarios might not be too interesting enough so that the users had not taken much time to read and empathize with the scenarios.

INSPIRATION/NEW QUESTIONS

Questions I had after User Testing were: "What other questions or prompts should I create?", "What different types of interactions could I design?", "Should I build a real Google Chrome extension to test it? How technical do I want the testing to be?"

I need to find prompts that aren't specifically for designers. Since the participants were design students, ethical concerns like copyright were more important to them than to others.

There are other AI usage trackers available as Google Chrome extensions but most of them are about time used in AI models.

CONCRETE ACTION

I will try coding with Claude for the fake Chat box style AI model screen. Learn how to make Goolge Chrome extension. Design a physical cards with the same nudge content to prompt the users physically. Maybe use different medium to try prompting the user in different way other than just popup.