User Experience with LLM-based Conversational Systems

Welcome to participate in THUIR's research on Conversational Service Systems!
This questionnaire consists of 14 questions and will take 5-10 minutes to complete. The questionnaire will not record your personal information, and all answers will only be used for general statistics only.

Please select the services you have used (in Descending Order of Frequency)
【排序题】

Options are sorted alphabetically

Baichuan
Bard
Bloom
Character.AI
ChatGLM
ChatGPT
Claude
DeepSeek
ERNIE Bot (Baidu)
Gemini
Kimi Chat
Llama
New Bing
Poe
Qwen (Alibaba)
Others
*

Your frequency of usage

Daily

Weekly

Monthly

Tried a few times

Never

Which of the following do you use the system for【多选题】

Information Retrieval

Fast and direct access to factual information

Solve Problem in Specialized Areas

Seek answers or explanations in the field of programming, natural sciences, humanities, social sciences, etc.

Address and learn about the profession

Use through API

Use through Application Programming Interface instead of user interfaces

Explore and test the capabilities of LLM, such as evaluating it on various tasks, simulating agents, environments, or datasets, etc.

Text Assistant

Summarize, translate, revise, generate text, etc.

Seek Creativity and Inspiration

Brainstorming for inspiration, innovative ideas, etc.

Ask for Advice

Career development, personal counseling, gift recommendation, etc.,

or creating personal schedules, travel plans, shopping lists, etc.

Leisure

Movie and music recommendations, games, and other entertaining activities

Others

Do you have anything to add or comment to the categories above?

For the following usage scenarios, your rating on the system performance is?

If you have not used the system for certain requirements, please select N/A

	very dissatisfied	dissatisfied	neutral	satisfied	very satisfied	N/A
Information Retrieval
Information Retrieval
Solve Problem in Specialized Areas
Solve Problem in Specialized Areas
Use through API
Use through API
Text Assistant
Text Assistant
Seek Creativity and Inspiration
Seek Creativity and Inspiration
Ask for Advice
Ask for Advice
Leisure
Leisure
Others
Others

In each scenario, which of the following tools helps the system generate higher-quality responses?
【多选题】

Function Description:
Web Browsing: use search engines, browse and analyze web content, etc.
Input Analysis: understand uploaded documents, pictures, etc.
Personalization: recommend content based on user interests
Programming: write and execute code
Mathematical Operations: Assist with mathematical calculations, logical reasoning
Documentation Generation: create diagrams, documents and slides
Multimedia Creation: generate pictures and videos
Other tools: please specify

	Web Browsing	Input Analysis	Personalization	Programming	Math Operations	Doc Generation	Multimedia creation	Others
Information Retrieval
Information Retrieval
Solve Problem in Specialized Areas
Solve Problem in Specialized Areas
Use through API
Use through API
Text Assistant
Text Assistant

In each scenario, which of the following tools helps the system generate higher-quality responses?
【多选题】

	Web Browsing	Input Analysis	Personalization	Programming	Math Operations	Doc generation	Multimedia Creation	Others
Seek Creativity and Inspiration
Seek Creativity and Inspiration
Ask for Advice
Ask for Advice
Leisure
Leisure

For the following usage scenarios, do you think the system’s response needs to be [direct answer] or [detailed explanation]?

	Direct Answer	Detailed Explanation	It Depends
Information Retrieval
Information Retrieval
Solve Problem in Specialized Areas
Solve Problem in Specialized Areas
Use through API
Use through API
Text Assistant
Text Assistant
Seek Creativity and Inspiration
Seek Creativity and Inspiration
Ask for Advice
Ask for Advice
Leisure
Leisure

Please select B for this question

10.

For the following usage scenarios, do you think the system’s answers need to be [based on facts] or [based on creations]?

Based on facts
Based on creation: based on opinions, imagination and creativity

	Factual	Creative	It Depends
Information Retrieval
Information Retrieval
Solve Problem in Specialized Areas
Solve Problem in Specialized Areas
Use through API
Use through API
Text Assistant
Text Assistant
Seek Creativity and Inspiration
Seek Creativity and Inspiration
Ask for Advice
Ask for Advice
Leisure
Leisure

11.

For the following usage scenarios, do you think the system’s answers need to be [based on professional knowledge] or [based on common sense]?

	Professional Knowledge	Common Sense	It Depends
Information Retrieval
Information Retrieval
Solve Problem in Specialized Areas
Solve Problem in Specialized Areas
Use through API
Use through API
Text Assistant
Text Assistant
Seek Creativity and Inspiration
Seek Creativity and Inspiration
Ask for Advice
Ask for Advice
Leisure
Leisure

12.

Please briefly describe 3-5 recent conversations you have had with the Generative AI system (you can refer to the title generated by the system, or summarize by yourself)【多选题】

Please fill in the description in the corresponding category, if you cannot determine please fill in the others

Information Retrieval

Solve Problem in Specialized Areas

Use through API

Text Assistant

Seek Creativity and Inspiration

Ask for Advice

Leisure

Others

13.

Which aspects of the existing system do you think are lacking and most in need of optimization?【多选题】

Hallucination

The system may produce content that is inconsistent with facts or user instructions and needs to improve accuracy and reliability

Long context processing

Ability to process and accurately understand long input text without problems such as forgetting

Multimodal understanding and generation

Enhance the system's ability to process and generate multimedia content such as images, audio or video

Personalization

Provide personalized services based on users' history, such as recommending contents based on user interest

Privacy

Protect the privacy of user data

Safety

Not creating dangerous or inflammatory content

Other

14.

Do you have other comments and suggestions about this questionnaire or generative AI services? Thanks for your participation and support

评价对象得分 