Assignment 1: Evaluating ChatGPT for academic writing
Assessment Overview
Evaluating ChatGPT for Academic Writing
A vital part of this course is understanding the value of well written text, containing correctly cited and referenced material. We are going to evaluate ChatGPT to determine how well it creates academic text. Your Task We are going to use ChatGPT to create an essay and then assess the quality and reliability of the references. To complete this task, you will need to complete 6 steps. These steps are being released in line with some additional sessions and surveys to enable students to obtain the most benefit from the exercise. Please ensure that you regularly check the assessment material when updates are announced. Theses updates are being released over the next few weeks. The following tasks that form this assignment are:
1. Create a ChatGPT essay using the provided title.
2. Review the content of the Essay.
3. Evaluate the sources used in the essay
. 4. Complete the questionnaire provided on Moodle.
5. Write a reflection on what you have discovered and how this impacts your use of ChatGPT and similar systems.
6. Submission of your work via Moodle.
Task 1: Create a ChatGPT Essay
Students should use one of the following degree specific phrases to use ChatGPT to create an essay. Copy the title and paste this into the” Send message” box in ChatGPT.
• Cyber Security students:” Write 1000 words with at least five in text citations and references on the challenges of malware in cyber security”.
ChatGPT does not always produce the output you desire, so you may need to use additional prompts to improve the essay. Possible suggestions are:
• Repeat the request and ask for in line citations
• Repeat the request and ask for journal papers to be cited.
Copy the essay from the ChatGPT web interface to a Word-processor document on your local computer as this will form part of your final submission. Do not alter the ChatGPT essay in anyway. You should next complete the following survey on the use of ChatGPT. The survey responses are completely anonymous and confidential. The aggregated opinion provides a class perspective that ensures the topic of ChatGPT can be appropriately addressed. Aggregated results of the data from this research will be used to adjust the content of our next lecture on the same topic and to provide formative feedback as part of your educational journey. A QR code in Figure 1 also provides a link to the survey.
Task 2: Review the ChatGPT Essay
Read through the essay. Using the following outline properties write a short summary (around 500 words) outlining your opinion of the essay.
• Structure and flow: How well structured is the essay? Does it provide a good outline to the arguments? How well does it flow with one paragraph leading to the next?
• Critical insight: Does it demonstrate a critical understanding of the subject or is it just a summary of the topic?
• Inherent bias: Does it appear biased in the way it is written or does it present a balanced argument? Is the language used objective?
• Writing style: How well written is the essay, is it overly wordy and verbose? Does it use consistent terminology?
• Objectivity and evidence: Does it cite research? Does it provide data and statistics? Does it use peer reviewed resources?
• Writing tone: Does it avoid informal expressions and jargon? Is the grammar and punctuation appropirate and correct? Your submission for this task (due at the end of the assignment on the 6th October 2023) is:
• Your summary (maximum of 500 words) giving your impression on the essay
Task 3 : Evaluation of ChatGPT Sources
The next task is to evaluate the sources used in the essay. You should consider the following questions, producing your results in the form of a table.
1. Select one of the journal references
2. Does the reference exist? This can be checked with a search of Google scholar Google Scholar, or possibly by using Research Gate or one of NUC’s online library facilities such as Science Direct or ACM Journals
3. For those references that exist, what is the quality of the reference? For journals this would be one of Q1, Q2, Q3 or Q4.
4. Does the citation used in the ChatGPT text appear to be appropriate?
The table for reviewing references
Reference used in essay
Reference used in essay Correct Reference (if found, otherwise ’Not found’) Quality of Reference (if found, otherwise ’Not found Citation Appropriate (yes / no / not sure)
Smith (2022) Security of Whatsit Smith B. and Jones A. (2022) Security of Whatsits, Journal of Made up Stuff. Vol 4. Issue 3 p10-19 DOI 12345.67.89 Q3 No
Task 4 : ChatGPT Evaluation Questionnaire
You should use the information you have created in task 2 and task 3 to help you to complete the questionnaire provided on the Moodle course page. This questionnaire will become available after the lectures on Friday 8 September. The title of the questionnaire is shown in figure 2. You will find the questionnaire in the assessment block of the course page.
This questionnaire must be completed. It is mandatory as it forms part of the assessment for the course. Your submission for this task is:
• To complete the Moodle questionnaire, it is important that this task is completed in the period 8 September 2023 to 22 September 2023 to allow time for the next task.
Task 5 : A reflection on the use of ChatGPT
Write a short reflection (a maximum of 1500 words) outlining your thoughts on the use of ChatGPT. The following questions might help focus your reflection.
1. In your opinion, did the text of the ChatGPT essay address the subject in sufficient detail?
2. Was the output from ChatGPT better or worse than you expected?
3. What do you think is the best use for tools like ChatGPT?
4. How might you use ChatGPT in the future?
Your submission (due at the end of the assignment) for this task is:
• A short essay reflecting on how you might use ChatGPT in the future
. • It is important that this task is completed by 6th October, the submission deadline for the whole assignment.
Task 6 : Your Submission
Your submission for the course containing tasks 1, 2 3, and 5 should be in the form of a single wordprocessed PDF document. Task4, the questionnaire is completed in Moodle. A template for the submission is provided and contains the following:
1. A title page. Complete this with your name, the degree you are studying, the course name and code, the title of the ’essay’.
2. Table of Contents.
3. The essay you created and copied from ChatGPT. (Task 1)
4. Your short summary outlining your thoughts on the quality of the essay. (Task 2) 5. Your table containing the evaluation of one reference used in the ’essay’ (Task 3). 6. Your reflection on the use of ChatGPT (Task 5).