essay in references

How to Reference in an Essay (9 Strategies of Top Students)

Are you feeling overwhelmed by referencing?

When you’re first asked to do referencing in an essay it can be hard to get your head around it. If it’s been a while since you were first taught how to reference, it can be intimidating to ask again how to do it!

I have so many students who consistently lose marks just because they didn’t get referencing right! They’re either embarrassed to ask for extra help or too lazy to learn how to solve the issues.

So, here’s a post that will help you solve the issues on your own.

Already think you’re good at referencing? No worries. This post goes through some surprising and advanced strategies for anyone to improve no matter what level you are at!

In this post I’m going to show you exactly how to reference in an essay. I’ll explain why we do it and I’ll show you 9 actionable tips on getting referencing right that I’m sure you will not have heard anywhere else!

The post is split into three parts:

• What is a Reference and What is a Citation?
• Why Reference? (4 Things you Should Know)
• How to Reference (9 Strategies of Top Students)

If you think you’ve already got a good understanding of the basics, you can jump to our 9 Advanced Strategies section.

Part 1: What is a Reference and What is a Citation?

What is a citation.

An in-text mention of your source. A citation is a short mention of the source you got the information from, usually in the middle or end of a sentence in the body of your paragraph. It is usually abbreviated so as not to distract the reader too much from your own writing. Here’s two examples of citations. The first is in APA format. The second is in MLA format:

• APA: Archaeological records trace the original human being to equatorial Africa about 250,000–350,000 years ago (Schlebusch & Jakobsson, 2018) .
• MLA: Archaeological records trace the original human being to equatorial Africa about 250,000–350,000 years ago (Schlebusch and Jakobsson 1) .

In APA format, you’ve got the authors and year of publication listed. In MLA format, you’ve got the authors and page number listed. If you keep reading, I’ll give some more tips on formatting further down in this article.

And a Reference is:

What is a Reference?

A reference is the full details of a source that you list at the end of the article. For every citation (see above) there needs to be a corresponding reference at the end of the essay showing more details about that source. The idea is that the reader can see the source in-text (i.e. they can look at the citation) and if they want more information they can jump to the end of the page and find out exactly how to go about finding the source.

Here’s how you would go about referencing the Schlebusch and Jakobsson source in a list at the end of the essay. Again, I will show you how to do it in APA and MLA formats:

• APA: Schlebusch, C. & Jakobsson, M. (2018). Tales of Human Migration, Admixture, and Selection in Africa. Annual Review of Genomics and Human Genetics , 11 (33), 1–24.
• MLA: Schlebusch, Carina and Mattias Jakobsson. “Tales of Human Migration, Admixture, and Selection in Africa.” Annual Review of Genomics and Human Genetics , vol. 11, no. 33, 2018, pp. 1–24.

In strategy 1 below I’ll show you the easiest and fool proof way to write these references perfectly every time.

One last quick note: sometimes we say ‘reference’ when we mean ‘citation’. That’s pretty normal. Just roll with the punches. It’s usually pretty easy to pick up on what our teacher means regardless of whether they use the word ‘reference’ or ‘citation’.

Part 2: Why Reference in an Essay? (4 Things you Should Know)

Referencing in an essay is important. By the time you start doing 200-level courses, you probably won’t pass the course unless you reference appropriately. So, the biggest answer to ‘why reference?’ is simple: Because you Have To!

Okay let’s be serious though … here’s the four top ‘real’ reasons to reference:

1. Referencing shows you Got an Expert’s Opinion

You can’t just write an essay on what you think you know. This is a huge mistake of beginning students. Instead this is what you need to do:

Top Tip: Essays at university are supposed to show off that you’ve learned new information by reading the opinions of experts.

Every time you place a citation in your paragraph, you’re showing that the information you’re presenting in that paragraph was provided to you by an expert. In other words, it means you consulted an expert’s opinion to build your knowledge.

If you have citations throughout the essay with links to a variety of different expert opinions, you’ll show your marker that you did actually genuinely look at what the experts said with an open mind and considered their ideas.

This will help you to grow your grades.

2. Referencing shows you read your Assigned Readings

Your teacher will most likely give you scholarly journal articles or book chapters to read for homework between classes. You might have even talked about those assigned readings in your seminars and tutorials.

Great! The assigned readings are very important to you.

You should definitely cite the assigned readings relevant to your essay topic in your evaluative essay (unless your teacher tells you not to). Why? I’ll explain below.

• Firstly, the assigned readings were selected by your teacher because your teacher (you know, the person who’s going to mark your essay) believes they’re the best quality articles on the topic. Translation: your teacher gave you the best source you’re going to find. Make sure you use it!
• Secondly, by citing the assigned readings you are showing your teacher that you have been paying attention throughout the course. You are showing your teacher that you have done your homework, read those assigned readings and paid attention to them. When my students submit an essay that has references to websites, blogs, wikis and magazines I get very frustrated. Why would you cite low quality non-expert sources like websites when I gave you the expert’s article!? Really, it frustrates me so, so much.

So, cite the assigned readings to show your teacher you read the scholarly articles your teacher gave to you. It’ll help you grow your marks.

3. Referencing deepens your Knowledge

Okay, so you understand that you need to use referencing to show you got experts’ opinions on the topic.

But there’s more to it than that. There’s actually a real benefit for your learning.

If you force yourself to cite two expert sources per paragraph, you’re actually forcing yourself to get two separate pieces of expert knowledge. This will deepen your knowledge!

So, don’t treat referencing like a vanity exercise to help you gain more marks. Actually view it as an opportunity to develop deeper understandings of the topic!

When you read expert sources, aim to pick up on some new gems of knowledge that you can discuss in your essays. Some things you should look out for when finding sources to reference:

• Examples that link ideas to real life. Do the experts provide real-life examples that you can mention in your essay?
• Facts and figures. Usually experts have conducted research on a topic and provide you with facts and figures from their research. Use those facts and figures to deepen your essay!
• Short Quotes. Did your source say something in a really interesting, concise or surprising way? Great! You can quote that source in your essay .
• New Perspectives. Your source might give you another perspective, angle or piece of information that you can add to your paragraph so that it’s a deep, detailed and interesting paragraph.

So, the reason we ask you to reference is at the end of the day because it’s good for you: it helps you learn!

4. Referencing backs up your Claims

You might think you already know a ton of information about the topic and be ready to share your mountains of knowledge with your teacher. Great!

So, should you still reference?

Yes. Definitely.

You need to show that you’re not the only person with your opinion. You need to ‘stand on the shoulders of giants.’ Show what other sources have said about your points to prove that experts agree with you.

You should be saying: this is my opinion and it’s based on facts, expert opinions and deep, close scrutiny of all the arguments that exist out there .

If you make a claim that no one else has made, your teacher is going to be like “Have you even been reading the evidence on this topic?” The answer, if there are no citations is likely: No. You haven’t.

Even if you totally disagree with the experts, you still need to say what their opinions are! You’ll need to say: “This is the experts’ opinions. And this is why I disagree.”

So, yes, you need to reference to back up every claim. Try to reference twice in every paragraph to achieve this.

Part 3: Strategies for How to Reference in an Essay (9 Strategies of Top Students)

Let’s get going with our top strategies for how to reference in an essay! These are strategies that you probably haven’t heard elsewhere. They work for everyone – from beginner to advanced! Let’s get started:

1. Print out your Reference Style Cheat Sheet

Referencing is hard and very specific. You need to know where to place your italics, where the commas go and whether to use an initial for full name for an author.

There are so many details to get right.

And here’s the bad news: The automated referencing apps and websites nearly always get it wrong! They tell you they can generate the citation for you. The fact of the matter is: they can’t!

Here’s the best way to get referencing right: Download a referencing cheat sheet and have it by your side while writing your essay.

Your assignment outline should tell you what type of referencing you should use. Different styles include: APA Style, MLA Style, Chicago Style, Harvard Style, Vancouver Style … and many more!

You need to find out which style you need to use and download your cheat sheet. You can jump onto google to find a cheat sheet by typing in the google bar:

Download a pdf version of the referencing style cheat sheet, print it out, and place it on your pinboard or by your side when writing your essay.

2. Only cite Experts

There are good and bad sources to cite in an essay.

You should only cite sources written, critiqued and edited by experts. This shows that you have got the skill of finding information that is authoritative. You haven’t just used information that any old person popped up on their blog. You haven’t just gotten information from your local newspaper. Instead, you got information from the person who is an absolute expert on the topic.

Here’s an infographic listing sources that you should and shouldn’t cite. Feel free to share this infographic on social media, with your teachers and your friends:

3. Always use Google Scholar

Always. Use. Google. Scholar.

Ten years ago students only had their online university search database to find articles. Those university databases suck. They rarely find the best quality sources and there’s always a big mix of completely irrelevant sources mixed in there.

Google Scholar is better at finding the sources you want. That’s because it looks through the whole article abstract and analyses it to see if it’s relevant to your search keywords. By contrast, most university search databases rely only on the titles of articles.

Use the power of the best quality search engine in the world to find scholarly sources .

Note: Google and Google Scholar are different search engines.

To use Google Scholar, go to: https://scholar.google.com

Then, search on google scholar using keywords. I’m going to search keywords for an essay on the topic: “What are the traits of a good nurse?”

If you really like the idea of that first source, I recommend copying the title and trying your University online search database. Your university may give you free access.

4. Cite at least 50% sources you found on your Own Research

Okay, so I’ve told you that you should cite both assigned readings and readings you find from Google Scholar.

Here’s the ideal mix of assigned sources and sources that you found yourself: 50/50.

Your teacher will want to see that you can use both assigned readings and do your own additional research to write a top essay . This shows you’ve got great research skills but also pay attention to what is provided in class.

I recommend that you start with the assigned readings and try to get as much information out of them, then find your own additional sources beyond that using Google Scholar.

So, if your essay has 10 citations, a good mix is 5 assigned readings and 5 readings you found by yourself.

5. Cite Newer Sources

As a general rule, the newer the source the better .

The best rule of thumb that most teachers follow is that you should aim to mostly cite sources from the past 10 years . I usually accept sources from the past 15 years when marking essays.

However, sometimes you have a really great source that’s 20, 30 or 40 years old. You should only cite these sources if they’re what we call ‘seminal texts’. A seminal text is one that was written by an absolute giant in your field and revolutionized the subject.

Here’s some examples of seminal authors whose old articles you would be able to cite despite the fact that they’re old:

• Education: Vygotsky, Friere, Piaget
• Sociology: Weber, Marx, C. Wright Mills
• Psychology: Freud, Rogers, Jung

Even if I cite seminal authors, I always aim for at least 80% of my sources to have been written in the past 10 years.

6. Reference twice per Paragraph

How much should you reference?

Here’s a good strategy: Provide two citations in every paragraph in the body of the essay.

It’s not compulsory to reference in the introduction and conclusion . However, in all the other paragraphs, aim for two citations.

Let’s go over the key strategies for achieving this:

• These two citations should be to different sources, not the same sources twice;
• Two citations per paragraph shows your points are backed up by not one, but two expert sources;
• Place one citation in the first half of the paragraph and one in the second half. This will indicate to your marker that all the points in the whole paragraph are backed up by your citations.

This is a good rule of thumb for you when you’re not sure when and how often to reference. When you get more confident with your referencing, you can mix this up a little.

7. The sum total of your sources should be minimum 1 per 150 words

You can, of course, cite one source more than once throughout the essay. You might cite the same source in the second, fourth and fifth paragraphs. That’s okay.

But, you don’t want your whole essay to be based on a narrow range of sources. You want your marker to see that you have consulted multiple sources to get a wide range of information on the topic. Your marker wants to know that you’ve seen a range of different opinions when coming to your conclusions.

When you get to the end of your essay, check to see how many sources are listed in the end-text reference list. A good rule of thumb is 1 source listed in the reference list per 150 words. Here’s how that breaks down by essay size:

• 1500 word essay: 10 sources (or more) listed in the reference list
• 2000 word essay: 13 sources (or more) listed in the reference list
• 3000 word essay: 20 sources (or more) listed in the reference list
• 5000 word essay: 33 sources (or more) listed in the reference list

8. Instantly improve your Reference List with these Three Tips

Here’s two things you can do to instantly improve your reference list. It takes less than 20 seconds and gives your reference list a strong professional finish:

a) Ensure the font size and style are the same

You will usually find that your whole reference list ends up being in different font sizes and styles. This is because you tend to copy and paste the titles and names in the citations from other sources. If you submit the reference list with font sizes and styles that are not the same as the rest of the essay, the piece looks really unprofessional.

So, quickly highlight the whole reference list and change its font to the same font size and style as the rest of your essay. The screencast at the end of Step 8 walks you through this if you need a hand!

b) List your sources in alphabetical order.

Nearly every referencing style insists that references be listed in alphabetical order. It’s a simple thing to do before submitting and makes the piece look far more professional.

If you’re using Microsoft Word, simply highlight your whole reference list and click the A>Z button in the toolbar. If you can’t see it, you need to be under the ‘home’ tab (circled below):

You’ve probably never heard of a hanging indent. It’s a style where the second line of the reference list is indented further from the left-hand side of the page than the first line. It’s a strategy that’s usually used in reference lists provided in professional publications.

If you use the hanging indent, your reference list will look far more professional.

Here’s a quick video of me doing it for you:

9. Do one special edit especially for Referencing Style

The top students edit their essays three to five times spaced out over a week or more before submitting. One of those edits should be specifically for ensuring your reference list adheres to the referencing style that your teacher requires.

To do this, I recommend you get that cheat sheet printout that I mentioned in Step 1 and have it by your side while you read through the piece. Pay special attention to the use of commas, capital letters, brackets and page numbers for all citations. Also pay attention to the reference list: correct formatting of the reference list can be the difference between getting the top mark in the class and the fifth mark in the class. At the higher end of the marking range, things get competitive and formatting of the reference list counts.

A Quick Summary of the 9 Top Strategies…

Follow the rules of your referencing style guide (and that cheat sheet I recommended!) and use the top 9 tips above to improve your referencing and get top marks. Not only will your referencing look more professional, you’ll probably increase the quality of the content of your piece as well when you follow these tips!

Here’s a final summary of the 9 top tips:

Strategies for How to Reference in an Essay (9 Strategies of Top Students)

• Print out your Reference Style Cheat Sheet
• Only cite Experts
• Always use Google Scholar
• Cite at least 50% sources you found on your Own Research
• Cite Newer Sources
• Reference twice per Paragraph
• The sum total of your sources should be minimum 1 per 150 words
• Instantly improve your Reference List with these Three Tips
• Do one special edit especially for Referencing Style

Chris Drew (PhD)

Dr. Chris Drew is the founder of the Helpful Professor. He holds a PhD in education and has published over 20 articles in scholarly journals. He is the former editor of the Journal of Learning Development in Higher Education. [Image Descriptor: Photo of Chris]

• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ 15 Self-Actualization Examples (Maslow's Hierarchy)
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ Forest Schools Philosophy & Curriculum, Explained!
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ Montessori's 4 Planes of Development, Explained!
• Chris Drew (PhD) https://helpfulprofessor.com/author/chris-drew-phd/ Montessori vs Reggio Emilia vs Steiner-Waldorf vs Froebel

Leave a Comment Cancel Reply

Your email address will not be published. Required fields are marked *

How to Write an Academic Essay with References and Citations

#scribendiinc

Written by  Scribendi

If you're wondering how to write an academic essay with references, look no further. In this article, we'll discuss how to use in-text citations and references, including how to cite a website, how to cite a book, and how to cite a Tweet, according to various style guides.

You might need to cite sources when writing a paper that references other sources. For example, when writing an essay, you may use information from other works, such as books, articles, or websites. You must then inform readers where this information came from. Failure to do so, even accidentally, is plagiarism—passing off another person's work as your own.

You can avoid plagiarism and show readers where to find information by using citations and references. 

Citations tell readers where a piece of information came from. They take the form of footnotes, endnotes, or parenthetical elements, depending on your style guide. In-text citations are usually placed at the end of a sentence containing the relevant information. 

A reference list , bibliography, or works cited list at the end of a text provides additional details about these cited sources. This list includes enough publication information allowing readers to look up these sources themselves.

Referencing is important for more than simply avoiding plagiarism. Referring to a trustworthy source shows that the information is reliable. Referring to reliable information can also support your major points and back up your argument. 

Learning how to write an academic essay with references and how to use in-text citations will allow you to cite authors who have made similar arguments. This helps show that your argument is objective and not entirely based on personal biases.

How Do You Determine Which Style Guide to Use?

Often, a professor will assign a style guide. The purpose of a style guide is to provide writers with formatting instructions. If your professor has not assigned a style guide, they should still be able to recommend one. 

If you are entirely free to choose, pick one that aligns with your field (for example, APA is frequently used for scientific writing). 

Some of the most common style guides are as follows:

AP style for journalism

Chicago style for publishing

APA style for scholarly writing (commonly used in scientific fields)

MLA style for scholarly citations (commonly used in English literature fields)

Some journals have their own style guides, so if you plan to publish, check which guide your target journal uses. You can do this by locating your target journal's website and searching for author guidelines.

How Do You Pick Your Sources?

When learning how to write an academic essay with references, you must identify reliable sources that support your argument. 

As you read, think critically and evaluate sources for:

Objectivity

Keep detailed notes on the sources so that you can easily find them again, if needed.

Tip: Record these notes in the format of your style guide—your reference list will then be ready to go.

How to Use In-Text Citations in MLA

An in-text citation in MLA includes the author's last name and the relevant page number: 

(Author 123)

How to Cite a Website in MLA

Here's how to cite a website in MLA:

Author's last name, First name. "Title of page."

Website. Website Publisher, date. Web. Date

retrieved. <URL>

With information from a real website, this looks like:

Morris, Nancy. "How to Cite a Tweet in APA,

Chicago, and MLA." Scribendi. Scribendi

Inc., n.d. Web. 22 Dec. 2021.

<https://www.scribendi.com/academy/articles/how_to_cite_a_website.en.html>

How Do You Cite a Tweet in MLA ?

MLA uses the full text of a short Tweet (under 140 characters) as its title. Longer Tweets can be shortened using ellipses. 

MLA Tweet references should be formatted as follows:

@twitterhandle (Author Name). "Text of Tweet." Twitter, Date Month, Year, time of

publication, URL.

With information from an actual Tweet, this looks like:

@neiltyson (Neil deGrasse Tyson). "You can't use reason to convince anyone out of an

argument that they didn't use reason to get into." Twitter, 29 Sept. 2020, 10:15 p.m.,

https://twitter.com/neiltyson/status/1311127369785192449 .

How to Cite a Book in MLA

Here's how to cite a book in MLA:

Author's last name, First name. Book Title. Publisher, Year.

With publication information from a real book, this looks like:

Montgomery, L.M. Rainbow Valley. Frederick A. Stokes Company, 1919.

How to Cite a Chapter in a Book in MLA

Author's last name, First name. "Title of Chapter." Book Title , edited by Editor Name,

Publisher, Year, pp. page range.

With publication information from an actual book, this looks like:

Ezell, Margaret J.M. "The Social Author: Manuscript Culture, Writers, and Readers." The

Broadview Reader in Book History , edited by Michelle Levy and Tom Mole, Broadview

Press, 2015,pp. 375–394.

How to  Cite a Paraphrase in MLA

You can cite a paraphrase in MLA exactly the same way as you would cite a direct quotation. 

Make sure to include the author's name (either in the text or in the parenthetical citation) and the relevant page number.

How to Use In-Text Citations in APA

In APA, in-text citations include the author's last name and the year of publication; a page number is included only if a direct quotation is used: 

(Author, 2021, p. 123)

How to Cite a Website in APA

Here's how to cite a website in APA:

Author, A. A., & Author, B. B. (Year, Month. date of publication). Title of page. https://URL

Morris, N. (n.d.). How to cite a Tweet in APA, Chicago, and MLA. 

https://www.scribendi.com/academy/articles/how_to_cite_a_website.en.html       

Tip: Learn more about how to write an academic essay with  references to websites .

How Do You  Cite a Tweet in APA ?

APA refers to Tweets using their first 20 words. 

Tweet references should be formatted as follows:

Author, A. A. [@twitterhandle). (Year, Month. date of publication). First 20 words of the

Tweet. [Tweet] Twitter. URL

When we input information from a real Tweet, this looks like:

deGrasse Tyson, N. [@neiltyson]. (2020, Sept. 29). You can't use reason to convince anyone

out of an argument that they didn't use reason to get into. [Tweet] Twitter.

https://twitter.com/neiltyson/status/1311127369785192449

How to Cite a Book in APA

Here's how to cite a book in APA:   

Author, A. A. (Year). Book title. Publisher.

For a real book, this looks like:

Montgomery, L. M. (1919). Rainbow valley.

Frederick A. Stokes Company.

How to Cite a Chapter in a Book in APA

Author, A. A. (Year). Chapter title. In Editor Name (Ed.), Book Title (pp. page range).

With information from a real book, this looks like:

Ezell, M. J. M. (2014). The social author: Manuscript culture, writers, and readers. In

Michelle Levy and Tom Mole (Eds.), The Broadview Reader in Book History (pp. 375–

394). Broadview Press.

Knowing how to cite a book and how to cite a chapter in a book correctly will take you a long way in creating an effective reference list.

How to Cite a Paraphrase in APA

You can cite a paraphrase in APA the same way as you would cite a direct quotation, including the author's name and year of publication. 

In APA, you may also choose to pinpoint the page from which the information is taken.

Referencing is an essential part of academic integrity. Learning how to write an academic essay with references and how to use in-text citations shows readers that you did your research and helps them locate your sources.

Learning how to cite a website, how to cite a book, and how to cite a paraphrase can also help you avoid plagiarism —an academic offense with serious consequences for your education or professional reputation.

Scribendi can help format your citations or review your whole paper with our Academic Editing services .

Take Your Essay from Good to Great

Hire an expert academic editor , or get a free sample, about the author.

Scribendi's in-house editors work with writers from all over the globe to perfect their writing. They know that no piece of writing is complete without a professional edit, and they love to see a good piece of writing transformed into a great one. Scribendi's in-house editors are unrivaled in both experience and education, having collectively edited millions of words and obtained numerous degrees. They love consuming caffeinated beverages, reading books of various genres, and relaxing in quiet, dimly lit spaces.

Have You Read?

"The Complete Beginner's Guide to Academic Writing"

Related Posts

APA Style and APA Formatting

How to Research a Term Paper

MLA Formatting and MLA Style: An Introduction

Upload your file(s) so we can calculate your word count, or enter your word count manually.

We will also recommend a service based on the file(s) you upload.

English is not my first language. I need English editing and proofreading so that I sound like a native speaker.

I need to have my journal article, dissertation, or term paper edited and proofread, or I need help with an admissions essay or proposal.

I have a novel, manuscript, play, or ebook. I need editing, copy editing, proofreading, a critique of my work, or a query package.

I need editing and proofreading for my white papers, reports, manuals, press releases, marketing materials, and other business documents.

I need to have my essay, project, assignment, or term paper edited and proofread.

I want to sound professional and to get hired. I have a resume, letter, email, or personal document that I need to have edited and proofread.

 Prices include your personal % discount.

 Prices include % sales tax ( ).

• Walden University
• Faculty Portal

Reference List: Common Reference List Examples

Article (with doi).

Alvarez, E., & Tippins, S. (2019). Socialization agents that Puerto Rican college students use to make financial decisions. Journal of Social Change , 11 (1), 75–85. https://doi.org/10.5590/JOSC.2019.11.1.07

Laplante, J. P., & Nolin, C. (2014). Consultas and socially responsible investing in Guatemala: A case study examining Maya perspectives on the Indigenous right to free, prior, and informed consent. Society & Natural Resources , 27 , 231–248. https://doi.org/10.1080/08941920.2013.861554

Use the DOI number for the source whenever one is available. DOI stands for "digital object identifier," a number specific to the article that can help others locate the source. In APA 7, format the DOI as a web address. Active hyperlinks for DOIs and URLs should be used for documents meant for screen reading. Present these hyperlinks in blue and underlined text (the default formatting in Microsoft Word), although plain black text is also acceptable. Be consistent in your formatting choice for DOIs and URLs throughout your reference list. Also see our Quick Answer FAQ, "Can I use the DOI format provided by library databases?"

Jerrentrup, A., Mueller, T., Glowalla, U., Herder, M., Henrichs, N., Neubauer, A., & Schaefer, J. R. (2018). Teaching medicine with the help of “Dr. House.” PLoS ONE , 13 (3), Article e0193972. https://doi.org/10.1371/journal.pone.0193972

For journal articles that are assigned article numbers rather than page ranges, include the article number in place of the page range.

For more on citing electronic resources, see  Electronic Sources References .

Article (Without DOI)

Found in a common academic research database or in print.

Casler , T. (2020). Improving the graduate nursing experience through support on a social media platform. MEDSURG Nursing , 29 (2), 83–87.

If an article does not have a DOI and you retrieved it from a common academic research database through the university library, there is no need to include any additional electronic retrieval information. The reference list entry looks like the entry for a print copy of the article. (This format differs from APA 6 guidelines that recommended including the URL of a journal's homepage when the DOI was not available.) Note that APA 7 has additional guidance on reference list entries for articles found only in specific databases or archives such as Cochrane Database of Systematic Reviews, UpToDate, ProQuest Dissertations and Theses Global, and university archives. See APA 7, Section 9.30 for more information.

Found on an Open Access Website

Eaton, T. V., & Akers, M. D. (2007). Whistleblowing and good governance. CPA Journal , 77 (6), 66–71. http://archives.cpajournal.com/2007/607/essentials/p58.htm

Provide the direct web address/URL to a journal article found on the open web, often on an open access journal's website. In APA 7, active hyperlinks for DOIs and URLs should be used for documents meant for screen reading. Present these hyperlinks in blue and underlined text (the default formatting in Microsoft Word), although plain black text is also acceptable. Be consistent in your formatting choice for DOIs and URLs throughout your reference list.

Weinstein, J. A. (2010).  Social change  (3rd ed.). Rowman & Littlefield.

If the book has an edition number, include it in parentheses after the title of the book. If the book does not list any edition information, do not include an edition number. The edition number is not italicized.

American Nurses Association. (2015). Nursing: Scope and standards of practice (3rd ed.).

If the author and publisher are the same, only include the author in its regular place and omit the publisher.

Lencioni, P. (2012). The advantage: Why organizational health trumps everything else in business . Jossey-Bass. https://amzn.to/343XPSJ

As a change from APA 6 to APA 7, it is no longer necessary to include the ebook format in the title. However, if you listened to an audiobook and the content differs from the text version (e.g., abridged content) or your discussion highlights elements of the audiobook (e.g., narrator's performance), then note that it is an audiobook in the title element in brackets. For ebooks and online audiobooks, also include the DOI number (if available) or nondatabase URL but leave out the electronic retrieval element if the ebook was found in a common academic research database, as with journal articles. APA 7 allows for the shortening of long DOIs and URLs, as shown in this example. See APA 7, Section 9.36 for more information.

Chapter in an Edited Book

Poe, M. (2017). Reframing race in teaching writing across the curriculum. In F. Condon & V. A. Young (Eds.), Performing antiracist pedagogy in rhetoric, writing, and communication (pp. 87–105). University Press of Colorado.

Include the page numbers of the chapter in parentheses after the book title.

Christensen, L. (2001). For my people: Celebrating community through poetry. In B. Bigelow, B. Harvey, S. Karp, & L. Miller (Eds.), Rethinking our classrooms: Teaching for equity and justice (Vol. 2, pp. 16–17). Rethinking Schools.

Also include the volume number or edition number in the parenthetical information after the book title when relevant.

Freud, S. (1961). The ego and the id. In J. Strachey (Ed.),  The standard edition of the complete psychological works of Sigmund Freud  (Vol. 19, pp. 3-66). Hogarth Press. (Original work published 1923)

When a text has been republished as part of an anthology collection, after the author’s name include the date of the version that was read. At the end of the entry, place the date of the original publication inside parenthesis along with the note “original work published.” For in-text citations of republished work, use both dates in the parenthetical citation, original date first with a slash separating the years, as in this example: Freud (1923/1961). For more information on reprinted or republished works, see APA 7, Sections 9.40-9.41.

Classroom Resources

Citing classroom resources.

If you need to cite content found in your online classroom, use the author (if there is one listed), the year of publication (if available), the title of the document, and the main URL of Walden classrooms. For example, you are citing study notes titled "Health Effects of Exposure to Forest Fires," but you do not know the author's name, your reference entry will look like this:

Health effects of exposure to forest fires [Lecture notes]. (2005). Walden University Canvas. https://waldenu.instructure.com

If you do know the author of the document, your reference will look like this:

Smith, A. (2005). Health effects of exposure to forest fires [PowerPoint slides]. Walden University Canvas. https://waldenu.instructure.com  

A few notes on citing course materials:

• [Lecture notes]
• [Course handout]
• [Study notes]
• It can be difficult to determine authorship of classroom documents. If an author is listed on the document, use that. If the resource is clearly a product of Walden (such as the course-based videos), use Walden University as the author. If you are unsure or if no author is indicated, place the title in the author spot, as above.
• If you cannot determine a date of publication, you can use n.d. (for "no date") in place of the year.

Note:  The web location for Walden course materials is not directly retrievable without a password, and therefore, following APA guidelines, use the main URL for the class sites: https://class.waldenu.edu.

Citing Tempo Classroom Resources

Clear author: 

Smith, A. (2005). Health effects of exposure to forest fires [PowerPoint slides]. Walden University Brightspace. https://mytempo.waldenu.edu

Unclear author:

Health effects of exposure to forest fires [Lecture notes]. (2005). Walden University Brightspace. https://mytempo.waldenu.edu

Conference Sessions and Presentations

Feinman, Y. (2018, July 27). Alternative to proctoring in introductory statistics community college courses [Poster presentation]. Walden University Research Symposium, Minneapolis, MN, United States. https://scholarworks.waldenu.edu/symposium2018/23/

Torgerson, K., Parrill, J., & Haas, A. (2019, April 5-9). Tutoring strategies for online students [Conference session]. The Higher Learning Commission Annual Conference, Chicago, IL, United States. http://onlinewritingcenters.org/scholarship/torgerson-parrill-haas-2019/

Dictionary Entry

Merriam-Webster. (n.d.). Leadership. In Merriam-Webster.com dictionary . Retrieved May 28, 2020, from https://www.merriam-webster.com/dictionary/leadership

When constructing a reference for an entry in a dictionary or other reference work that has no byline (i.e., no named individual authors), use the name of the group—the institution, company, or organization—as author (e.g., Merriam Webster, American Psychological Association, etc.). The name of the entry goes in the title position, followed by "In" and the italicized name of the reference work (e.g., Merriam-Webster.com dictionary , APA dictionary of psychology ). In this instance, APA 7 recommends including a retrieval date as well for this online source since the contents of the page change over time. End the reference entry with the specific URL for the defined word.

Discussion Board Post

Osborne, C. S. (2010, June 29). Re: Environmental responsibility [Discussion post]. Walden University Canvas.  https://waldenu.instructure.com  

Dissertations or Theses

Retrieved From a Database

Nalumango, K. (2019). Perceptions about the asylum-seeking process in the United States after 9/11 (Publication No. 13879844) [Doctoral dissertation, Walden University]. ProQuest Dissertations and Theses.

Retrieved From an Institutional or Personal Website

Evener. J. (2018). Organizational learning in libraries at for-profit colleges and universities [Doctoral dissertation, Walden University]. ScholarWorks. https://scholarworks.waldenu.edu/cgi/viewcontent.cgi?article=6606&context=dissertations

Unpublished Dissertation or Thesis

Kirwan, J. G. (2005). An experimental study of the effects of small-group, face-to-face facilitated dialogues on the development of self-actualization levels: A movement towards fully functional persons [Unpublished doctoral dissertation]. Saybrook Graduate School and Research Center.

For further examples and information, see APA 7, Section 10.6.

Legal Material

For legal references, APA follows the recommendations of The Bluebook: A Uniform System of Citation , so if you have any questions beyond the examples provided in APA, seek out that resource as well.

Court Decisions

Reference format:

Name v. Name, Volume Reporter Page (Court Date). URL

Sample reference entry:

Brown v. Board of Education, 347 U.S. 483 (1954). https://www.oyez.org/cases/1940-1955/347us483

Sample citation:

In Brown v. Board of Education (1954), the Supreme Court ruled racial segregation in schools unconstitutional.

Note: Italicize the case name when it appears in the text of your paper.

Name of Act, Title Source § Section Number (Year). URL

Sample reference entry for a federal statute:

Individuals With Disabilities Education Act, 20 U.S.C. § 1400 et seq. (2004). https://www.congress.gov/108/plaws/publ446/PLAW-108publ446.pdf

Sample reference entry for a state statute:

Minnesota Nurse Practice Act, Minn. Stat. §§ 148.171 et seq. (2019). https://www.revisor.mn.gov/statutes/cite/148.171

Sample citation: Minnesota nurses must maintain current registration in order to practice (Minnesota Nurse Practice Act, 2010).

Note: The § symbol stands for "section." Use §§ for sections (plural). To find this symbol in Microsoft Word, go to "Insert" and click on Symbol." Look in the Latin 1-Supplement subset. Note: U.S.C. stands for "United States Code." Note: The Latin abbreviation " et seq. " means "and what follows" and is used when the act includes the cited section and ones that follow. Note: List the chapter first followed by the section or range of sections.

Unenacted Bills and Resolutions

(Those that did not pass and become law)

Title [if there is one], bill or resolution number, xxx Cong. (year). URL

Sample reference entry for Senate bill:

Anti-Phishing Act, S. 472, 109th Cong. (2005). https://www.congress.gov/bill/109th-congress/senate-bill/472

Sample reference entry for House of Representatives resolution:

Anti-Phishing Act, H.R. 1099, 109th Cong. (2005). https://www.congress.gov/bill/109th-congress/house-bill/1099

The Anti-Phishing Act (2005) proposed up to 5 years prison time for people running Internet scams.

These are the three legal areas you may be most apt to cite in your scholarly work. For more examples and explanation, see APA 7, Chapter 11.

Magazine Article

Clay, R. (2008, June). Science vs. ideology: Psychologists fight back about the misuse of research. Monitor on Psychology , 39 (6). https://www.apa.org/monitor/2008/06/ideology

Note that for citations, include only the year: Clay (2008). For magazine articles retrieved from a common academic research database, leave out the URL. For magazine articles from an online news website that is not an online version of a print magazine, follow the format for a webpage reference list entry.

Newspaper Article (Retrieved Online)

Baker, A. (2014, May 7). Connecticut students show gains in national tests. New York Times . http://www.nytimes.com/2014/05/08/nyregion/national-assessment-of-educational-progress-results-in-Connecticut-and-New-Jersey.html

Include the full date in the format Year, Month Day. Do not include a retrieval date for periodical sources found on websites. Note that for citations, include only the year: Baker (2014). For newspaper articles retrieved from a common academic research database, leave out the URL. For newspaper articles from an online news website that is not an online version of a print newspaper, follow the format for a webpage reference list entry.

OASIS Resources

Oasis webpage.

OASIS. (n.d.). Common reference list examples . Walden University. https://academicguides.waldenu.edu/writingcenter/apa/references/examples

For all OASIS content, list OASIS as the author. Because OASIS webpages do not include publication dates, use “n.d.” for the year.

Interactive Guide

OASIS. (n.d.). Embrace iterative research and writing [Interactive guide]. Walden University. https://academics.waldenu.edu/oasis/iterative-research-writing-web

For OASIS multimedia resources, such as interactive guides, include a description of the resource in brackets after the title.

Online Video/Webcast

Walden University. (2013).  An overview of learning  [Video]. Walden University Canvas.  https://waldenu.instructure.com  

Use this format for online videos such as Walden videos in classrooms. Most of our classroom videos are produced by Walden University, which will be listed as the author in your reference and citation. Note: Some examples of audiovisual materials in the APA manual show the word “Producer” in parentheses after the producer/author area. In consultation with the editors of the APA manual, we have determined that parenthetical is not necessary for the videos in our courses. The manual itself is unclear on the matter, however, so either approach should be accepted. Note that the speaker in the video does not appear in the reference list entry, but you may want to mention that person in your text. For instance, if you are viewing a video where Tobias Ball is the speaker, you might write the following: Tobias Ball stated that APA guidelines ensure a consistent presentation of information in student papers (Walden University, 2013). For more information on citing the speaker in a video, see our page on Common Citation Errors .

Taylor, R. [taylorphd07]. (2014, February 27). Scales of measurement [Video]. YouTube. https://www.youtube.com/watch?v=PDsMUlexaMY

OASIS. (2020, April 15). One-way ANCOVA: Introduction [Video]. YouTube. https://youtu.be/_XnNDQ5CNW8

For videos from streaming sites, use the person or organization who uploaded the video in the author space to ensure retrievability, whether or not that person is the speaker in the video. A username can be provided in square brackets. As a change from APA 6 to APA 7, include the publisher after the title, and do not use "Retrieved from" before the URL. See APA 7, Section 10.12 for more information and examples.

See also reference list entry formats for TED Talks .

Technical and Research Reports

Edwards, C. (2015). Lighting levels for isolated intersections: Leading to safety improvements (Report No. MnDOT 2015-05). Center for Transportation Studies. http://www.cts.umn.edu/Publications/ResearchReports/reportdetail.html?id=2402

Technical and research reports by governmental agencies and other research institutions usually follow a different publication process than scholarly, peer-reviewed journals. However, they present original research and are often useful for research papers. Sometimes, researchers refer to these types of reports as gray literature , and white papers are a type of this literature. See APA 7, Section 10.4 for more information.

Reference list entires for TED Talks follow the usual guidelines for multimedia content found online. There are two common places to find TED talks online, with slightly different reference list entry formats for each.

TED Talk on the TED website

If you find the TED Talk on the TED website, follow the format for an online video on an organizational website:

Owusu-Kesse, K. (2020, June). 5 needs that any COVID-19 response should meet [Video]. TED Conferences. https://www.ted.com/talks/kwame_owusu_kesse_5_needs_that_any_covid_19_response_should_meet

The speaker is the author in the reference list entry if the video is posted on the TED website. For citations, use the speaker's surname.

TED Talk on YouTube

If you find the TED Talk on YouTube or another streaming video website, follow the usual format for streaming video sites:

TED. (2021, February 5). The shadow pandemic of domestic violence during COVID-19 | Kemi DaSilvalbru [Video]. YouTube. https://www.youtube.com/watch?v=PGdID_ICFII

TED is the author in the reference list entry if the video is posted on YouTube since it is the channel on which the video is posted. For citations, use TED as the author.

Walden University Course Catalog

To include the Walden course catalog in your reference list, use this format:

Walden University. (2020). 2019-2020 Walden University catalog . https://catalog.waldenu.edu/index.php

If you cite from a specific portion of the catalog in your paper, indicate the appropriate section and paragraph number in your text:

...which reflects the commitment to social change expressed in Walden University's mission statement (Walden University, 2020, Vision, Mission, and Goals section, para. 2).

And in the reference list:

Walden University. (2020). Vision, mission, and goals. In 2019-2020 Walden University catalog. https://catalog.waldenu.edu/content.php?catoid=172&navoid=59420&hl=vision&returnto=search

Vartan, S. (2018, January 30). Why vacations matter for your health . CNN. https://www.cnn.com/travel/article/why-vacations-matter/index.html

For webpages on the open web, include the author, date, webpage title, organization/site name, and URL. (There is a slight variation for online versions of print newspapers or magazines. For those sources, follow the models in the previous sections of this page.)

American Federation of Teachers. (n.d.). Community schools . http://www.aft.org/issues/schoolreform/commschools/index.cfm

If there is no specified author, then use the organization’s name as the author. In such a case, there is no need to repeat the organization's name after the title.

In APA 7, active hyperlinks for DOIs and URLs should be used for documents meant for screen reading. Present these hyperlinks in blue and underlined text (the default formatting in Microsoft Word), although plain black text is also acceptable. Be consistent in your formatting choice for DOIs and URLs throughout your reference list.

Related Resources

Knowledge Check: Common Reference List Examples

Didn't find what you need? Email us at [email protected] .

• Previous Page: Reference List: Overview
• Next Page: Common Military Reference List Examples
• Office of Student Disability Services

Walden Resources

Departments.

• Academic Residencies
• Academic Skills
• Career Planning and Development
• Customer Care Team
• Field Experience
• Military Services
• Student Success Advising
• Writing Skills

Centers and Offices

• Center for Social Change
• Office of Academic Support and Instructional Services
• Office of Degree Acceleration
• Office of Research and Doctoral Services
• Office of Student Affairs

Student Resources

• Doctoral Writing Assessment
• Form & Style Review
• Quick Answers
• ScholarWorks
• SKIL Courses and Workshops
• Walden Bookstore
• Walden Catalog & Student Handbook
• Student Safety/Title IX
• Legal & Consumer Information
• Website Terms and Conditions
• Cookie Policy
• Accessibility
• Accreditation
• State Authorization
• Net Price Calculator
• Contact Walden

Walden University is a member of Adtalem Global Education, Inc. www.adtalem.com Walden University is certified to operate by SCHEV © 2024 Walden University LLC. All rights reserved.

APA In-Text Citations and Sample Essay 7th Edition

This handout focuses on how to format in-text citations in APA.

Proper citation of sources is a two-part process . You must first cite each source in the body of your essay; these citations within the essay are called in-text citations . You MUST cite all quoted, paraphrased, or summarized words, ideas, and facts from sources. Without in-text citations, you are technically in danger of plagiarism, even if you have listed your sources at the end of the essay.

In-text citations point the reader to the sources’ information on the references page. The in-text citation typically includes the author's last name and the year of publication. If you use a direct quote, the page number is also provided.

More information can be found on p. 253 of the 7th edition of the Publication Manual of the American Psychological Association.

Citation Rules

Direct quotation with the author named in the text.

Heinze and Lu (2017) stated, “The NFL shifted its responses to institutional change around concussions significantly as the field itself evolved” (p. 509).

Note: The year of publication is listed in parenthesis after the names of the authors, and the page number is listed in parenthesis at the end of the quote.

Direct Quotation without the Author Named in the Text

As the NFL developed as an organization, it “shifted its responses to institutional change around concussions significantly” (Heinze & Lu, 2017, p. 509).

Note: At the end of the quote, the names of the authors, year of publication, and page number are listed in parenthesis.

Paraphrase with 1-2 Authors

As the NFL developed as an organization, its reactions toward concussions also transformed (Heinze & Lu, 2017).

Note: For paraphrases, page numbers are encouraged but not required.

Paraphrase with 3 or More Authors

To work toward solving the issue of violence in prisons begins with determining aspects that might connect with prisoners' violent conduct (Thomson et al., 2019).

Direct Quotation without an Author

The findings were astonishing "in a recent study of parent and adult child relationships" ("Parents and Their Children," 2007, p. 2).

Note: Since the author of the text is not stated, a shortened version of the title is used instead.

Secondary Sources

When using secondary sources, use the phrase "as cited in" and cite the secondary source on the References page.

In 1936, Keynes said, “governments should run deficits when the economy is slow to avoid unemployment” (as cited in Richardson, 2008, p. 257).

Long (Block) Quotations

When using direct quotations of 40 or more words, indent five spaces from the left margin without using quotation marks. The final period should come before the parenthetical citation.

At Meramec, an English department policy states:

To honor and protect their own work and that of others, all students must give credit to proprietary sources that are used for course work. It is assumed that any information that is not documented is either common knowledge in that field or the original work of that student. (St. Louis Community College, 2001, p. 1)

Website Citations

If citing a specific web document without a page number, include the name of the author, date, title of the section, and paragraph number in parentheses:

In America, “Two out of five deaths among U.S. teens are the result of a motor vehicle crash” (National Center for Injury Prevention and Control, 2004, Overview section, para. 1).

Here is a print-friendly version of this content.

Learn more about the APA References page by reviewing this handout .

For information on STLCC's academic integrity policy, check out this webpage .

For additional information on APA, check out STLCC's LibGuide on APA .

Sample Essay

A sample APA essay is available at this link .

Scribbr Referencing Generator

Accurate Harvard, APA, MLA, and Chicago references, verified by experts, trusted by millions.

Reference sources in seconds with Autocite

Look up your source by its title, URL, ISBN, or DOI, and let Scribbr do the rest! The reference generator will automatically find all the necessary information to generate a perfect reference, including the author(s), publication date, and publisher.

Perfectly formatted references every time

Inaccurate references can cost you points on your assignments, so our seasoned referencing experts have invested countless hours in perfecting Scribbr’s reference generator algorithms. We’re proud to be recommended by teachers and universities across the UK.

Enjoy a referencing generator without flashy ads

Staying focused is already difficult enough, so unlike other reference generators, Scribbr won’t slow you down with flashing banner ads and video pop-ups. That’s a promise!

Citation Generator features you'll love

Look up your source by its title, URL, ISBN, or DOI, and let Scribbr find and fill in all the relevant information automatically.

Harvard, APA, MLA, Chicago

Generate flawless references according to the official Harvard , APA , MLA, or Chicago style rules. More referencing styles will be available soon!

Export to Word

When your reference list is complete, export it to Word. We’ll apply the official formatting guidelines automatically.

Lists and folders

Create separate reference lists for each of your assignments to stay organized. You can also group related lists into folders.

Export to Bib(La)TeX

Are you using a LaTex editor like Overleaf? If so, you can easily export your references in Bib(La)TeX format with a single click.

Custom fonts

Change the typeface used for your reference list to match the rest of your document. Options include Times New Roman, Arial, and Calibri.

Industry-standard technology

Scribbr’s Referencing Generator is built using the same citation software (CSL) as Mendeley and Zotero, but with an added layer for improved accuracy.

Annotations

Describe or evaluate your sources in annotations, and Scribbr will generate a perfectly formatted annotated bibliography.

Referencing guides

Scribbr’s popular guides and videos will help you understand everything related to finding, evaluating, and referencing sources.

Secure backup

Your work is saved automatically after every change and stored securely in your Scribbr account.

• Introduction
• Finding sources

Evaluating sources

• Integrating sources

Referencing sources

Tools and resources, a quick guide to working with sources.

Working with sources is an important skill that you’ll need throughout your academic career.

It includes knowing how to find relevant sources, assessing their authority and credibility, and understanding how to integrate sources into your work with proper referencing.

This quick guide will help you get started!

Finding relevant sources

Sources commonly used in academic writing include academic journals, scholarly books, websites, newspapers, and encyclopedias. There are three main places to look for such sources:

• Research databases: Databases can be general or subject-specific. To get started, check out this list of databases by academic discipline . Another good starting point is Google Scholar .
• Your institution’s library: Use your library’s database to narrow down your search using keywords to find relevant articles, books, and newspapers matching your topic.
• Other online resources: Consult popular online sources like websites, blogs, or Wikipedia to find background information. Be sure to carefully evaluate the credibility of those online sources.

When using academic databases or search engines, you can use Boolean operators to refine your results.

Generate Harvard, APA, MLA, and Chicago style references in seconds

Get started

In academic writing, your sources should be credible, up to date, and relevant to your research topic. Useful approaches to evaluating sources include the CRAAP test and lateral reading.

CRAAP is an abbreviation that reminds you of a set of questions to ask yourself when evaluating information.

• Currency: Does the source reflect recent research?
• Relevance: Is the source related to your research topic?
• Authority: Is it a respected publication? Is the author an expert in their field?
• Accuracy: Does the source support its arguments and conclusions with evidence?
• Purpose: What is the author’s intention?

Lateral reading

Lateral reading means comparing your source to other sources. This allows you to:

• Verify evidence
• Contextualize information
• Find potential weaknesses

If a source is using methods or drawing conclusions that are incompatible with other research in its field, it may not be reliable.

Integrating sources into your work

Once you have found information that you want to include in your paper, signal phrases can help you to introduce it. Here are a few examples:

Following the signal phrase, you can choose to quote, paraphrase or summarize the source.

• Quoting : This means including the exact words of another source in your paper. The quoted text must be enclosed in quotation marks or (for longer quotes) presented as a block quote . Quote a source when the meaning is difficult to convey in different words or when you want to analyze the language itself.
• Paraphrasing: This means putting another person’s ideas into your own words. It allows you to integrate sources more smoothly into your text, maintaining a consistent voice. It also shows that you have understood the meaning of the source.
• Summarizing : This means giving an overview of the essential points of a source. Summaries should be much shorter than the original text. You should describe the key points in your own words and not quote from the original text.

Whenever you quote, paraphrase, or summarize a source, you must include a citation crediting the original author.

Referencing your sources is important because it:

• Allows you to avoid plagiarism
• Establishes the credentials of your sources
• Backs up your arguments with evidence
• Allows your reader to verify the legitimacy of your conclusions

The most common citation styles in the UK are APA, MLA, Harvard, Vancouver, MHRA, and Oscola. Each citation style has specific rules for formatting citations.

Scribbr’s free Reference Generator can generate perfect references and in-text citations in both APA and MLA styles. More citation styles will be available soon!

Scribbr and partners offer tons of tools and resources to make working with sources easier and faster. Take a look at our top picks:

• Reference Generator: Automatically generate Harvard and APA references .
• Plagiarism Checker : Detect plagiarism in your paper using the most accurate Turnitin-powered plagiarism software available to students.
• Proofreading services : Have a human editor improve your writing.
• Knowledge Base : Explore hundreds of articles, bite-sized videos, time-saving templates, and handy checklists that guide you through the process of research, writing, and citation.

• Bipolar Disorder
• Therapy Center
• When To See a Therapist
• Types of Therapy
• Best Online Therapy
• Best Couples Therapy
• Best Family Therapy
• Managing Stress
• Sleep and Dreaming
• Understanding Emotions
• Self-Improvement
• Healthy Relationships
• Student Resources
• Personality Types
• Guided Meditations
• Verywell Mind Insights
• 2024 Verywell Mind 25
• Mental Health in the Classroom
• Editorial Process
• Meet Our Review Board
• Crisis Support

How to Write an Essay in APA Format

Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

Emily is a board-certified science editor who has worked with top digital publishing brands like Voices for Biodiversity, Study.com, GoodTherapy, Vox, and Verywell.

What Is APA Format?

Apa essay format basics.

• Steps to Follow

Frequently Asked Questions

If your instructor has asked you to write an APA format essay, it might at first seem like a daunting task, especially if you are accustomed to using another style such as MLA or Chicago. But you can master the rules of APA essay format, too.

An essay is one type of paper that can be written in APA format; others include lab reports, experimental reports, and case studies. Before you begin, familiarize yourself with some of the basic guidelines for writing a paper in APA format. Of course, it will also be important to follow any other formatting instructions that are part of your assignment.

How do you write an essay in APA format? The basic elements you need to include are:

• A title page
• An abstract
• An introduction, main body, and conclusion
• A reference section
• Proper APA formatting with regard to margins, layout, spacing, titles, and indentations

This article discusses how to write an essay in APA format, including the basic steps you should follow and tips for how to get started.

Whether you’re taking an introductory or graduate-level psychology class, chances are strong that you will have to write at least one paper during the course of the semester. In almost every case, you will need to write your paper in APA format, the official publication style of the American Psychological Association . It is also used for academic journals.

Such rules are generally the same whether you are writing a high school essay, college essay, or professional essay for publication.

APA format is used in a range of disciplines including psychology , education, and other social sciences. The format dictates presentation elements of your paper including spacing, margins, and how the content is structured.

Most instructors and publication editors have strict guidelines when it comes to how your format your writing. Not only does adhering to APA format allow readers to know what to expect from your paper, but it also means that your work will not lose critical points over minor formatting errors. 

While the formatting requirements for your paper might vary depending on your instructor's directions, writing APA essay format means you will most likely need to include a title page, abstract, introduction, body, conclusion, and reference sections.

Your APA format essay should have a title page . This title page should include the title of your paper, your name, and your school affiliation. In some instances, your teacher might require additional information such as the course title, instructor name, and the date.

• The title of your paper should be concise and clearly describe what your paper is about.
• Your title can extend to two lines, but it should be no longer than 12 words.

An abstract is a brief summary of your paper that immediately follows the title page. It is not required for student papers, according to APA style. However, your instructor may request one.

If you include an abstract , it should be no more than 100 to 200 words, although this may vary depending upon the instructor requirements.

Your essay should also include a reference list with all of the sources that were cited in your essay,

• The reference section is located at the end of your paper.
• References should be listed alphabetically by the last name of the author.
• References should be double-spaced.
• Any source that is cited in your paper should be included in your reference section.

When writing in APA essay format, the text will include the actual essay itself: The introduction, body, and conclusion.

• There should be uniform margins of at least one inch at the top, bottom, left, and right sides of your essay.
• The text should be in Times New Roman size 12 font or another serif typeface that is easily readable.
• Your paper should be double-spaced.
• Every page should include a page number in the top right corner.
• The first word of each paragraph in your paper should be indented one-half inch.

For professional papers (usually not student papers), every page of the essay also includes a running head at the top left. The running head is a shortened form of the title, often the first few words, and should be no more than 50 characters (including spaces).

Steps to a Successful APA Format Essay

In addition to ensuring that you cite your sources properly and present information according to the rules of APA style, there are a number of things you can do to make the writing process a little bit easier.

Choose a Topic

Start by choosing a good topic to write about. Ideally, you want to select a subject that is specific enough to let you fully research and explore the topic, but not so specific that you have a hard time finding sources of information.

If you choose something too specific, you may find yourself with not enough to write about. If you choose something too general, you might find yourself overwhelmed with information.

Research Your Topic

Start doing research as early as possible. Begin by looking at some basic books and articles on your topic to help develop it further. What is the question you are going to answer with your essay? What approach will you take to the topic?

Once you are more familiar with the subject, create a preliminary source list of potential books, articles, essays, and studies that you may end up using in your essay.

Remember, any source used in your essay must be included in your reference section. Conversely, any source listed in your references must be cited somewhere in the body of your paper.

Write Your Rough Draft

With research in hand, you are ready to begin. Some people like to create an outline to organize their argument prior to drafting. You may want to start with a very rough outline, and then add details.

Once you have a detailed outline, the next step is to translate it from notes to complete sentences and paragraphs. Remember, this is a first draft. It doesn't have to be perfect.

As you write your paper in APA essay format, be sure to keep careful track of the sources that you cite.

How do you start an APA paper? Your paper should begin with an introduction that includes a thesis statement that presents your main ideas, points, or arguments. Your introduction should start on the third page of your paper (after the title page and abstract). The title of your paper should be centered, bolded, and typed in title case at the top of the page.

Review and Revise

After you have prepared a rough draft of your essay, it's time to revise, review, and prepare your final draft. In addition to making sure that your writing is cohesive and supported by your sources, you should also check carefully for typos, grammar errors, and possible formatting mistakes.

When citing information or quotations taken from an interview, APA format requires that you cite the source, how the information was collected, and the date of the interview. They should not be included in the reference section, however, because they are not something that can be located by a reader in any published source or searchable database.

Instead, the information should be cited parenthetically in the main body of the text. For example: “There was an increase in the number of college students who screened positive for depression/anxiety” (R. Heathfield, personal communication, May 9, 2021).

If the essay is in a chapter of a book, edited collection, or anthology, APA format states that you should cite the last name, first name, title of essay, title of collection, publisher, year, and page range. For example: Smith, John, "The Light House," A Book of Poems , editing by Peter Roberts, Allworth Press, 2005, pp. 20-25.

According to APA format, a two-part essay is formatted the same as an essay, however, you'll need to create two title pages.

If you're including a short direct quote in your APA-format essay, you will need to cite the author, year of publication, and page number (p.) or page number span (pp.). Quotations longer than 40 words should omit the quotation marks and be put in the text using block quotation formatting, on its own line and indented 1/2 inch from the left margin.

The cover page or "title page" in APA essay format should always include the title of your paper, your name, and school affiliation as well as the course title, instructor name, and date, if requested by your teacher.

Nagda S.  How to write a scientific abstract.   J Indian Prosthodont Soc.  2013;13(3):382-383. doi:10.1007/s13191-013-0299-x

American Psychological Association.  Publication Manual of the American Psychological Association  (7th ed.). Washington DC: The American Psychological Association; 2019.

By Kendra Cherry, MSEd Kendra Cherry, MS, is a psychosocial rehabilitation specialist, psychology educator, and author of the "Everything Psychology Book."

• PRO Courses Guides New Tech Help Pro Expert Videos About wikiHow Pro Upgrade Sign In
• EDIT Edit this Article
• EXPLORE Tech Help Pro About Us Random Article Quizzes Request a New Article Community Dashboard This Or That Game Popular Categories Arts and Entertainment Artwork Books Movies Computers and Electronics Computers Phone Skills Technology Hacks Health Men's Health Mental Health Women's Health Relationships Dating Love Relationship Issues Hobbies and Crafts Crafts Drawing Games Education & Communication Communication Skills Personal Development Studying Personal Care and Style Fashion Hair Care Personal Hygiene Youth Personal Care School Stuff Dating All Categories Arts and Entertainment Finance and Business Home and Garden Relationship Quizzes Cars & Other Vehicles Food and Entertaining Personal Care and Style Sports and Fitness Computers and Electronics Health Pets and Animals Travel Education & Communication Hobbies and Crafts Philosophy and Religion Work World Family Life Holidays and Traditions Relationships Youth
• Browse Articles
• Learn Something New
• Quizzes Hot
• This Or That Game
• Train Your Brain
• Explore More
• Support wikiHow
• About wikiHow
• Log in / Sign up
• Education and Communications
• College University and Postgraduate
• Academic Writing

How to Reference Essays

Last Updated: January 8, 2024 Fact Checked

This article was co-authored by Alexander Peterman, MA . Alexander Peterman is a Private Tutor in Florida. He received his MA in Education from the University of Florida in 2017. There are 14 references cited in this article, which can be found at the bottom of the page. This article has been fact-checked, ensuring the accuracy of any cited facts and confirming the authority of its sources. This article has been viewed 371,837 times.

When you begin writing a research essay, you must take into account the format of your writing and reference pages. There are several reference styles that may be assigned to you, including MLA (Modern Language Association), APA (American Psychological Association), and Chicago. Each one has its own set of rules. There's no need to familiarize yourself with all 3 unless you have to, but you do need to learn at least one if you’re in any field involving academic writing. Here are summaries of each style to help you start your essay on the right track.

Referencing Essays Templates

• You will need a citation directly after every sentence (or group of sentences if you're citing the same source in multiple consecutive sentences) containing information you didn't think of yourself. These include: paraphrases, facts, statistics, quotes, and examples.
• An in-text citation using MLA will simply have the author last name (or title if no author) followed by the page number. No comma between author and page number. For example: (Richards 456) Richards is the author last name, and 456 is the page number.
• If you have an author name (or title, if no author) but no page number, simply use author last name (or title).

• The easiest way to keep track of MLA citations while doing research is to copy and paste copyright information into a word processing document as you go, or to write it down in a notebook.
• Things to include for any source are author(s), date published, publisher, page number, volume and issue number, website, date accessed, anything that appears on the copyright page or indicates how to find it again. [2] X Trustworthy Source Purdue Online Writing Lab Trusted resource for writing and citation guidelines Go to source

• As an example, the format for a standard book citation using MLA style is as follows: Last name of author, First name. Title of Book. City published: Publisher Name, Year published. Source Medium.
• An MLA website citation looks like the following. If there's no author listed, begin citation with the name of the page: Last name, first name. "Page Title." Website Title. Publisher. Date published. Source Medium. Date accessed.
• An MLA scholarly article citation looks like the following: Last name, First name. "Title of Article." Title of Journal . Volume.Issue (Year): page numbers. Source Medium.
• Write the title of the main work (book, magazine, journal, website, etc.) in italics, or underline if you’re writing references by hand.
• Chapter or article titles should be in quotation marks.

• If there is no author listed, as is common on websites, simply skip the author’s name and begin the entry with the title of the work.
• Alphabetize by the first letter that appears in the entry, whether it has an author name or not.

• The formatting should be in Times New Roman font, size 12, with “Works Cited” centered at the top of a new page.
• Each entry should have hanging indent, meaning all lines below the first line are indented by half an inch.
• Make sure there is a period after each section of the citations. A period should always end the citation.

• Place a parenthetical citation at the end of every sentence (or group of sentences if you're using the same source for multiple consecutive sentences) containing information you didn't know before doing research.
• An in-text citation using APA will simply have the author last name (or title if no author) followed by the year it was published. No comma between name and year. For example: (Richards 2005) Richards is the author last name, and 2005 is the year.
• If you have an author name (or title if no author) but no page number, simply use author last name (or title). This is common when citing websites.
• APA document formatting is very important. APA papers are divided up into 4 sections: the title page, the abstract, the main body, and the references page. The citations of a research paper using APA appear in the References section, the last portion of an APA document. [7] X Research source

• To form APA reference page citations, you will need such information as author name(s), date published, website URL, date you accessed the website, title of work, and so on. [8] X Trustworthy Source Purdue Online Writing Lab Trusted resource for writing and citation guidelines Go to source

• For example, the format for an APA reference of a scholarly journal article is as follows: Author last name, First initial. (Year published). Article or chapter title. Journal or book title, Issue number , page number range. [10] X Trustworthy Source Purdue Online Writing Lab Trusted resource for writing and citation guidelines Go to source
• The format for an APA book reference looks like: Last name, First name. (Year.) Title of Book: Capital letter also for subtitle . Location: Publisher.
• The format for an APA website reference looks like: Author, A.A. First name, & Author, B.B. (Date published.) Title of article. In Title of webpage or larger document or book (chapter or section number). Retrieved from URL address

• Capitalize the author's last name and first initial, followed by a period.
• Only capitalize the first word of a journal article title, unless the title contains a proper noun (called sentence case). Titles of books should preserve the published capitalization.
• Capitalize the city of publication, and use correct state abbreviations for states. Also capitalize the name of the publisher and end the reference with a period.
• The title of larger works, whether a book, journal, website, or magazine, is in italics (or underlined if handwriting), as is the issue number that appears right after the title. Titles for shorter works like articles and chapters should not have any indicative punctuation in an APA entry. [12] X Trustworthy Source Purdue Online Writing Lab Trusted resource for writing and citation guidelines Go to source
• A period should end all citations.

Using Chicago Manual of Style

• For Notes and Bibliography, you will use a superscript at the instance of each quote in the text with a corresponding footnote at the end of the page. All footnotes are compiled into endnotes at the end of the work, on the bibliography page. [14] X Trustworthy Source Purdue Online Writing Lab Trusted resource for writing and citation guidelines Go to source
• For Author Date, you will use parenthetical in-text citations that include author last name and year published, using no punctuation between name and year. The full version of each parenthetical citation is listed alphabetically on the references page. For example: (Simon 2011) Simon is the author last name, and 2011 is the year.
• You will need a citation directly after every sentence (or group of sentences if you're using the same source for multiple consecutive sentences) containing information you didn't think of yourself. These include: paraphrases, facts, statistics, quotes, and examples.

• If using a book, write down all pertinent information found on the copyright page, including the name of the publisher and the city and year of publication.
• For other sources, look for this information near the title of the piece you’re looking at. Publication date is often at the bottom of webpages.

• Title your references page “Bibliography” centered at the top of the page. Leave 2 blank lines between this title and the first entry, and one blank line between entries.
• Notes and Bibliography style uses footnotes for page endings and endnotes for chapter endings. The bibliography page will be an alphabetized list of all sources in hanging indent.
• An example format for a book is as follows: Last name, First name. Book Title . City: Publisher, Year.
• An example format for a chapter in a print scholarly journal is as follows: Author last name, first name. "Title of Chapter or Article." Book or journal Title Issue Number (Year): Page number range. (For an online scholarly journal article, tack on the following at the end: Date accessed. URL address.)
• When there is no known author, the entry should begin with the title of the document, whether it's a webpage, chapter, article, and so on.
• When there are multiple authors, the first listed author appears last name, first name, so that the citation is alphabetized by this author's last name. Subsequent authors are listed by first name, like this: Alcott, Louisa May, Charles Dickens, and Elizabeth Gaskell.
• Always end a citation with a period.

• When using Author Date style, title your references page “References” centered at the top of the page. Leave 2 blank lines between this title and the first entry, and 1 blank line between entries.
• Author Date style bibliographies should be organized alphabetically by last name (or by title if no author) in hanging indent.
• An example format for a book is as follows: Last name, first name. Year. Book Title . City Published: Publisher.
• An example format for a chapter in a print scholarly journal is as follows: Author last name, first name. Year. "Title of Chapter or Article." Book or journal title issue number: page numbers. (for an online scholarly journal article tack this onto the end: Date accessed. URL address.)
• An example format for a website is as follows: Name of Website. Year. "Page Title." Date last modified. Date accessed. URL address.

Expert Q&A

• You don't have to write each bibliography or reference entry on your own. You can download citation management software like Endnote [17] X Research source (purchase required on this one), Zotero [18] X Research source (it's free), or use websites like http://www.bibme.org/ and http://www.easybib.com/ . Select the name of your style manual before you begin creating citations. Copy and paste the citation into your bibliography or references list. Thanks Helpful 0 Not Helpful 0
• If you are assigned to write a paper or other written document in one of these styles, you need to purchase the style manual. It will contain nearly every instance not only of source citation, but paper formatting as well as grammar and punctuation that is unique to that style. Thanks Helpful 0 Not Helpful 0

• This article only lists how to cite research for each style manual. Each style has its own instructions for setting up the format of the essay, including heading, spacing, margins, font, and so on. Thanks Helpful 1 Not Helpful 0

You Might Also Like

• ↑ https://owl.purdue.edu/owl/research_and_citation/mla_style/mla_formatting_and_style_guide/mla_in_text_citations_the_basics.html
• ↑ https://owl.purdue.edu/owl/research_and_citation/mla_style/mla_formatting_and_style_guide/mla_formatting_and_style_guide.html
• ↑ https://owl.purdue.edu/owl/research_and_citation/mla_style/mla_formatting_and_style_guide/mla_works_cited_page_books.html
• ↑ https://owl.purdue.edu/owl/research_and_citation/apa_style/apa_formatting_and_style_guide/in_text_citations_author_authors.html
• ↑ https://owl.english.purdue.edu/owl/resource/560/01/
• ↑ https://owl.purdue.edu/owl/research_and_citation/apa_style/apa_formatting_and_style_guide/reference_list_electronic_sources.html
• ↑ https://libguides.jcu.edu.au/apa/reference-list
• ↑ https://owl.purdue.edu/owl/research_and_citation/apa_style/apa_formatting_and_style_guide/reference_list_author_authors.html
• ↑ https://owl.purdue.edu/owl/research_and_citation/apa_style/apa_formatting_and_style_guide/reference_list_basic_rules.html
• ↑ https://www.chicagomanualofstyle.org/tools_citationguide.html
• ↑ https://owl.purdue.edu/owl/research_and_citation/chicago_manual_17th_edition/cmos_formatting_and_style_guide/chicago_manual_of_style_17th_edition.html
• ↑ http://guides.nyu.edu/c.php?g=276562&p=1844734
• ↑ http://endnote.com
• ↑ https://www.zotero.org

About This Article

To reference an essay using MLA style, add a citation after any information you found through a source, like facts or quotes. When citing the reference, include the author’s name and the page number you pulled the information from in parenthesis, like “(Richards 456).” Once you’ve finished your essay, add a Words Cited page with all of the information you used to research your essay, like books or articles. To create a Works Cited page, list the sources in alphabetical order using the author’s last name, and include additional information, like year published and the medium. For more tips from our Writing reviewer, like how to reference an essay using APA style, read on! Did this summary help you? Yes No

• Send fan mail to authors

Reader Success Stories

Alicia Hammond

Sep 20, 2021

Did this article help you?

Oct 30, 2019

Brenda Nats

Jun 14, 2021

Featured Articles

Trending Articles

Watch Articles

• Terms of Use
• Privacy Policy
• Do Not Sell or Share My Info
• Not Selling Info

wikiHow Tech Help Pro:

Develop the tech skills you need for work and life

• Free Tools for Students
• Harvard Referencing Generator

Free Harvard Referencing Generator

Generate accurate Harvard reference lists quickly and for FREE, with MyBib!

🤔 What is a Harvard Referencing Generator?

A Harvard Referencing Generator is a tool that automatically generates formatted academic references in the Harvard style.

It takes in relevant details about a source -- usually critical information like author names, article titles, publish dates, and URLs -- and adds the correct punctuation and formatting required by the Harvard referencing style.

The generated references can be copied into a reference list or bibliography, and then collectively appended to the end of an academic assignment. This is the standard way to give credit to sources used in the main body of an assignment.

👩‍🎓 Who uses a Harvard Referencing Generator?

Harvard is the main referencing style at colleges and universities in the United Kingdom and Australia. It is also very popular in other English-speaking countries such as South Africa, Hong Kong, and New Zealand. University-level students in these countries are most likely to use a Harvard generator to aid them with their undergraduate assignments (and often post-graduate too).

🙌 Why should I use a Harvard Referencing Generator?

A Harvard Referencing Generator solves two problems:

• It provides a way to organise and keep track of the sources referenced in the content of an academic paper.
• It ensures that references are formatted correctly -- inline with the Harvard referencing style -- and it does so considerably faster than writing them out manually.

A well-formatted and broad bibliography can account for up to 20% of the total grade for an undergraduate-level project, and using a generator tool can contribute significantly towards earning them.

⚙️ How do I use MyBib's Harvard Referencing Generator?

Here's how to use our reference generator:

• If citing a book, website, journal, or video: enter the URL or title into the search bar at the top of the page and press the search button.
• Choose the most relevant results from the list of search results.
• Our generator will automatically locate the source details and format them in the correct Harvard format. You can make further changes if required.
• Then either copy the formatted reference directly into your reference list by clicking the 'copy' button, or save it to your MyBib account for later.

MyBib supports the following for Harvard style:

🍏 What other versions of Harvard referencing exist?

There isn't "one true way" to do Harvard referencing, and many universities have their own slightly different guidelines for the style. Our generator can adapt to handle the following list of different Harvard styles:

• Cite Them Right
• Manchester Metropolitan University (MMU)
• University of the West of England (UWE)

Daniel is a qualified librarian, former teacher, and citation expert. He has been contributing to MyBib since 2018.

Build my resume

• Build a better resume in minutes
• Resume examples
• 2,000+ examples that work in 2024
• Resume templates
• Free templates for all levels
• Cover letters
• Cover letter generator
• It's like magic, we promise
• Cover letter examples
• Free downloads in Word & Docs

How to List References on Resumes: Strategic Guide (With Examples)

• The Role of References in Job Applications
• Preparing Your Reference List
• Showcasing References
• Real World Examples

References in Resumes FAQs

References don’t usually live on a resume—they take up too much space and typically aren’t even needed at that stage of the application process.

Be that as it may, sometimes employers may go against the curve and ask for them straight away, and when that happens, you’ll need to rearrange your resume and make room.

Of course, the trickiest thing about references isn’t where to put them, but who to choose. In this article, we’ll show you plenty of resume templates and useful tips to help you handle references like a pro!

Understanding the Role of References in Job Applications

When employers are searching for someone to hire, there are a lot of different steps they have to take to determine whether a candidate is both technically capable and a good fit for the company . There’s the resume, the cover letter, the interview—maybe another interview, or two, or five—and the references.

You can learn a lot about a candidate from their own words, but there are certain things you can only learn from others. That’s why employers sometimes need a few recommendations from third parties as final confirmation that a candidate is a good choice.

Why include references?

References often aren’t optional. When an employer asks for them, you need to hand some over or it could greatly impact your application.

If you have prior work listed on your resume but you don’t have anyone that could act as a reference, it implies that you might have been fired, or that your manager refused to act as a reference for you. Those things can happen, but some employers may not overlook that.

Instead, it’s always best to prepare your references in advance and have them ready to send whenever the employer asks for them. This is the right way to look professional since the more experienced you are, the more people you know and the easier it is for you to get a few references.

Selecting the right references

There are a few rules you need to follow when it comes to picking references. There are three main types of reference and each has different relationships to you.

• Professional references: To list someone as a professional reference, they need to know you in a professional setting. This includes current colleagues, former colleagues, and managers. Some companies also deal with references through their HR department.
• Academic references: These are people that you know from school and college. It can include teachers, professors, academic advisors, academic counselors, and even classmates. You usually only use academic references as a fresh graduate and replace them with professional references once you get your first job or two.
• Character references: This type of reference can be a friend or a family member, and their role is to talk about your general character and personality. These aren’t always accepted by employers, and it’s important not to list a friend or family member if the employer explicitly asks for professional or academic references.

Common mistakes to avoid

Putting your references on your resume is really something you should only do if the employer asks for it directly. Otherwise, it’s best to avoid mentioning your references too early.

For most employers, checking references comes right at the end of the application process—because it would be too much work to check references for large numbers of candidates.

That’s why there’s usually no need to put your references on your resume. Some people like to put a line at the bottom of their resume saying “References available on request.” This isn’t a terrible idea, but it’s also not that necessary.

References are a requirement during many application processes so it’s assumed that anyone applying has references—it’s not something that really needs to be announced. They’ll ask you if they want them.

Preparing Your Reference List : Step-by-Step Instructions

There are a lot of things you need to think about when choosing your reference list. In this section, we’ll go over the best practices for picking references, asking permission, and formatting your reference list.

Choosing appropriate references

The first thing you need to think about is how many references you need. The most common number employers ask for is two or three, so it’s a good idea to prepare at least three.

The second thing to think about is reference types. If the employer asks for a specific type such as professional, academic, or character references, it’s best to comply with that request. If they don’t specify, your safest bet is to include only professional or academic references.

Now, as for who you pick, consider the following:

• Who they are
• What your relationship is
• What they’ll say about you
• How quickly they’re likely to respond

When you list someone as a reference, employers will contact them by email or phone to ask questions about you, so it’s important that they’re easy to get in contact with. If you choose someone busy like a CEO, there’s a chance they just won’t see the email or reply to it quickly enough.

How to ask for permission

Once you’ve decided who you want your references to be, you need to ask them if they’ll accept to do it. No one has to write you a reference if they don’t want to and it’s not very professional to list someone without asking them first.

You may have previously inquired whether someone could serve as your reference in the future—for instance, querying a manager upon leaving a job or a professor at graduation. However, it’s still necessary to reach out again with an email to confirm this arrangement.

In the email, include a quick reminder of who you are and how you’re acquainted, and then get straight to the point and ask if they’d be willing to be a reference for you. You can tell them a little about the opportunity you’re applying for and send your resume so they’re updated on what you’ve been doing.

One more important point is to give them room to refuse. It’s generally considered a responsibility to give references when requested—but sometimes people are busy, and you’re better off keeping them happy and primed for a different favor in the future than pushing them on this one now.

Formatting your reference list

Unlike when writing a cover letter or creating a resume , formatting a reference list is straightforward and very simple. All you need to include is:

• The reference’s name
• Their job title
• The company they work for
• Their phone number
• Their email address

A phone call is generally more inconvenient for references to take, so feel free to write “(preferred contact)” beside the email address to encourage the employer to use that.

Below the contact information, you should also add a short reference description. This literally only needs to be one sentence, explaining who they are, how you know each other, and how long you’ve known each other. Like this:

“Terry was my manager at Meta from 2019 to 2022 when I was working as a front-end developer.”

Showcasing References : Formats and Templates

There are two main ways to showcase your references: directly on your resume or on a separate sheet. In both cases, however, the format largely stays the same—just contact information and a reference description.

Reference List Templates

Here are some examples of how to format your reference list:

Contact Information

Reference Name:

Reference Description:

Reference Description: (In this case, use the reference’s full name)

Best practices in reference format

There isn’t really anything fancy you need to think about when it comes to reference formats. All that matters is clarity and organization. Clearly state who the reference is and provide their contact information.

One way to make the reference section match the rest of your resume (even if you’re using a separate sheet) is to use the same contact info format as you have for your own contact info at the top of your resume. By matching the order, font, bolding, and use of colons or titling, you can create a nice organized reference section that doesn’t look like it was added in a rush.

Tips for digital submission of references

Submitting your references digitally is often exactly the same as submitting them physically—just take the reference document you would have printed out and simply attach it to an email instead!

Some companies may have a special application portal that you’ve been using to submit other parts of your application, and if that’s the case, there might be a tab for references. The employer will likely tell you when and how to fill it in but in most cases, it will just involve filling in a contact form with the same information you’d usually provide. Just make sure to read the instructions carefully so you can fill everything in as the employer wants it.

Real-World Examples of Effective Reference Sections

To help you get a feel for what these sections look like and how you can write a good reference description, take a look at these resume examples . The reference list is on the second page of each example (which is recommended; there’s no need to take up valuable real estate on a resume) and closely matches the style of the resume.

Entry-Level Job Applicant

Mid-Career Professional

Senior-Level Executive

Sales Manager

School Teacher

Java Developer

In almost all cases, you should avoid putting your references directly on your resume because it takes up valuable space and isn’t useful in the early stages of the application. However, if the employer asks for references with your resume, you can include them—either on the resume itself or preferably on a separate sheet.

Most employers tend to ask for two or three references, so you want to have at least three you can use. This can feel like a lot for a fresh graduate or someone lacking experience but if you’re struggling, you can use academic references, ask people from part-time jobs you’ve worked at, or people you’ve volunteered with.

Family members can be used as a character reference but character references are not always accepted by employers. The easiest way to know if you can use a family member is to look at the type of reference the employer asked for. If they say “professional” or “academic” references, it means they don’t want character references.

Writing up your reference is easy, you just need their name, job title, company, phone number, and email address. Below, you can write a brief reference description that tells the employer how you know each other and for how long. It only needs to be one sentence, so make sure not to ramble!

You should ideally have at least one reference from every job you’ve had, so every time you get a new job, find a reference there and remove the oldest one you have from the list. If you’re at your first job, it could be a good idea to find multiple people in different roles to be your references so you don’t have to rely on academic references again.

In a normal situation, the best way to send your references is as an attachment to an email, when the references are asked for. If you prepare them in advance, you can shoot them over as soon as you get the email asking for them, and this will look professional and well-organized! Basically, references are something you don’t need to worry about until they’re asked for, which makes it pretty easy!

Related articles

• Letter of Interest Examples
• Letter of Introduction
• How to Accept a Job Offer
• Find Your Dream Job
• Reasons for Leaving a Job

Home — Essay Samples — Business — Leadership — Analyzing the Character of Snowball in George Orwell’s “Animal Farm”

Analyzing The Character of Snowball in George Orwell's "Animal Farm"

• Categories: Leadership Symbolism

About this sample

Words: 805 |

Published: Jun 6, 2024

Words: 805 | Pages: 2 | 5 min read

Table of contents

Snowball's vision and contributions, conflict and expulsion, the aftermath and legacy.

Cite this Essay

Let us write you an essay from scratch

• 450+ experts on 30 subjects ready to help
• Custom essay delivered in as few as 3 hours

Get high-quality help

Prof Ernest (PhD)

Verified writer

• Expert in: Business Literature

+ 120 experts online

By clicking “Check Writers’ Offers”, you agree to our terms of service and privacy policy . We’ll occasionally send you promo and account related email

No need to pay just yet!

Related Essays

3 pages / 1531 words

7 pages / 3063 words

1 pages / 666 words

1 pages / 416 words

Remember! This is just a sample.

You can get your custom paper by one of our expert writers.

121 writers online

Still can’t find what you need?

Browse our vast selection of original essay samples, each expertly formatted and styled

Related Essays on Leadership

Mijares, Primitivo. 'The Conjugal Dictatorship of Ferdinand and Imelda Marcos.' Bookmark, 1976

A personal evaluation of strengths is a tool that must be understood to effectively lead people. An effective leader must know how to reach goals by motivating the team. There are many strengths that translate to leadership [...]

The concept of leadership is multifaceted, encompassing various theories and models that aim to understand the dynamics of effective leadership. The 5 Levels of Leadership Essay explores one such model, highlighting the five [...]

In the realm of leadership, the ability to inspire action among followers is a hallmark of greatness. This essay delves into the techniques and principles that effective leaders employ to motivate and mobilize individuals [...]

Mr. Ghosn, at 64 years of age, has bagged plenty of achievements in his pocket to boast. Having saved Japan’s second largest automaker from the brink of doom in 1999, he has become somewhat of a cultural icon and a hero to the [...]

Leadership is directing a group of people to work together towards a common goal while assuring nobody's fundamental human rights are taken away from them. While reading Lord of the Flies, many teachers and historians argue if [...]

Related Topics

By clicking “Send”, you agree to our Terms of service and Privacy statement . We will occasionally send you account related emails.

Where do you want us to send this sample?

By clicking “Continue”, you agree to our terms of service and privacy policy.

Be careful. This essay is not unique

This essay was donated by a student and is likely to have been used and submitted before

Download this Sample

Free samples may contain mistakes and not unique parts

Sorry, we could not paraphrase this essay. Our professional writers can rewrite it and get you a unique paper.

Please check your inbox.

We can write you a custom essay that will follow your exact instructions and meet the deadlines. Let's fix your grades together!

Get Your Personalized Essay in 3 Hours or Less!

We use cookies to personalyze your web-site experience. By continuing we’ll assume you board with our cookie policy .

• Instructions Followed To The Letter
• Deadlines Met At Every Stage
• Unique And Plagiarism Free

Purdue Online Writing Lab Purdue OWL® College of Liberal Arts

Welcome to the Purdue Online Writing Lab

Welcome to the Purdue OWL

This page is brought to you by the OWL at Purdue University. When printing this page, you must include the entire legal notice.

Copyright ©1995-2018 by The Writing Lab & The OWL at Purdue and Purdue University. All rights reserved. This material may not be published, reproduced, broadcast, rewritten, or redistributed without permission. Use of this site constitutes acceptance of our terms and conditions of fair use.

The Online Writing Lab at Purdue University houses writing resources and instructional material, and we provide these as a free service of the Writing Lab at Purdue. Students, members of the community, and users worldwide will find information to assist with many writing projects. Teachers and trainers may use this material for in-class and out-of-class instruction.

The Purdue On-Campus Writing Lab and Purdue Online Writing Lab assist clients in their development as writers—no matter what their skill level—with on-campus consultations, online participation, and community engagement. The Purdue Writing Lab serves the Purdue, West Lafayette, campus and coordinates with local literacy initiatives. The Purdue OWL offers global support through online reference materials and services.

A Message From the Assistant Director of Content Development 

The Purdue OWL® is committed to supporting  students, instructors, and writers by offering a wide range of resources that are developed and revised with them in mind. To do this, the OWL team is always exploring possibilties for a better design, allowing accessibility and user experience to guide our process. As the OWL undergoes some changes, we welcome your feedback and suggestions by email at any time.

Please don't hesitate to contact us via our contact page  if you have any questions or comments.

All the best,

Social Media

Facebook twitter.

Systemic Racism in Education Essay

• To find inspiration for your paper and overcome writer’s block
• As a source of information (ensure proper referencing)
• As a template for you assignment

Introduction

The role of governments and institutions, measures and policy changes.

A form of racial inequality associated with education is the achievement gap, which refers to the persistent disparities in academic performance between white and non-white students. Theoretical perspectives suggest that this is due in part to the ways in which racism operates within educational institutions, including biased testing practices and the overrepresentation of students of color in disciplinary actions (Barber et al., 2020). Systemic racism also affects education through artificial institutional boundaries. For example, as Tourse et al. (2018) point out, young Native Americans were forbidden to speak their native languages at church-related schools in the late 19th century. At the same time, the First Nations Code Talkers were successfully employed during World War II to transmit messages. However, they only gained recognition more than half a century after the war ended.One form of racial inequality associated with the educational system is the achievement gap between students of different races. This refers to the disparity in academic achievement between students of color, particularly Black and Latino students, and their White counterparts (Howard, 2019). Structural perspectives such as the conflict theory suggest that the achievement gap results from systemic inequalities in the educational system. According to this perspective, the educational system is designed to maintain the social status quo, perpetuating the marginalization of students of color. The achievement gap can be attributed to the unequal distribution of resources, such as funding and qualified teachers, that favor White students and schools in affluent neighborhoods.On the other hand, cultural perspectives suggest that the achievement gap results from cultural factors that impact academic performance. For example, the cultural deficit theory argues that students of color are less successful in school because they lack the cultural capital valued by the dominant culture (Howard, 2019). This perspective holds that the dominant culture’s values and behaviors, such as being assertive, critical thinking, and speaking Standard English, are incompatible with those of students of color. Consequently, they are seen as inferior and not suited for academic success.Another example of the influence of systemic racism on education is seen in the historical practice of housing segregation. According to Rothstein, enclosing non-white communities into separate blocks led to students of color being more likely to attend underfunded schools with less experienced teachers and fewer resources (as cited in Tourse et al., 2018). This leads to worse quality of education, higher risk of dropping school, and generally less student retention rate, continuing to higher education. Systemic racism also manifests in the form of biased testing practices, disciplinary actions, and other forms of discrimination within educational institutions (Barber et al., 2020). This leads to people from the affected communities gaining even less opportunities to achieve quality education throughout their young years.

Governments and educational institutions have a crucial role to play in addressing racism in education. They need to recognize and acknowledge the existence of racism in education and work towards creating an inclusive and diverse environment where every student feels welcome and supported. They should also implement policies and procedures that address racism in education. Governments and educational institutions must also promote diversity and inclusion in the classroom and beyond.

Supporting marginalized groups, such as students of color, indigenous students, and students from low-income families, is also an essential aspect. This includes providing resources for tutoring and mentoring, creating safe spaces for these students to express themselves, and providing financial assistance for those in need. Governments and educational institutions can collaborate with community organizations that are working towards ending racism in education. This includes working with parents, community leaders, and civil rights organizations to develop strategies that address racism and promote diversity and inclusion in education. Today, governmental programs like affirmative action and activist movements such as Black Lives Matter exist to fight racism in social institutions, including education. Acknowledging systemic injustices and inequities requires recognizing the ways in which racism operates in various social institutions and accepting the involvement of all society members in systemic racism (Bonilla-Silva, 2021). This can involve listening to the experiences and perspectives of people from marginalized communities, educating oneself on the history and legacy of racism, and actively working to dismantle unjust systems and structures.

Measures and policy changes that educational institutions can implement to help alleviate racial inequality include increasing funding for schools in underprivileged areas and hiring and retaining diverse teaching staff. Additionally, anti-bias training for teachers and administrators can be implemented to combat unconscious bias and discrimination within the classroom (Sue et al., 2020). Furthermore, it is necessary to re-evaluate the adequacy and efficiency of the existing academic achievement evaluation systems in schools and in universities that have been regarded as flawed and biased for a long time (Baber et al., 2020). Lastly, providing additional support and resources for students from marginalized communities can help them succeed academically and prepare for college and career opportunities.

In conclusion, addressing systemic racism in education requires a concerted effort from governments, educational institutions, and the broader community. Institutions and governments can implement policies and programs that promote diversity and inclusivity through cooperation, collaboration, and communication. Societies can create and support a more equitable and just educational system to address long-term issues such as the achievement gap. Addressing racial inequality requires a concerted effort to dismantle systemic barriers and cultural biases that perpetuate the marginalization of students of color. Educational policies that provide equal opportunities for all students, such as equitable funding, teacher training, and curriculum reform, are essential in closing the achievement gap and promoting racial equality in education.

Barber, P. H., Hayes, T. B., Johnson, T. L., & Márquez-Magaña, L. (2020). Systemic racism in higher education . Science, 369 (6510), 1440–1441. Web.

Bonilla‐Silva, E. (2021). What makes systemic racism systemic? Sociological Inquiry, 91 (3), 513–533. Web.

Howard, T. C. (2019). Why race and culture matter in schools: Closing the achievement gap in America’s classrooms . Teachers College Press.

Sue, D. W., Calle, C. Z., Mendez, N., Alsaidi, S., & Glaeser, E. (2020). Microintervention strategies: What you can do to disarm and dismantle individual and systemic racism and bias. John Wiley & Sons.

Tourse, R. W. C., Hamilton-Mason, J., & Wewiorski, N. J. (2018). Systemic racism in the United States. Springer.

• Self-Reflection on Cultural Diversity and Systemic Oppression
• Systemic Racism in Lee's "To Kill a Mockingbird" & Whitehead's "The Nickel Boys"
• Sociological Imagination: Unveiling Systemic Issues in Society
• Nikki Giovanni: Challenging Racism and Patriarchy
• The Relationship between Race and Politics in America
• Discussion: The Film Green Book and Racism
• Racial Discrimination: Diann "Sally" Case
• Racism as a Dragging Factor for Children's Education, Health and General Welfare
• Chicago (A-D)
• Chicago (N-B)

IvyPanda. (2024, June 5). Systemic Racism in Education. https://ivypanda.com/essays/systemic-racism-in-education/

"Systemic Racism in Education." IvyPanda , 5 June 2024, ivypanda.com/essays/systemic-racism-in-education/.

IvyPanda . (2024) 'Systemic Racism in Education'. 5 June.

IvyPanda . 2024. "Systemic Racism in Education." June 5, 2024. https://ivypanda.com/essays/systemic-racism-in-education/.

1. IvyPanda . "Systemic Racism in Education." June 5, 2024. https://ivypanda.com/essays/systemic-racism-in-education/.

Bibliography

IvyPanda . "Systemic Racism in Education." June 5, 2024. https://ivypanda.com/essays/systemic-racism-in-education/.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 03 June 2024

Applying large language models for automated essay scoring for non-native Japanese

• Wenchao Li 1 &
• Haitao Liu 2  

Humanities and Social Sciences Communications volume  11 , Article number:  723 ( 2024 ) Cite this article

129 Accesses

1 Altmetric

Metrics details

• Language and linguistics

Recent advancements in artificial intelligence (AI) have led to an increased use of large language models (LLMs) for language assessment tasks such as automated essay scoring (AES), automated listening tests, and automated oral proficiency assessments. The application of LLMs for AES in the context of non-native Japanese, however, remains limited. This study explores the potential of LLM-based AES by comparing the efficiency of different models, i.e. two conventional machine training technology-based methods (Jess and JWriter), two LLMs (GPT and BERT), and one Japanese local LLM (Open-Calm large model). To conduct the evaluation, a dataset consisting of 1400 story-writing scripts authored by learners with 12 different first languages was used. Statistical analysis revealed that GPT-4 outperforms Jess and JWriter, BERT, and the Japanese language-specific trained Open-Calm large model in terms of annotation accuracy and predicting learning levels. Furthermore, by comparing 18 different models that utilize various prompts, the study emphasized the significance of prompts in achieving accurate and reliable evaluations using LLMs.

Similar content being viewed by others

Accurate structure prediction of biomolecular interactions with AlphaFold 3

Testing theory of mind in large language models and humans

Highly accurate protein structure prediction with AlphaFold

Conventional machine learning technology in aes.

AES has experienced significant growth with the advancement of machine learning technologies in recent decades. In the earlier stages of AES development, conventional machine learning-based approaches were commonly used. These approaches involved the following procedures: a) feeding the machine with a dataset. In this step, a dataset of essays is provided to the machine learning system. The dataset serves as the basis for training the model and establishing patterns and correlations between linguistic features and human ratings. b) the machine learning model is trained using linguistic features that best represent human ratings and can effectively discriminate learners’ writing proficiency. These features include lexical richness (Lu, 2012 ; Kyle and Crossley, 2015 ; Kyle et al. 2021 ), syntactic complexity (Lu, 2010 ; Liu, 2008 ), text cohesion (Crossley and McNamara, 2016 ), and among others. Conventional machine learning approaches in AES require human intervention, such as manual correction and annotation of essays. This human involvement was necessary to create a labeled dataset for training the model. Several AES systems have been developed using conventional machine learning technologies. These include the Intelligent Essay Assessor (Landauer et al. 2003 ), the e-rater engine by Educational Testing Service (Attali and Burstein, 2006 ; Burstein, 2003 ), MyAccess with the InterlliMetric scoring engine by Vantage Learning (Elliot, 2003 ), and the Bayesian Essay Test Scoring system (Rudner and Liang, 2002 ). These systems have played a significant role in automating the essay scoring process and providing quick and consistent feedback to learners. However, as touched upon earlier, conventional machine learning approaches rely on predetermined linguistic features and often require manual intervention, making them less flexible and potentially limiting their generalizability to different contexts.

In the context of the Japanese language, conventional machine learning-incorporated AES tools include Jess (Ishioka and Kameda, 2006 ) and JWriter (Lee and Hasebe, 2017 ). Jess assesses essays by deducting points from the perfect score, utilizing the Mainichi Daily News newspaper as a database. The evaluation criteria employed by Jess encompass various aspects, such as rhetorical elements (e.g., reading comprehension, vocabulary diversity, percentage of complex words, and percentage of passive sentences), organizational structures (e.g., forward and reverse connection structures), and content analysis (e.g., latent semantic indexing). JWriter employs linear regression analysis to assign weights to various measurement indices, such as average sentence length and total number of characters. These weights are then combined to derive the overall score. A pilot study involving the Jess model was conducted on 1320 essays at different proficiency levels, including primary, intermediate, and advanced. However, the results indicated that the Jess model failed to significantly distinguish between these essay levels. Out of the 16 measures used, four measures, namely median sentence length, median clause length, median number of phrases, and maximum number of phrases, did not show statistically significant differences between the levels. Additionally, two measures exhibited between-level differences but lacked linear progression: the number of attributives declined words and the Kanji/kana ratio. On the other hand, the remaining measures, including maximum sentence length, maximum clause length, number of attributive conjugated words, maximum number of consecutive infinitive forms, maximum number of conjunctive-particle clauses, k characteristic value, percentage of big words, and percentage of passive sentences, demonstrated statistically significant between-level differences and displayed linear progression.

Both Jess and JWriter exhibit notable limitations, including the manual selection of feature parameters and weights, which can introduce biases into the scoring process. The reliance on human annotators to label non-native language essays also introduces potential noise and variability in the scoring. Furthermore, an important concern is the possibility of system manipulation and cheating by learners who are aware of the regression equation utilized by the models (Hirao et al. 2020 ). These limitations emphasize the need for further advancements in AES systems to address these challenges.

Deep learning technology in AES

Deep learning has emerged as one of the approaches for improving the accuracy and effectiveness of AES. Deep learning-based AES methods utilize artificial neural networks that mimic the human brain’s functioning through layered algorithms and computational units. Unlike conventional machine learning, deep learning autonomously learns from the environment and past errors without human intervention. This enables deep learning models to establish nonlinear correlations, resulting in higher accuracy. Recent advancements in deep learning have led to the development of transformers, which are particularly effective in learning text representations. Noteworthy examples include bidirectional encoder representations from transformers (BERT) (Devlin et al. 2019 ) and the generative pretrained transformer (GPT) (OpenAI).

BERT is a linguistic representation model that utilizes a transformer architecture and is trained on two tasks: masked linguistic modeling and next-sentence prediction (Hirao et al. 2020 ; Vaswani et al. 2017 ). In the context of AES, BERT follows specific procedures, as illustrated in Fig. 1 : (a) the tokenized prompts and essays are taken as input; (b) special tokens, such as [CLS] and [SEP], are added to mark the beginning and separation of prompts and essays; (c) the transformer encoder processes the prompt and essay sequences, resulting in hidden layer sequences; (d) the hidden layers corresponding to the [CLS] tokens (T[CLS]) represent distributed representations of the prompts and essays; and (e) a multilayer perceptron uses these distributed representations as input to obtain the final score (Hirao et al. 2020 ).

AES system with BERT (Hirao et al. 2020 ).

The training of BERT using a substantial amount of sentence data through the Masked Language Model (MLM) allows it to capture contextual information within the hidden layers. Consequently, BERT is expected to be capable of identifying artificial essays as invalid and assigning them lower scores (Mizumoto and Eguchi, 2023 ). In the context of AES for nonnative Japanese learners, Hirao et al. ( 2020 ) combined the long short-term memory (LSTM) model proposed by Hochreiter and Schmidhuber ( 1997 ) with BERT to develop a tailored automated Essay Scoring System. The findings of their study revealed that the BERT model outperformed both the conventional machine learning approach utilizing character-type features such as “kanji” and “hiragana”, as well as the standalone LSTM model. Takeuchi et al. ( 2021 ) presented an approach to Japanese AES that eliminates the requirement for pre-scored essays by relying solely on reference texts or a model answer for the essay task. They investigated multiple similarity evaluation methods, including frequency of morphemes, idf values calculated on Wikipedia, LSI, LDA, word-embedding vectors, and document vectors produced by BERT. The experimental findings revealed that the method utilizing the frequency of morphemes with idf values exhibited the strongest correlation with human-annotated scores across different essay tasks. The utilization of BERT in AES encounters several limitations. Firstly, essays often exceed the model’s maximum length limit. Second, only score labels are available for training, which restricts access to additional information.

Mizumoto and Eguchi ( 2023 ) were pioneers in employing the GPT model for AES in non-native English writing. Their study focused on evaluating the accuracy and reliability of AES using the GPT-3 text-davinci-003 model, analyzing a dataset of 12,100 essays from the corpus of nonnative written English (TOEFL11). The findings indicated that AES utilizing the GPT-3 model exhibited a certain degree of accuracy and reliability. They suggest that GPT-3-based AES systems hold the potential to provide support for human ratings. However, applying GPT model to AES presents a unique natural language processing (NLP) task that involves considerations such as nonnative language proficiency, the influence of the learner’s first language on the output in the target language, and identifying linguistic features that best indicate writing quality in a specific language. These linguistic features may differ morphologically or syntactically from those present in the learners’ first language, as observed in (1)–(3).

我-送了-他-一本-书

Wǒ-sòngle-tā-yī běn-shū

1 sg .-give. past- him-one .cl- book

“I gave him a book.”

Agglutinative

彼-に-本-を-あげ-まし-た

Kare-ni-hon-o-age-mashi-ta

3 sg .- dat -hon- acc- give.honorification. past

Inflectional

give, give-s, gave, given, giving

Additionally, the morphological agglutination and subject-object-verb (SOV) order in Japanese, along with its idiomatic expressions, pose additional challenges for applying language models in AES tasks (4).

足-が 棒-に なり-ました

Ashi-ga bo-ni nar-mashita

leg- nom stick- dat become- past

“My leg became like a stick (I am extremely tired).”

The example sentence provided demonstrates the morpho-syntactic structure of Japanese and the presence of an idiomatic expression. In this sentence, the verb “なる” (naru), meaning “to become”, appears at the end of the sentence. The verb stem “なり” (nari) is attached with morphemes indicating honorification (“ます” - mashu) and tense (“た” - ta), showcasing agglutination. While the sentence can be literally translated as “my leg became like a stick”, it carries an idiomatic interpretation that implies “I am extremely tired”.

To overcome this issue, CyberAgent Inc. ( 2023 ) has developed the Open-Calm series of language models specifically designed for Japanese. Open-Calm consists of pre-trained models available in various sizes, such as Small, Medium, Large, and 7b. Figure 2 depicts the fundamental structure of the Open-Calm model. A key feature of this architecture is the incorporation of the Lora Adapter and GPT-NeoX frameworks, which can enhance its language processing capabilities.

GPT-NeoX Model Architecture (Okgetheng and Takeuchi 2024 ).

In a recent study conducted by Okgetheng and Takeuchi ( 2024 ), they assessed the efficacy of Open-Calm language models in grading Japanese essays. The research utilized a dataset of approximately 300 essays, which were annotated by native Japanese educators. The findings of the study demonstrate the considerable potential of Open-Calm language models in automated Japanese essay scoring. Specifically, among the Open-Calm family, the Open-Calm Large model (referred to as OCLL) exhibited the highest performance. However, it is important to note that, as of the current date, the Open-Calm Large model does not offer public access to its server. Consequently, users are required to independently deploy and operate the environment for OCLL. In order to utilize OCLL, users must have a PC equipped with an NVIDIA GeForce RTX 3060 (8 or 12 GB VRAM).

In summary, while the potential of LLMs in automated scoring of nonnative Japanese essays has been demonstrated in two studies—BERT-driven AES (Hirao et al. 2020 ) and OCLL-based AES (Okgetheng and Takeuchi, 2024 )—the number of research efforts in this area remains limited.

Another significant challenge in applying LLMs to AES lies in prompt engineering and ensuring its reliability and effectiveness (Brown et al. 2020 ; Rae et al. 2021 ; Zhang et al. 2021 ). Various prompting strategies have been proposed, such as the zero-shot chain of thought (CoT) approach (Kojima et al. 2022 ), which involves manually crafting diverse and effective examples. However, manual efforts can lead to mistakes. To address this, Zhang et al. ( 2021 ) introduced an automatic CoT prompting method called Auto-CoT, which demonstrates matching or superior performance compared to the CoT paradigm. Another prompt framework is trees of thoughts, enabling a model to self-evaluate its progress at intermediate stages of problem-solving through deliberate reasoning (Yao et al. 2023 ).

Beyond linguistic studies, there has been a noticeable increase in the number of foreign workers in Japan and Japanese learners worldwide (Ministry of Health, Labor, and Welfare of Japan, 2022 ; Japan Foundation, 2021 ). However, existing assessment methods, such as the Japanese Language Proficiency Test (JLPT), J-CAT, and TTBJ Footnote 1 , primarily focus on reading, listening, vocabulary, and grammar skills, neglecting the evaluation of writing proficiency. As the number of workers and language learners continues to grow, there is a rising demand for an efficient AES system that can reduce costs and time for raters and be utilized for employment, examinations, and self-study purposes.

This study aims to explore the potential of LLM-based AES by comparing the effectiveness of five models: two LLMs (GPT Footnote 2 and BERT), one Japanese local LLM (OCLL), and two conventional machine learning-based methods (linguistic feature-based scoring tools - Jess and JWriter).

The research questions addressed in this study are as follows:

To what extent do the LLM-driven AES and linguistic feature-based AES, when used as automated tools to support human rating, accurately reflect test takers’ actual performance?

What influence does the prompt have on the accuracy and performance of LLM-based AES methods?

The subsequent sections of the manuscript cover the methodology, including the assessment measures for nonnative Japanese writing proficiency, criteria for prompts, and the dataset. The evaluation section focuses on the analysis of annotations and rating scores generated by LLM-driven and linguistic feature-based AES methods.

Methodology

The dataset utilized in this study was obtained from the International Corpus of Japanese as a Second Language (I-JAS) Footnote 3 . This corpus consisted of 1000 participants who represented 12 different first languages. For the study, the participants were given a story-writing task on a personal computer. They were required to write two stories based on the 4-panel illustrations titled “Picnic” and “The key” (see Appendix A). Background information for the participants was provided by the corpus, including their Japanese language proficiency levels assessed through two online tests: J-CAT and SPOT. These tests evaluated their reading, listening, vocabulary, and grammar abilities. The learners’ proficiency levels were categorized into six levels aligned with the Common European Framework of Reference for Languages (CEFR) and the Reference Framework for Japanese Language Education (RFJLE): A1, A2, B1, B2, C1, and C2. According to Lee et al. ( 2015 ), there is a high level of agreement (r = 0.86) between the J-CAT and SPOT assessments, indicating that the proficiency certifications provided by J-CAT are consistent with those of SPOT. However, it is important to note that the scores of J-CAT and SPOT do not have a one-to-one correspondence. In this study, the J-CAT scores were used as a benchmark to differentiate learners of different proficiency levels. A total of 1400 essays were utilized, representing the beginner (aligned with A1), A2, B1, B2, C1, and C2 levels based on the J-CAT scores. Table 1 provides information about the learners’ proficiency levels and their corresponding J-CAT and SPOT scores.

A dataset comprising a total of 1400 essays from the story writing tasks was collected. Among these, 714 essays were utilized to evaluate the reliability of the LLM-based AES method, while the remaining 686 essays were designated as development data to assess the LLM-based AES’s capability to distinguish participants with varying proficiency levels. The GPT 4 API was used in this study. A detailed explanation of the prompt-assessment criteria is provided in Section Prompt . All essays were sent to the model for measurement and scoring.

Measures of writing proficiency for nonnative Japanese

Japanese exhibits a morphologically agglutinative structure where morphemes are attached to the word stem to convey grammatical functions such as tense, aspect, voice, and honorifics, e.g. (5).

食べ-させ-られ-まし-た-か

tabe-sase-rare-mashi-ta-ka

[eat (stem)-causative-passive voice-honorification-tense. past-question marker]

Japanese employs nine case particles to indicate grammatical functions: the nominative case particle が (ga), the accusative case particle を (o), the genitive case particle の (no), the dative case particle に (ni), the locative/instrumental case particle で (de), the ablative case particle から (kara), the directional case particle へ (e), and the comitative case particle と (to). The agglutinative nature of the language, combined with the case particle system, provides an efficient means of distinguishing between active and passive voice, either through morphemes or case particles, e.g. 食べる taberu “eat concusive . ” (active voice); 食べられる taberareru “eat concusive . ” (passive voice). In the active voice, “パン を 食べる” (pan o taberu) translates to “to eat bread”. On the other hand, in the passive voice, it becomes “パン が 食べられた” (pan ga taberareta), which means “(the) bread was eaten”. Additionally, it is important to note that different conjugations of the same lemma are considered as one type in order to ensure a comprehensive assessment of the language features. For example, e.g., 食べる taberu “eat concusive . ”; 食べている tabeteiru “eat progress .”; 食べた tabeta “eat past . ” as one type.

To incorporate these features, previous research (Suzuki, 1999 ; Watanabe et al. 1988 ; Ishioka, 2001 ; Ishioka and Kameda, 2006 ; Hirao et al. 2020 ) has identified complexity, fluency, and accuracy as crucial factors for evaluating writing quality. These criteria are assessed through various aspects, including lexical richness (lexical density, diversity, and sophistication), syntactic complexity, and cohesion (Kyle et al. 2021 ; Mizumoto and Eguchi, 2023 ; Ure, 1971 ; Halliday, 1985 ; Barkaoui and Hadidi, 2020 ; Zenker and Kyle, 2021 ; Kim et al. 2018 ; Lu, 2017 ; Ortega, 2015 ). Therefore, this study proposes five scoring categories: lexical richness, syntactic complexity, cohesion, content elaboration, and grammatical accuracy. A total of 16 measures were employed to capture these categories. The calculation process and specific details of these measures can be found in Table 2 .

T-unit, first introduced by Hunt ( 1966 ), is a measure used for evaluating speech and composition. It serves as an indicator of syntactic development and represents the shortest units into which a piece of discourse can be divided without leaving any sentence fragments. In the context of Japanese language assessment, Sakoda and Hosoi ( 2020 ) utilized T-unit as the basic unit to assess the accuracy and complexity of Japanese learners’ speaking and storytelling. The calculation of T-units in Japanese follows the following principles:

A single main clause constitutes 1 T-unit, regardless of the presence or absence of dependent clauses, e.g. (6).

ケンとマリはピクニックに行きました (main clause): 1 T-unit.

If a sentence contains a main clause along with subclauses, each subclause is considered part of the same T-unit, e.g. (7).

天気が良かった の で (subclause)、ケンとマリはピクニックに行きました (main clause): 1 T-unit.

In the case of coordinate clauses, where multiple clauses are connected, each coordinated clause is counted separately. Thus, a sentence with coordinate clauses may have 2 T-units or more, e.g. (8).

ケンは地図で場所を探して (coordinate clause)、マリはサンドイッチを作りました (coordinate clause): 2 T-units.

Lexical diversity refers to the range of words used within a text (Engber, 1995 ; Kyle et al. 2021 ) and is considered a useful measure of the breadth of vocabulary in L n production (Jarvis, 2013a , 2013b ).

The type/token ratio (TTR) is widely recognized as a straightforward measure for calculating lexical diversity and has been employed in numerous studies. These studies have demonstrated a strong correlation between TTR and other methods of measuring lexical diversity (e.g., Bentz et al. 2016 ; Čech and Miroslav, 2018 ; Çöltekin and Taraka, 2018 ). TTR is computed by considering both the number of unique words (types) and the total number of words (tokens) in a given text. Given that the length of learners’ writing texts can vary, this study employs the moving average type-token ratio (MATTR) to mitigate the influence of text length. MATTR is calculated using a 50-word moving window. Initially, a TTR is determined for words 1–50 in an essay, followed by words 2–51, 3–52, and so on until the end of the essay is reached (Díez-Ortega and Kyle, 2023 ). The final MATTR scores were obtained by averaging the TTR scores for all 50-word windows. The following formula was employed to derive MATTR:

\({\rm{MATTR}}({\rm{W}})=\frac{{\sum }_{{\rm{i}}=1}^{{\rm{N}}-{\rm{W}}+1}{{\rm{F}}}_{{\rm{i}}}}{{\rm{W}}({\rm{N}}-{\rm{W}}+1)}\)

Here, N refers to the number of tokens in the corpus. W is the randomly selected token size (W < N). \({F}_{i}\) is the number of types in each window. The \({\rm{MATTR}}({\rm{W}})\) is the mean of a series of type-token ratios (TTRs) based on the word form for all windows. It is expected that individuals with higher language proficiency will produce texts with greater lexical diversity, as indicated by higher MATTR scores.

Lexical density was captured by the ratio of the number of lexical words to the total number of words (Lu, 2012 ). Lexical sophistication refers to the utilization of advanced vocabulary, often evaluated through word frequency indices (Crossley et al. 2013 ; Haberman, 2008 ; Kyle and Crossley, 2015 ; Laufer and Nation, 1995 ; Lu, 2012 ; Read, 2000 ). In line of writing, lexical sophistication can be interpreted as vocabulary breadth, which entails the appropriate usage of vocabulary items across various lexicon-grammatical contexts and registers (Garner et al. 2019 ; Kim et al. 2018 ; Kyle et al. 2018 ). In Japanese specifically, words are considered lexically sophisticated if they are not included in the “Japanese Education Vocabulary List Ver 1.0”. Footnote 4 Consequently, lexical sophistication was calculated by determining the number of sophisticated word types relative to the total number of words per essay. Furthermore, it has been suggested that, in Japanese writing, sentences should ideally have a length of no more than 40 to 50 characters, as this promotes readability. Therefore, the median and maximum sentence length can be considered as useful indices for assessment (Ishioka and Kameda, 2006 ).

Syntactic complexity was assessed based on several measures, including the mean length of clauses, verb phrases per T-unit, clauses per T-unit, dependent clauses per T-unit, complex nominals per clause, adverbial clauses per clause, coordinate phrases per clause, and mean dependency distance (MDD). The MDD reflects the distance between the governor and dependent positions in a sentence. A larger dependency distance indicates a higher cognitive load and greater complexity in syntactic processing (Liu, 2008 ; Liu et al. 2017 ). The MDD has been established as an efficient metric for measuring syntactic complexity (Jiang, Quyang, and Liu, 2019 ; Li and Yan, 2021 ). To calculate the MDD, the position numbers of the governor and dependent are subtracted, assuming that words in a sentence are assigned in a linear order, such as W1 … Wi … Wn. In any dependency relationship between words Wa and Wb, Wa is the governor and Wb is the dependent. The MDD of the entire sentence was obtained by taking the absolute value of governor – dependent:

MDD = \(\frac{1}{n}{\sum }_{i=1}^{n}|{\rm{D}}{{\rm{D}}}_{i}|\)

In this formula, \(n\) represents the number of words in the sentence, and \({DD}i\) is the dependency distance of the \({i}^{{th}}\) dependency relationship of a sentence. Building on this, the annotation of sentence ‘Mary-ga-John-ni-keshigomu-o-watashita was [Mary- top -John- dat -eraser- acc -give- past] ’. The sentence’s MDD would be 2. Table 3 provides the CSV file as a prompt for GPT 4.

Cohesion (semantic similarity) and content elaboration aim to capture the ideas presented in test taker’s essays. Cohesion was assessed using three measures: Synonym overlap/paragraph (topic), Synonym overlap/paragraph (keywords), and word2vec cosine similarity. Content elaboration and development were measured as the number of metadiscourse markers (type)/number of words. To capture content closely, this study proposed a novel-distance based representation, by encoding the cosine distance between the essay (by learner) and essay task’s (topic and keyword) i -vectors. The learner’s essay is decoded into a word sequence, and aligned to the essay task’ topic and keyword for log-likelihood measurement. The cosine distance reveals the content elaboration score in the leaners’ essay. The mathematical equation of cosine similarity between target-reference vectors is shown in (11), assuming there are i essays and ( L i , …. L n ) and ( N i , …. N n ) are the vectors representing the learner and task’s topic and keyword respectively. The content elaboration distance between L i and N i was calculated as follows:

\(\cos \left(\theta \right)=\frac{{\rm{L}}\,\cdot\, {\rm{N}}}{\left|{\rm{L}}\right|{\rm{|N|}}}=\frac{\mathop{\sum }\nolimits_{i=1}^{n}{L}_{i}{N}_{i}}{\sqrt{\mathop{\sum }\nolimits_{i=1}^{n}{L}_{i}^{2}}\sqrt{\mathop{\sum }\nolimits_{i=1}^{n}{N}_{i}^{2}}}\)

A high similarity value indicates a low difference between the two recognition outcomes, which in turn suggests a high level of proficiency in content elaboration.

To evaluate the effectiveness of the proposed measures in distinguishing different proficiency levels among nonnative Japanese speakers’ writing, we conducted a multi-faceted Rasch measurement analysis (Linacre, 1994 ). This approach applies measurement models to thoroughly analyze various factors that can influence test outcomes, including test takers’ proficiency, item difficulty, and rater severity, among others. The underlying principles and functionality of multi-faceted Rasch measurement are illustrated in (12).

\(\log \left(\frac{{P}_{{nijk}}}{{P}_{{nij}(k-1)}}\right)={B}_{n}-{D}_{i}-{C}_{j}-{F}_{k}\)

(12) defines the logarithmic transformation of the probability ratio ( P nijk /P nij(k-1) )) as a function of multiple parameters. Here, n represents the test taker, i denotes a writing proficiency measure, j corresponds to the human rater, and k represents the proficiency score. The parameter B n signifies the proficiency level of test taker n (where n ranges from 1 to N). D j represents the difficulty parameter of test item i (where i ranges from 1 to L), while C j represents the severity of rater j (where j ranges from 1 to J). Additionally, F k represents the step difficulty for a test taker to move from score ‘k-1’ to k . P nijk refers to the probability of rater j assigning score k to test taker n for test item i . P nij(k-1) represents the likelihood of test taker n being assigned score ‘k-1’ by rater j for test item i . Each facet within the test is treated as an independent parameter and estimated within the same reference framework. To evaluate the consistency of scores obtained through both human and computer analysis, we utilized the Infit mean-square statistic. This statistic is a chi-square measure divided by the degrees of freedom and is weighted with information. It demonstrates higher sensitivity to unexpected patterns in responses to items near a person’s proficiency level (Linacre, 2002 ). Fit statistics are assessed based on predefined thresholds for acceptable fit. For the Infit MNSQ, which has a mean of 1.00, different thresholds have been suggested. Some propose stricter thresholds ranging from 0.7 to 1.3 (Bond et al. 2021 ), while others suggest more lenient thresholds ranging from 0.5 to 1.5 (Eckes, 2009 ). In this study, we adopted the criterion of 0.70–1.30 for the Infit MNSQ.

Moving forward, we can now proceed to assess the effectiveness of the 16 proposed measures based on five criteria for accurately distinguishing various levels of writing proficiency among non-native Japanese speakers. To conduct this evaluation, we utilized the development dataset from the I-JAS corpus, as described in Section Dataset . Table 4 provides a measurement report that presents the performance details of the 14 metrics under consideration. The measure separation was found to be 4.02, indicating a clear differentiation among the measures. The reliability index for the measure separation was 0.891, suggesting consistency in the measurement. Similarly, the person separation reliability index was 0.802, indicating the accuracy of the assessment in distinguishing between individuals. All 16 measures demonstrated Infit mean squares within a reasonable range, ranging from 0.76 to 1.28. The Synonym overlap/paragraph (topic) measure exhibited a relatively high outfit mean square of 1.46, although the Infit mean square falls within an acceptable range. The standard error for the measures ranged from 0.13 to 0.28, indicating the precision of the estimates.

Table 5 further illustrated the weights assigned to different linguistic measures for score prediction, with higher weights indicating stronger correlations between those measures and higher scores. Specifically, the following measures exhibited higher weights compared to others: moving average type token ratio per essay has a weight of 0.0391. Mean dependency distance had a weight of 0.0388. Mean length of clause, calculated by dividing the number of words by the number of clauses, had a weight of 0.0374. Complex nominals per T-unit, calculated by dividing the number of complex nominals by the number of T-units, had a weight of 0.0379. Coordinate phrases rate, calculated by dividing the number of coordinate phrases by the number of clauses, had a weight of 0.0325. Grammatical error rate, representing the number of errors per essay, had a weight of 0.0322.

Criteria (output indicator)

The criteria used to evaluate the writing ability in this study were based on CEFR, which follows a six-point scale ranging from A1 to C2. To assess the quality of Japanese writing, the scoring criteria from Table 6 were utilized. These criteria were derived from the IELTS writing standards and served as assessment guidelines and prompts for the written output.

A prompt is a question or detailed instruction that is provided to the model to obtain a proper response. After several pilot experiments, we decided to provide the measures (Section Measures of writing proficiency for nonnative Japanese ) as the input prompt and use the criteria (Section Criteria (output indicator) ) as the output indicator. Regarding the prompt language, considering that the LLM was tasked with rating Japanese essays, would prompt in Japanese works better Footnote 5 ? We conducted experiments comparing the performance of GPT-4 using both English and Japanese prompts. Additionally, we utilized the Japanese local model OCLL with Japanese prompts. Multiple trials were conducted using the same sample. Regardless of the prompt language used, we consistently obtained the same grading results with GPT-4, which assigned a grade of B1 to the writing sample. This suggested that GPT-4 is reliable and capable of producing consistent ratings regardless of the prompt language. On the other hand, when we used Japanese prompts with the Japanese local model “OCLL”, we encountered inconsistent grading results. Out of 10 attempts with OCLL, only 6 yielded consistent grading results (B1), while the remaining 4 showed different outcomes, including A1 and B2 grades. These findings indicated that the language of the prompt was not the determining factor for reliable AES. Instead, the size of the training data and the model parameters played crucial roles in achieving consistent and reliable AES results for the language model.

The following is the utilized prompt, which details all measures and requires the LLM to score the essays using holistic and trait scores.

Please evaluate Japanese essays written by Japanese learners and assign a score to each essay on a six-point scale, ranging from A1, A2, B1, B2, C1 to C2. Additionally, please provide trait scores and display the calculation process for each trait score. The scoring should be based on the following criteria:

Moving average type-token ratio.

Number of lexical words (token) divided by the total number of words per essay.

Number of sophisticated word types divided by the total number of words per essay.

Mean length of clause.

Verb phrases per T-unit.

Clauses per T-unit.

Dependent clauses per T-unit.

Complex nominals per clause.

Adverbial clauses per clause.

Coordinate phrases per clause.

Mean dependency distance.

Synonym overlap paragraph (topic and keywords).

Word2vec cosine similarity.

Connectives per essay.

Conjunctions per essay.

Number of metadiscourse markers (types) divided by the total number of words.

Number of errors per essay.

Japanese essay text

出かける前に二人が地図を見ている間に、サンドイッチを入れたバスケットに犬が入ってしまいました。それに気づかずに二人は楽しそうに出かけて行きました。やがて突然犬がバスケットから飛び出し、二人は驚きました。バスケット の 中を見ると、食べ物はすべて犬に食べられていて、二人は困ってしまいました。(ID_JJJ01_SW1)

The score of the example above was B1. Figure 3 provides an example of holistic and trait scores provided by GPT-4 (with a prompt indicating all measures) via Bing Footnote 6 .

Example of GPT-4 AES and feedback (with a prompt indicating all measures).

Statistical analysis

The aim of this study is to investigate the potential use of LLM for nonnative Japanese AES. It seeks to compare the scoring outcomes obtained from feature-based AES tools, which rely on conventional machine learning technology (i.e. Jess, JWriter), with those generated by AI-driven AES tools utilizing deep learning technology (BERT, GPT, OCLL). To assess the reliability of a computer-assisted annotation tool, the study initially established human-human agreement as the benchmark measure. Subsequently, the performance of the LLM-based method was evaluated by comparing it to human-human agreement.

To assess annotation agreement, the study employed standard measures such as precision, recall, and F-score (Brants 2000 ; Lu 2010 ), along with the quadratically weighted kappa (QWK) to evaluate the consistency and agreement in the annotation process. Assume A and B represent human annotators. When comparing the annotations of the two annotators, the following results are obtained. The evaluation of precision, recall, and F-score metrics was illustrated in equations (13) to (15).

\({\rm{Recall}}(A,B)=\frac{{\rm{Number}}\,{\rm{of}}\,{\rm{identical}}\,{\rm{nodes}}\,{\rm{in}}\,A\,{\rm{and}}\,B}{{\rm{Number}}\,{\rm{of}}\,{\rm{nodes}}\,{\rm{in}}\,A}\)

\({\rm{Precision}}(A,\,B)=\frac{{\rm{Number}}\,{\rm{of}}\,{\rm{identical}}\,{\rm{nodes}}\,{\rm{in}}\,A\,{\rm{and}}\,B}{{\rm{Number}}\,{\rm{of}}\,{\rm{nodes}}\,{\rm{in}}\,B}\)

The F-score is the harmonic mean of recall and precision:

\({\rm{F}}-{\rm{score}}=\frac{2* ({\rm{Precision}}* {\rm{Recall}})}{{\rm{Precision}}+{\rm{Recall}}}\)

The highest possible value of an F-score is 1.0, indicating perfect precision and recall, and the lowest possible value is 0, if either precision or recall are zero.

In accordance with Taghipour and Ng ( 2016 ), the calculation of QWK involves two steps:

Step 1: Construct a weight matrix W as follows:

\({W}_{{ij}}=\frac{{(i-j)}^{2}}{{(N-1)}^{2}}\)

i represents the annotation made by the tool, while j represents the annotation made by a human rater. N denotes the total number of possible annotations. Matrix O is subsequently computed, where O_( i, j ) represents the count of data annotated by the tool ( i ) and the human annotator ( j ). On the other hand, E refers to the expected count matrix, which undergoes normalization to ensure that the sum of elements in E matches the sum of elements in O.

Step 2: With matrices O and E, the QWK is obtained as follows:

K = 1- \(\frac{\sum i,j{W}_{i,j}\,{O}_{i,j}}{\sum i,j{W}_{i,j}\,{E}_{i,j}}\)

The value of the quadratic weighted kappa increases as the level of agreement improves. Further, to assess the accuracy of LLM scoring, the proportional reductive mean square error (PRMSE) was employed. The PRMSE approach takes into account the variability observed in human ratings to estimate the rater error, which is then subtracted from the variance of the human labels. This calculation provides an overall measure of agreement between the automated scores and true scores (Haberman et al. 2015 ; Loukina et al. 2020 ; Taghipour and Ng, 2016 ). The computation of PRMSE involves the following steps:

Step 1: Calculate the mean squared errors (MSEs) for the scoring outcomes of the computer-assisted tool (MSE tool) and the human scoring outcomes (MSE human).

Step 2: Determine the PRMSE by comparing the MSE of the computer-assisted tool (MSE tool) with the MSE from human raters (MSE human), using the following formula:

\({\rm{PRMSE}}=1-\frac{({\rm{MSE}}\,{\rm{tool}})\,}{({\rm{MSE}}\,{\rm{human}})\,}=1-\,\frac{{\sum }_{i}^{n}=1{({{\rm{y}}}_{i}-{\hat{{\rm{y}}}}_{{\rm{i}}})}^{2}}{{\sum }_{i}^{n}=1{({{\rm{y}}}_{i}-\hat{{\rm{y}}})}^{2}}\)

In the numerator, ŷi represents the scoring outcome predicted by a specific LLM-driven AES system for a given sample. The term y i − ŷ i represents the difference between this predicted outcome and the mean value of all LLM-driven AES systems’ scoring outcomes. It quantifies the deviation of the specific LLM-driven AES system’s prediction from the average prediction of all LLM-driven AES systems. In the denominator, y i − ŷ represents the difference between the scoring outcome provided by a specific human rater for a given sample and the mean value of all human raters’ scoring outcomes. It measures the discrepancy between the specific human rater’s score and the average score given by all human raters. The PRMSE is then calculated by subtracting the ratio of the MSE tool to the MSE human from 1. PRMSE falls within the range of 0 to 1, with larger values indicating reduced errors in LLM’s scoring compared to those of human raters. In other words, a higher PRMSE implies that LLM’s scoring demonstrates greater accuracy in predicting the true scores (Loukina et al. 2020 ). The interpretation of kappa values, ranging from 0 to 1, is based on the work of Landis and Koch ( 1977 ). Specifically, the following categories are assigned to different ranges of kappa values: −1 indicates complete inconsistency, 0 indicates random agreement, 0.0 ~ 0.20 indicates extremely low level of agreement (slight), 0.21 ~ 0.40 indicates moderate level of agreement (fair), 0.41 ~ 0.60 indicates medium level of agreement (moderate), 0.61 ~ 0.80 indicates high level of agreement (substantial), 0.81 ~ 1 indicates almost perfect level of agreement. All statistical analyses were executed using Python script.

Results and discussion

Annotation reliability of the llm.

This section focuses on assessing the reliability of the LLM’s annotation and scoring capabilities. To evaluate the reliability, several tests were conducted simultaneously, aiming to achieve the following objectives:

Assess the LLM’s ability to differentiate between test takers with varying levels of oral proficiency.

Determine the level of agreement between the annotations and scoring performed by the LLM and those done by human raters.

The evaluation of the results encompassed several metrics, including: precision, recall, F-Score, quadratically-weighted kappa, proportional reduction of mean squared error, Pearson correlation, and multi-faceted Rasch measurement.

Inter-annotator agreement (human–human annotator agreement)

We started with an agreement test of the two human annotators. Two trained annotators were recruited to determine the writing task data measures. A total of 714 scripts, as the test data, was utilized. Each analysis lasted 300–360 min. Inter-annotator agreement was evaluated using the standard measures of precision, recall, and F-score and QWK. Table 7 presents the inter-annotator agreement for the various indicators. As shown, the inter-annotator agreement was fairly high, with F-scores ranging from 1.0 for sentence and word number to 0.666 for grammatical errors.

The findings from the QWK analysis provided further confirmation of the inter-annotator agreement. The QWK values covered a range from 0.950 ( p  = 0.000) for sentence and word number to 0.695 for synonym overlap number (keyword) and grammatical errors ( p  = 0.001).

Agreement of annotation outcomes between human and LLM

To evaluate the consistency between human annotators and LLM annotators (BERT, GPT, OCLL) across the indices, the same test was conducted. The results of the inter-annotator agreement (F-score) between LLM and human annotation are provided in Appendix B-D. The F-scores ranged from 0.706 for Grammatical error # for OCLL-human to a perfect 1.000 for GPT-human, for sentences, clauses, T-units, and words. These findings were further supported by the QWK analysis, which showed agreement levels ranging from 0.807 ( p  = 0.001) for metadiscourse markers for OCLL-human to 0.962 for words ( p  = 0.000) for GPT-human. The findings demonstrated that the LLM annotation achieved a significant level of accuracy in identifying measurement units and counts.

Reliability of LLM-driven AES’s scoring and discriminating proficiency levels

This section examines the reliability of the LLM-driven AES scoring through a comparison of the scoring outcomes produced by human raters and the LLM ( Reliability of LLM-driven AES scoring ). It also assesses the effectiveness of the LLM-based AES system in differentiating participants with varying proficiency levels ( Reliability of LLM-driven AES discriminating proficiency levels ).

Reliability of LLM-driven AES scoring

Table 8 summarizes the QWK coefficient analysis between the scores computed by the human raters and the GPT-4 for the individual essays from I-JAS Footnote 7 . As shown, the QWK of all measures ranged from k  = 0.819 for lexical density (number of lexical words (tokens)/number of words per essay) to k  = 0.644 for word2vec cosine similarity. Table 9 further presents the Pearson correlations between the 16 writing proficiency measures scored by human raters and GPT 4 for the individual essays. The correlations ranged from 0.672 for syntactic complexity to 0.734 for grammatical accuracy. The correlations between the writing proficiency scores assigned by human raters and the BERT-based AES system were found to range from 0.661 for syntactic complexity to 0.713 for grammatical accuracy. The correlations between the writing proficiency scores given by human raters and the OCLL-based AES system ranged from 0.654 for cohesion to 0.721 for grammatical accuracy. These findings indicated an alignment between the assessments made by human raters and both the BERT-based and OCLL-based AES systems in terms of various aspects of writing proficiency.

Reliability of LLM-driven AES discriminating proficiency levels

After validating the reliability of the LLM’s annotation and scoring, the subsequent objective was to evaluate its ability to distinguish between various proficiency levels. For this analysis, a dataset of 686 individual essays was utilized. Table 10 presents a sample of the results, summarizing the means, standard deviations, and the outcomes of the one-way ANOVAs based on the measures assessed by the GPT-4 model. A post hoc multiple comparison test, specifically the Bonferroni test, was conducted to identify any potential differences between pairs of levels.

As the results reveal, seven measures presented linear upward or downward progress across the three proficiency levels. These were marked in bold in Table 10 and comprise one measure of lexical richness, i.e. MATTR (lexical diversity); four measures of syntactic complexity, i.e. MDD (mean dependency distance), MLC (mean length of clause), CNT (complex nominals per T-unit), CPC (coordinate phrases rate); one cohesion measure, i.e. word2vec cosine similarity and GER (grammatical error rate). Regarding the ability of the sixteen measures to distinguish adjacent proficiency levels, the Bonferroni tests indicated that statistically significant differences exist between the primary level and the intermediate level for MLC and GER. One measure of lexical richness, namely LD, along with three measures of syntactic complexity (VPT, CT, DCT, ACC), two measures of cohesion (SOPT, SOPK), and one measure of content elaboration (IMM), exhibited statistically significant differences between proficiency levels. However, these differences did not demonstrate a linear progression between adjacent proficiency levels. No significant difference was observed in lexical sophistication between proficiency levels.

To summarize, our study aimed to evaluate the reliability and differentiation capabilities of the LLM-driven AES method. For the first objective, we assessed the LLM’s ability to differentiate between test takers with varying levels of oral proficiency using precision, recall, F-Score, and quadratically-weighted kappa. Regarding the second objective, we compared the scoring outcomes generated by human raters and the LLM to determine the level of agreement. We employed quadratically-weighted kappa and Pearson correlations to compare the 16 writing proficiency measures for the individual essays. The results confirmed the feasibility of using the LLM for annotation and scoring in AES for nonnative Japanese. As a result, Research Question 1 has been addressed.

Comparison of BERT-, GPT-, OCLL-based AES, and linguistic-feature-based computation methods

This section aims to compare the effectiveness of five AES methods for nonnative Japanese writing, i.e. LLM-driven approaches utilizing BERT, GPT, and OCLL, linguistic feature-based approaches using Jess and JWriter. The comparison was conducted by comparing the ratings obtained from each approach with human ratings. All ratings were derived from the dataset introduced in Dataset . To facilitate the comparison, the agreement between the automated methods and human ratings was assessed using QWK and PRMSE. The performance of each approach was summarized in Table 11 .

The QWK coefficient values indicate that LLMs (GPT, BERT, OCLL) and human rating outcomes demonstrated higher agreement compared to feature-based AES methods (Jess and JWriter) in assessing writing proficiency criteria, including lexical richness, syntactic complexity, content, and grammatical accuracy. Among the LLMs, the GPT-4 driven AES and human rating outcomes showed the highest agreement in all criteria, except for syntactic complexity. The PRMSE values suggest that the GPT-based method outperformed linguistic feature-based methods and other LLM-based approaches. Moreover, an interesting finding emerged during the study: the agreement coefficient between GPT-4 and human scoring was even higher than the agreement between different human raters themselves. This discovery highlights the advantage of GPT-based AES over human rating. Ratings involve a series of processes, including reading the learners’ writing, evaluating the content and language, and assigning scores. Within this chain of processes, various biases can be introduced, stemming from factors such as rater biases, test design, and rating scales. These biases can impact the consistency and objectivity of human ratings. GPT-based AES may benefit from its ability to apply consistent and objective evaluation criteria. By prompting the GPT model with detailed writing scoring rubrics and linguistic features, potential biases in human ratings can be mitigated. The model follows a predefined set of guidelines and does not possess the same subjective biases that human raters may exhibit. This standardization in the evaluation process contributes to the higher agreement observed between GPT-4 and human scoring. Section Prompt strategy of the study delves further into the role of prompts in the application of LLMs to AES. It explores how the choice and implementation of prompts can impact the performance and reliability of LLM-based AES methods. Furthermore, it is important to acknowledge the strengths of the local model, i.e. the Japanese local model OCLL, which excels in processing certain idiomatic expressions. Nevertheless, our analysis indicated that GPT-4 surpasses local models in AES. This superior performance can be attributed to the larger parameter size of GPT-4, estimated to be between 500 billion and 1 trillion, which exceeds the sizes of both BERT and the local model OCLL.

Prompt strategy

In the context of prompt strategy, Mizumoto and Eguchi ( 2023 ) conducted a study where they applied the GPT-3 model to automatically score English essays in the TOEFL test. They found that the accuracy of the GPT model alone was moderate to fair. However, when they incorporated linguistic measures such as cohesion, syntactic complexity, and lexical features alongside the GPT model, the accuracy significantly improved. This highlights the importance of prompt engineering and providing the model with specific instructions to enhance its performance. In this study, a similar approach was taken to optimize the performance of LLMs. GPT-4, which outperformed BERT and OCLL, was selected as the candidate model. Model 1 was used as the baseline, representing GPT-4 without any additional prompting. Model 2, on the other hand, involved GPT-4 prompted with 16 measures that included scoring criteria, efficient linguistic features for writing assessment, and detailed measurement units and calculation formulas. The remaining models (Models 3 to 18) utilized GPT-4 prompted with individual measures. The performance of these 18 different models was assessed using the output indicators described in Section Criteria (output indicator) . By comparing the performances of these models, the study aimed to understand the impact of prompt engineering on the accuracy and effectiveness of GPT-4 in AES tasks.

Based on the PRMSE scores presented in Fig. 4 , it was observed that Model 1, representing GPT-4 without any additional prompting, achieved a fair level of performance. However, Model 2, which utilized GPT-4 prompted with all measures, outperformed all other models in terms of PRMSE score, achieving a score of 0.681. These results indicate that the inclusion of specific measures and prompts significantly enhanced the performance of GPT-4 in AES. Among the measures, syntactic complexity was found to play a particularly significant role in improving the accuracy of GPT-4 in assessing writing quality. Following that, lexical diversity emerged as another important factor contributing to the model’s effectiveness. The study suggests that a well-prompted GPT-4 can serve as a valuable tool to support human assessors in evaluating writing quality. By utilizing GPT-4 as an automated scoring tool, the evaluation biases associated with human raters can be minimized. This has the potential to empower teachers by allowing them to focus on designing writing tasks and guiding writing strategies, while leveraging the capabilities of GPT-4 for efficient and reliable scoring.

PRMSE scores of the 18 AES models.

This study aimed to investigate two main research questions: the feasibility of utilizing LLMs for AES and the impact of prompt engineering on the application of LLMs in AES.

To address the first objective, the study compared the effectiveness of five different models: GPT, BERT, the Japanese local LLM (OCLL), and two conventional machine learning-based AES tools (Jess and JWriter). The PRMSE values indicated that the GPT-4-based method outperformed other LLMs (BERT, OCLL) and linguistic feature-based computational methods (Jess and JWriter) across various writing proficiency criteria. Furthermore, the agreement coefficient between GPT-4 and human scoring surpassed the agreement among human raters themselves, highlighting the potential of using the GPT-4 tool to enhance AES by reducing biases and subjectivity, saving time, labor, and cost, and providing valuable feedback for self-study. Regarding the second goal, the role of prompt design was investigated by comparing 18 models, including a baseline model, a model prompted with all measures, and 16 models prompted with one measure at a time. GPT-4, which outperformed BERT and OCLL, was selected as the candidate model. The PRMSE scores of the models showed that GPT-4 prompted with all measures achieved the best performance, surpassing the baseline and other models.

In conclusion, this study has demonstrated the potential of LLMs in supporting human rating in assessments. By incorporating automation, we can save time and resources while reducing biases and subjectivity inherent in human rating processes. Automated language assessments offer the advantage of accessibility, providing equal opportunities and economic feasibility for individuals who lack access to traditional assessment centers or necessary resources. LLM-based language assessments provide valuable feedback and support to learners, aiding in the enhancement of their language proficiency and the achievement of their goals. This personalized feedback can cater to individual learner needs, facilitating a more tailored and effective language-learning experience.

There are three important areas that merit further exploration. First, prompt engineering requires attention to ensure optimal performance of LLM-based AES across different language types. This study revealed that GPT-4, when prompted with all measures, outperformed models prompted with fewer measures. Therefore, investigating and refining prompt strategies can enhance the effectiveness of LLMs in automated language assessments. Second, it is crucial to explore the application of LLMs in second-language assessment and learning for oral proficiency, as well as their potential in under-resourced languages. Recent advancements in self-supervised machine learning techniques have significantly improved automatic speech recognition (ASR) systems, opening up new possibilities for creating reliable ASR systems, particularly for under-resourced languages with limited data. However, challenges persist in the field of ASR. First, ASR assumes correct word pronunciation for automatic pronunciation evaluation, which proves challenging for learners in the early stages of language acquisition due to diverse accents influenced by their native languages. Accurately segmenting short words becomes problematic in such cases. Second, developing precise audio-text transcriptions for languages with non-native accented speech poses a formidable task. Last, assessing oral proficiency levels involves capturing various linguistic features, including fluency, pronunciation, accuracy, and complexity, which are not easily captured by current NLP technology.

Data availability

The dataset utilized was obtained from the International Corpus of Japanese as a Second Language (I-JAS). The data URLs: [ https://www2.ninjal.ac.jp/jll/lsaj/ihome2.html ].

J-CAT and TTBJ are two computerized adaptive tests used to assess Japanese language proficiency.

SPOT is a specific component of the TTBJ test.

J-CAT: https://www.j-cat2.org/html/ja/pages/interpret.html

SPOT: https://ttbj.cegloc.tsukuba.ac.jp/p1.html#SPOT .

The study utilized a prompt-based GPT-4 model, developed by OpenAI, which has an impressive architecture with 1.8 trillion parameters across 120 layers. GPT-4 was trained on a vast dataset of 13 trillion tokens, using two stages: initial training on internet text datasets to predict the next token, and subsequent fine-tuning through reinforcement learning from human feedback.

https://www2.ninjal.ac.jp/jll/lsaj/ihome2-en.html .

http://jhlee.sakura.ne.jp/JEV/ by Japanese Learning Dictionary Support Group 2015.

We express our sincere gratitude to the reviewer for bringing this matter to our attention.

On February 7, 2023, Microsoft began rolling out a major overhaul to Bing that included a new chatbot feature based on OpenAI’s GPT-4 (Bing.com).

Appendix E-F present the analysis results of the QWK coefficient between the scores computed by the human raters and the BERT, OCLL models.

Attali Y, Burstein J (2006) Automated essay scoring with e-rater® V.2. J. Technol., Learn. Assess., 4

Barkaoui K, Hadidi A (2020) Assessing Change in English Second Language Writing Performance (1st ed.). Routledge, New York. https://doi.org/10.4324/9781003092346

Bentz C, Tatyana R, Koplenig A, Tanja S (2016) A comparison between morphological complexity. measures: Typological data vs. language corpora. In Proceedings of the workshop on computational linguistics for linguistic complexity (CL4LC), 142–153. Osaka, Japan: The COLING 2016 Organizing Committee

Bond TG, Yan Z, Heene M (2021) Applying the Rasch model: Fundamental measurement in the human sciences (4th ed). Routledge

Brants T (2000) Inter-annotator agreement for a German newspaper corpus. Proceedings of the Second International Conference on Language Resources and Evaluation (LREC’00), Athens, Greece, 31 May-2 June, European Language Resources Association

Brown TB, Mann B, Ryder N, et al. (2020) Language models are few-shot learners. Advances in Neural Information Processing Systems, Online, 6–12 December, Curran Associates, Inc., Red Hook, NY

Burstein J (2003) The E-rater scoring engine: Automated essay scoring with natural language processing. In Shermis MD and Burstein JC (ed) Automated Essay Scoring: A Cross-Disciplinary Perspective. Lawrence Erlbaum Associates, Mahwah, NJ

Čech R, Miroslav K (2018) Morphological richness of text. In Masako F, Václav C (ed) Taming the corpus: From inflection and lexis to interpretation, 63–77. Cham, Switzerland: Springer Nature

Çöltekin Ç, Taraka, R (2018) Exploiting Universal Dependencies treebanks for measuring morphosyntactic complexity. In Aleksandrs B, Christian B (ed), Proceedings of first workshop on measuring language complexity, 1–7. Torun, Poland

Crossley SA, Cobb T, McNamara DS (2013) Comparing count-based and band-based indices of word frequency: Implications for active vocabulary research and pedagogical applications. System 41:965–981. https://doi.org/10.1016/j.system.2013.08.002

Article   Google Scholar  

Crossley SA, McNamara DS (2016) Say more and be more coherent: How text elaboration and cohesion can increase writing quality. J. Writ. Res. 7:351–370

CyberAgent Inc (2023) Open-Calm series of Japanese language models. Retrieved from: https://www.cyberagent.co.jp/news/detail/id=28817

Devlin J, Chang MW, Lee K, Toutanova K (2019) BERT: Pre-training of deep bidirectional transformers for language understanding. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics, Minneapolis, Minnesota, 2–7 June, pp. 4171–4186. Association for Computational Linguistics

Diez-Ortega M, Kyle K (2023) Measuring the development of lexical richness of L2 Spanish: a longitudinal learner corpus study. Studies in Second Language Acquisition 1-31

Eckes T (2009) On common ground? How raters perceive scoring criteria in oral proficiency testing. In Brown A, Hill K (ed) Language testing and evaluation 13: Tasks and criteria in performance assessment (pp. 43–73). Peter Lang Publishing

Elliot S (2003) IntelliMetric: from here to validity. In: Shermis MD, Burstein JC (ed) Automated Essay Scoring: A Cross-Disciplinary Perspective. Lawrence Erlbaum Associates, Mahwah, NJ

Google Scholar  

Engber CA (1995) The relationship of lexical proficiency to the quality of ESL compositions. J. Second Lang. Writ. 4:139–155

Garner J, Crossley SA, Kyle K (2019) N-gram measures and L2 writing proficiency. System 80:176–187. https://doi.org/10.1016/j.system.2018.12.001

Haberman SJ (2008) When can subscores have value? J. Educat. Behav. Stat., 33:204–229

Haberman SJ, Yao L, Sinharay S (2015) Prediction of true test scores from observed item scores and ancillary data. Brit. J. Math. Stat. Psychol. 68:363–385

Halliday MAK (1985) Spoken and Written Language. Deakin University Press, Melbourne, Australia

Hirao R, Arai M, Shimanaka H et al. (2020) Automated essay scoring system for nonnative Japanese learners. Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), pp. 1250–1257. European Language Resources Association

Hunt KW (1966) Recent Measures in Syntactic Development. Elementary English, 43(7), 732–739. http://www.jstor.org/stable/41386067

Ishioka T (2001) About e-rater, a computer-based automatic scoring system for essays [Konpyūta ni yoru essei no jidō saiten shisutemu e − rater ni tsuite]. University Entrance Examination. Forum [Daigaku nyūshi fōramu] 24:71–76

Hochreiter S, Schmidhuber J (1997) Long short- term memory. Neural Comput. 9(8):1735–1780

Article   CAS   PubMed   Google Scholar  

Ishioka T, Kameda M (2006) Automated Japanese essay scoring system based on articles written by experts. Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics, Sydney, Australia, 17–18 July 2006, pp. 233-240. Association for Computational Linguistics, USA

Japan Foundation (2021) Retrieved from: https://www.jpf.gp.jp/j/project/japanese/survey/result/dl/survey2021/all.pdf

Jarvis S (2013a) Defining and measuring lexical diversity. In Jarvis S, Daller M (ed) Vocabulary knowledge: Human ratings and automated measures (Vol. 47, pp. 13–44). John Benjamins. https://doi.org/10.1075/sibil.47.03ch1

Jarvis S (2013b) Capturing the diversity in lexical diversity. Lang. Learn. 63:87–106. https://doi.org/10.1111/j.1467-9922.2012.00739.x

Jiang J, Quyang J, Liu H (2019) Interlanguage: A perspective of quantitative linguistic typology. Lang. Sci. 74:85–97

Kim M, Crossley SA, Kyle K (2018) Lexical sophistication as a multidimensional phenomenon: Relations to second language lexical proficiency, development, and writing quality. Mod. Lang. J. 102(1):120–141. https://doi.org/10.1111/modl.12447

Kojima T, Gu S, Reid M et al. (2022) Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems, New Orleans, LA, 29 November-1 December, Curran Associates, Inc., Red Hook, NY

Kyle K, Crossley SA (2015) Automatically assessing lexical sophistication: Indices, tools, findings, and application. TESOL Q 49:757–786

Kyle K, Crossley SA, Berger CM (2018) The tool for the automatic analysis of lexical sophistication (TAALES): Version 2.0. Behav. Res. Methods 50:1030–1046. https://doi.org/10.3758/s13428-017-0924-4

Article   PubMed   Google Scholar  

Kyle K, Crossley SA, Jarvis S (2021) Assessing the validity of lexical diversity using direct judgements. Lang. Assess. Q. 18:154–170. https://doi.org/10.1080/15434303.2020.1844205

Landauer TK, Laham D, Foltz PW (2003) Automated essay scoring and annotation of essays with the Intelligent Essay Assessor. In Shermis MD, Burstein JC (ed), Automated Essay Scoring: A Cross-Disciplinary Perspective. Lawrence Erlbaum Associates, Mahwah, NJ

Landis JR, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 159–174

Laufer B, Nation P (1995) Vocabulary size and use: Lexical richness in L2 written production. Appl. Linguist. 16:307–322. https://doi.org/10.1093/applin/16.3.307

Lee J, Hasebe Y (2017) jWriter Learner Text Evaluator, URL: https://jreadability.net/jwriter/

Lee J, Kobayashi N, Sakai T, Sakota K (2015) A Comparison of SPOT and J-CAT Based on Test Analysis [Tesuto bunseki ni motozuku ‘SPOT’ to ‘J-CAT’ no hikaku]. Research on the Acquisition of Second Language Japanese [Dainigengo to shite no nihongo no shūtoku kenkyū] (18) 53–69

Li W, Yan J (2021) Probability distribution of dependency distance based on a Treebank of. Japanese EFL Learners’ Interlanguage. J. Quant. Linguist. 28(2):172–186. https://doi.org/10.1080/09296174.2020.1754611

Article   MathSciNet   Google Scholar  

Linacre JM (2002) Optimizing rating scale category effectiveness. J. Appl. Meas. 3(1):85–106

PubMed   Google Scholar  

Linacre JM (1994) Constructing measurement with a Many-Facet Rasch Model. In Wilson M (ed) Objective measurement: Theory into practice, Volume 2 (pp. 129–144). Norwood, NJ: Ablex

Liu H (2008) Dependency distance as a metric of language comprehension difficulty. J. Cognitive Sci. 9:159–191

Liu H, Xu C, Liang J (2017) Dependency distance: A new perspective on syntactic patterns in natural languages. Phys. Life Rev. 21. https://doi.org/10.1016/j.plrev.2017.03.002

Loukina A, Madnani N, Cahill A, et al. (2020) Using PRMSE to evaluate automated scoring systems in the presence of label noise. Proceedings of the Fifteenth Workshop on Innovative Use of NLP for Building Educational Applications, Seattle, WA, USA → Online, 10 July, pp. 18–29. Association for Computational Linguistics

Lu X (2010) Automatic analysis of syntactic complexity in second language writing. Int. J. Corpus Linguist. 15:474–496

Lu X (2012) The relationship of lexical richness to the quality of ESL learners’ oral narratives. Mod. Lang. J. 96:190–208

Lu X (2017) Automated measurement of syntactic complexity in corpus-based L2 writing research and implications for writing assessment. Lang. Test. 34:493–511

Lu X, Hu R (2022) Sense-aware lexical sophistication indices and their relationship to second language writing quality. Behav. Res. Method. 54:1444–1460. https://doi.org/10.3758/s13428-021-01675-6

Ministry of Health, Labor, and Welfare of Japan (2022) Retrieved from: https://www.mhlw.go.jp/stf/newpage_30367.html

Mizumoto A, Eguchi M (2023) Exploring the potential of using an AI language model for automated essay scoring. Res. Methods Appl. Linguist. 3:100050

Okgetheng B, Takeuchi K (2024) Estimating Japanese Essay Grading Scores with Large Language Models. Proceedings of 30th Annual Conference of the Language Processing Society in Japan, March 2024

Ortega L (2015) Second language learning explained? SLA across 10 contemporary theories. In VanPatten B, Williams J (ed) Theories in Second Language Acquisition: An Introduction

Rae JW, Borgeaud S, Cai T, et al. (2021) Scaling Language Models: Methods, Analysis & Insights from Training Gopher. ArXiv, abs/2112.11446

Read J (2000) Assessing vocabulary. Cambridge University Press. https://doi.org/10.1017/CBO9780511732942

Rudner LM, Liang T (2002) Automated Essay Scoring Using Bayes’ Theorem. J. Technol., Learning and Assessment, 1 (2)

Sakoda K, Hosoi Y (2020) Accuracy and complexity of Japanese Language usage by SLA learners in different learning environments based on the analysis of I-JAS, a learners’ corpus of Japanese as L2. Math. Linguist. 32(7):403–418. https://doi.org/10.24701/mathling.32.7_403

Suzuki N (1999) Summary of survey results regarding comprehensive essay questions. Final report of “Joint Research on Comprehensive Examinations for the Aim of Evaluating Applicability to Each Specialized Field of Universities” for 1996-2000 [shōronbun sōgō mondai ni kansuru chōsa kekka no gaiyō. Heisei 8 - Heisei 12-nendo daigaku no kaku senmon bun’ya e no tekisei no hyōka o mokuteki to suru sōgō shiken no arikata ni kansuru kyōdō kenkyū’ saishū hōkoku-sho]. University Entrance Examination Section Center Research and Development Department [Daigaku nyūshi sentā kenkyū kaihatsubu], 21–32

Taghipour K, Ng HT (2016) A neural approach to automated essay scoring. Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, 1–5 November, pp. 1882–1891. Association for Computational Linguistics

Takeuchi K, Ohno M, Motojin K, Taguchi M, Inada Y, Iizuka M, Abo T, Ueda H (2021) Development of essay scoring methods based on reference texts with construction of research-available Japanese essay data. In IPSJ J 62(9):1586–1604

Ure J (1971) Lexical density: A computational technique and some findings. In Coultard M (ed) Talking about Text. English Language Research, University of Birmingham, Birmingham, England

Vaswani A, Shazeer N, Parmar N, et al. (2017) Attention is all you need. In Advances in Neural Information Processing Systems, Long Beach, CA, 4–7 December, pp. 5998–6008, Curran Associates, Inc., Red Hook, NY

Watanabe H, Taira Y, Inoue Y (1988) Analysis of essay evaluation data [Shōronbun hyōka dēta no kaiseki]. Bulletin of the Faculty of Education, University of Tokyo [Tōkyōdaigaku kyōiku gakubu kiyō], Vol. 28, 143–164

Yao S, Yu D, Zhao J, et al. (2023) Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36

Zenker F, Kyle K (2021) Investigating minimum text lengths for lexical diversity indices. Assess. Writ. 47:100505. https://doi.org/10.1016/j.asw.2020.100505

Zhang Y, Warstadt A, Li X, et al. (2021) When do you need billions of words of pretraining data? Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Online, pp. 1112-1125. Association for Computational Linguistics. https://doi.org/10.18653/v1/2021.acl-long.90

Download references

This research was funded by National Foundation of Social Sciences (22BYY186) to Wenchao Li.

Author information

Authors and affiliations.

Department of Japanese Studies, Zhejiang University, Hangzhou, China

Department of Linguistics and Applied Linguistics, Zhejiang University, Hangzhou, China

You can also search for this author in PubMed   Google Scholar

Contributions

Wenchao Li is in charge of conceptualization, validation, formal analysis, investigation, data curation, visualization and writing the draft. Haitao Liu is in charge of supervision.

Corresponding author

Correspondence to Wenchao Li .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Ethical approval

Ethical approval was not required as the study did not involve human participants.

Informed consent

This article does not contain any studies with human participants performed by any of the authors.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary information

Supplemental material file #1, rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Li, W., Liu, H. Applying large language models for automated essay scoring for non-native Japanese. Humanit Soc Sci Commun 11 , 723 (2024). https://doi.org/10.1057/s41599-024-03209-9

Download citation

Received : 02 February 2024

Accepted : 16 May 2024

Published : 03 June 2024

DOI : https://doi.org/10.1057/s41599-024-03209-9

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies