Event Summary – Linguistics in the Pub panel (online experiments)

I recently spoke at a panel session for Linguistics in the Pub with my friends (who are also colleagues) Chloé Diskin-Holdaway and Olga Maxwell (pictured above). The three of us as a team have published some research about Indian English and about Australian English (Chloé and Debbie, with Penelope Schmidt).

Details of the Linguistics in the Pub event were publicised as follows:

Melbourne Linguistics in the Pub 9th November 2021: online linguistics research: stories from the ‘field’

During the pandemic much face-to-face research on language has been halted and many researchers have found ways to do their research online instead. In this session of Linguistics in the Pub, three linguists will share their experiences of doing research online in the areas of sociolinguistics, language planning, forensic linguistics, phonetics and psycholinguistics.

The session, held in Naughtons on Royal Parade near The University of Melbourne campus, attracted about 20 guests, ranging from undergraduate students to academic colleagues, and some members of the public. Linguistics in the Pub is something of an institution among the Melbourne linguistics community and has been running for over ten years. We also have a previous blog post about this. Chloé, Debbie and Olga spoke about our experiences collecting data online since the pandemic began, and pointed out some of the main benefits associated with this kind of research. Amongst the three of us, we covered survey data, perception experiments, and ways of collecting speech production data online – this has certainly been one of our activities in the Hub since we started in 2020!

Chloé started us off by talking about some online studies she has been doing. One is a project about language maintenance and bilingualism which she has written about here. This survey is still open, you can access it here. She also talked about a survey on language attitudes and identity, available here. Until the end of Dec 2021, you can also watch Chloé’s five minute talk at the AusEng Forum about early results relating to this project (but if you’re planning on taking part in her experiment, do that first!).  Finally, Chloé also talked about an online experiment for children aged 3-12 to share stories and experiences about life in lockdown. The “Bear in a Window” project is still running, and can be accessed here. You can also learn more about the Bear in a Window project in an interview with Chloé here.

I spoke next, starting with two studies in the Research Hub for Language in Forensic Evidence. I spoke about Hub Honours student Conor Clements’ research project as well as a study that Helen and I are currently running about the process of forensic transcription. In this experiment, we are asking listeners to transcribe some indistinct forensic-like audio. We have had great interest in our “warm-up” experiment already, and have had to cap numbers because we are still working out how to score transcripts (to compare “performance”). We are working with Ute Knoch from the Language Testing Research Centre at The University of Melbourne, so we are in good hands in terms of working out this process! I also spoke about some other work I am doing with WSU colleagues Anne Cutler and Laurence Bruggeman. We had originally planned to do this experiment using eye-tracking, but when the pandemic hit we moved this online, and WSU colleagues were able to design a fully functioning psycholinguistic experiment which tracks people’s responses to audio data – this is done via Zoom, “in person” (virtually). Specifically, this study involves cross-modal form priming with auditory primes and visual lexical decision targets. It is based on Experiment 1 from Weber, Di Betta, & McQueen (2014). We are still looking for participants (born, raised and living in Melbourne) for this experiment, so please get in touch with Debbie to take part!

Olga spoke next about her experiences with the transition to fully online data collection during the COVID-19 pandemic. She reported on two concurrently run projects involving collecting perception and production data in Australia, the UK, India and Germany. The first project is a collaboration with the colleagues at the University of Melbourne (including me!) and the University of Oxford (Elinor Payne). The study examines Indian English spoken in two diverse diaspora locations, Melbourne, Australia and Oxford, the UK, both with large diaspora communities. It investigates how recently-arrived speakers of Indian English adapt to two diverse linguistic contexts, which phonetic features they retain signalling their ‘Indian identity’, and which features of the local dialect they adopt. Olga talked about how speech production data was collected using participants’ mobile phones (more on this below). The second project is work with the collaborators at the Universities of Oxford (again with Elinor Payne) and Hamburg (Robert Fuchs), focusing on the perception and production of lexical stress.

Chloé, Olga and I all spoke about the fact that research experiments online can be:

• unsupervised (i.e. participants follow a link and complete the experiment). This works well for us in the Hub where people are listening to speech and responding, and also works well for surveys. It is also a very efficient way of collecting data, because the responses can just roll in! Conor was able to collect data from 152 people for his Honours thesis, and Chloé’s two online surveys have attracted over 1,100 respondents to date.
• supervised / semi-supervised – where the researcher needs to play some kind of role in the data collection. This works well for psycholinguistic experiments, such as the work I am doing with WSU colleagues. We need to explain the instructions and be present for the “practice” experiment (it is somewhat difficult), and we also monitor participants closely and watch for distractions – normally this would be easy to do in a lab so it is a way of replicating lab conditions to some degree. Supervised research is also needed for the collection of speech data. Note that this type of online data collection can be extremely time-consuming (when compared with unsupervised research) but as Chloé pointed out, it is still really beneficial for participants who do not need to come to campus. Olga found that when she collected speech data, participants needed to be given a set of step-by-step guidelines detailing how to make, save and share an audio recording on a phone. She added that having research assistance was crucial for collecting production data, more so than for perception experiments. Olga and her collaborators overseas (Elinor and Robert) were able to collect over 300 recordings: the process of saving and sorting the recordings was managed by the research assistant in each location.

I mentioned that for listening experiments, it is possible to use a headphone screener. We are using one for the WSU experiment (you can read about these here in an article by Woods et al. 2017). Helen and I also choose to ask participants about their headphone use, and for comments on the type of headphones, and we check people’s audio is working with a test devised by Helen that asks participants to report which colour term they hear at certain intervals in a recording.

I have also been pleasantly surprised to learn how easy it is to collect good quality recordings online. Olga reported very good experiences with this. There is also some other work you can read about, for example completely unsupervised recordings captured on mobile phones in the UK for the English Dialects App (Leeman, Kolly & Britain 2019), and Stephen Bird and colleagues (2014) have written about the use of mobile phones for collaborative language documentation purposes.

Finally, it is also important to be aware of the quality of data collected online when working with speech data. While it is generally very good, this will depend on the purpose. There are some articles that have been published recently about this issue, for example Penney and colleagues (2021) have published a “cautionary tale” about acoustic correlates of voice quality across different recording systems. Leeman and colleagues (2020) have also published an article about the app mentioned above, and extending its use during the pandemic via supervised research over Zoom – the article also includes a section on benefits and pitfalls of online data collection, and really importantly, a survey about participant experiences. A group of early career researchers around the world have been coming together to discuss similar challenges, check out their website for some interesting blog posts and interviews.

Some things we recommend for online experiments are:

• Piloting a lot, with small groups of participants who can feed back about their experiences. For our most recent experiment, Helen and I had some really helpful feedback about things such as the size of the audio player, instructions about the use of key commands (because some people prefer not to use a mouse), and we also had one pilot participant who lost an entire transcript due to loss of an internet connection – so we were able to put some checks and balances in place to stop that happening. Olga, Elinor and Debbie also trialled multiple versions of their perception experiment in order to finesse the instructions for the trainer and the main task.
• “on the ground” research assistants can also be very useful – Olga Maxwell spoke about the fact that for data collection to go ahead in locations outside Australia, research assistants who were physically in those locations were able to recruit participants who met the selection criteria and subsequently liaise with participants (by email or phone) and, if needed, respond to questions about the tasks.
• software designed for experimental research – the three of us use various software such Pavlovia, an open source online platform, and Qualtrics which has various options for online experiments. We have been using Qualtrics for our Hub listening experiments, and Chloé and Olga have also used it for surveys and background questionnaires.
• Be mindful about the number of participants – it is important for researchers to decide if they want a link to an unsupervised experiment shared widely online, or whether participants should be sent individualised links so the data collection process can be closely managed. We have used both options in the Hub, and it can be highly dependent on the task. Qualtrics allows us to input a “contacts” list, and it then generates personalised links which can be emailed out.
• Social media (Facebook, Twitter, Instagram) can be very useful for helping us find participants online, and Chloé also found that the opportunity to speak about her research on the radio also helped generate interest from participants.
• The use of comment boxes – this has been really useful for Hub experiments. It is really important to use these regularly throughout online experiments (when carrying out “unsupervised” research) so participants can give feedback. I have recently taken part in some online experiments where I could not feed back important information to the researchers, and the experiment ended without an option to do so. This was frustrating for me, but also I think the researchers missed out on hearing valuable detail, and in one case there was a glitch that I could not report.
• Be mindful of security – in our forensic linguistic experiments, Helen and I have had to manage security of audio files. Sometimes the “default” option is not the most appropriate.

In summary, Chloé, Olga and I really recommend using online methods as a way of gathering data, although of course this will not work for every experiment nor for every community. Some things will obviously work better in person. We also made a little joke about how it is not possible to collect articulatory data (such as ultrasound) online, and I am certainly disappointed about not being able to use the eye-tracking software for my WSU collaboration! Nevertheless, we all agreed we are lucky to have been able to collect data online – this was so beneficial to our research since the pandemic started.

References

Bird, S., F. Hanke, O. Adams and H. Lee. (2014). Aikuma: A mobile app for collaborative language documentation. In Proceedings of the Workshop on the Use of Computational Methods in the Study of Endangered Languages, pages 1–5, Baltimore, MD. https://doi.org/10.3115/v1/W14-2201

Gawne, L. and R. Singer (2021). “Ten years of Linguistics in the Pub” Australian Journal of Linguistics. Published online October 21. https://doi.org/10.1080/07268602.2021.1974341

Leeman, A., M-J. Kolly and D. Britain (2018). The English Dialects App: The creation of a crowdsourced dialect corpus. Ampersand, 5, 1-17. https://doi.org/10.1016/j.amper.2017.11.001

Leemann, A., P. Jeszenszky, C. Steiner, M. Studerus, and J. Messerli, (2020). “Linguistic fieldwork in a pandemic: Supervised data collection combining smartphone recordings and videoconferencing” Linguistics Vanguard, vol. 6, no. s3, 2020, pp. 20200061. https://doi.org/10.1515/lingvan-2020-0061

Maxwell, O., C. Diskin-Holdaway, and D. Loakes. (2021). “Attitudes towards Indian English among young urban professionals in Hyderabad, India.” World Englishes. doi: https://doi.org/10.1111/weng.12550

Penney, J., Gibson, A., Cox, F., Proctor, M., Szakay, A. (2021) A Comparison of Acoustic Correlates of Voice Quality Across Different Recording Devices: A Cautionary Tale. Proc. Interspeech 2021, 1389-1393, doi: 10.21437/Interspeech.2021-729

Schmidt, P., C. Diskin-Holdaway and D. Loakes. (2021). “New insights into /el/-/æl/ merging in Australian English.” Australian Journal of Linguistics. 41 (2): 66-95 https://doi.org/10.1080/07268602.2021.1905607

Weber, A., Di Betta, A. M., & McQueen, J. M. (2014). Treack or trit: Adaptation to genuine and arbitrary foreign accents by monolingual and bilingual listeners. Journal of Phonetics, 46, 34-51. doi: 10.1016/j.wocn.2014.05.002

Woods, K. J. P., Siegel, M. H., Traer, J., & McDermott, J. H. (2017). Headphone screening to facilitate web-based auditory experiments. Attention, Perception, & Psychophysics, 79(7), 2064-2072. doi: 10.3758/s13414-017-1361-2