Live virtual music experiences using emerging technologies with Sally Coleman

Michele Ong

The world of musical and creative expression is continuously evolving, and there are more ways to create and engage with music and audiences than ever before. It all starts with an enquiring mind and a desire to explore new frontiers.

Sally Coleman is a musician and creative director of the science-fiction virtual band, Big Sand. Join us as we speak with Sally about her journey to developing Big Sand, creating engaging virtual music experiences, and the role of emerging technologies in the creative industries.

I'm Michele Ong, and this is STEAM Powered.

Michele Ong

Good afternoon, Sally. Thank you so much for joining me on STEAM Powered. I'm really looking forward to speaking with you about your journey.

Sally Coleman

No worries. It's nice to be here. Hey Michele.

Sally's beginnings with communication studies.

Michele Ong

Hey. So you know, we'll kind of look back a little bit. Rewind. And you started out studying Bachelor of Comms with writing, culture studies, and film studies, but what made you pursue comms?

Sally Coleman

Gosh, I don't know. I guess I knew I wanted to go to university, I know not everyone does. But really I was like, oh, I like creative writing, I wanna do creative stuff, and kind of was really just looking around for a creative writing course.

UTS at the time had a really great creative writing course, but they'd recently combined it with this cultural studies degree.

I was like, oh, I don't know what that is. I'm not sure I wanna do it. But I've heard really good things about going to UTS and studying there. So kind of sucked it up, applied for that as one of my first choices, and yeah. This is a long time ago. What did I do after high school?

But I really enjoyed the blend of the creative side, but also cultural studies, which was this whole way of looking at the world and understanding how meaning gets made and how do we decide on what's true, and who's in power, and how does communication play a role in all of those things.

So–

Michele Ong

Big questions.

Sally Coleman

Yeah. Big questions, like a lot of philosophy for a little 19-year-old to kind of jump into.

Music has always been part of Sally's creative life.

Michele Ong

So I guess, because you're thinking about all these sorts of things, how did that transition, or was there a relationship between that and you wanting to create music?

Sally Coleman

Honestly, music was something that I'd already been doing since I was a kid. Yeah, I wouldn't say– I'm not like a virtuoso by any means. But yeah, I had played instruments throughout my childhood and then it was probably around when I started university that I started at the same time, like a hip-hop duo.

I just really loved Australian hip-hop. Had a friend who also loved Australian hip-hop and we just started doing little rap shows and I would travel back to Canberra to do open mic shows. So it was kind of happening side-by-side with my creative writing kind of degree, but they definitely informed each other. And I do remember like for a couple things at uni, handing in like raps and things crossing over. But music, yeah, it's just, I guess always been part of my life. And that led me to kind of, yeah, start a band around that same time.

Evolving within the music industry.

Michele Ong

That's cool. So was there any particular moment where you go, well actually I'm just gonna go all in on this music thing. This is a thing that I will do now.

Sally Coleman

It wasn't really a moment. It was just like, it just kept going. And I think more to the point, in a good way, it kept growing, as in it started out very much as a hobby and something that we would do for fun. And then you know after a couple of years, we were like, our song got played on the radio, amazing. And then it was like, oh, we got asked to play this kind of, you know, real deal festival.

And that kept growing 'til we got a manager, and a record deal, and a song on rotation. And then we were touring with Listen Out Festival. And it just really organically grew over a long period of time and, not gonna call it sunk-cost fallacy because it wasn't sunk-cost in this instance, but it was like, we spent nearly a decade, I reckon, on that project, just building it from grassroots up.

And–

Michele Ong

That entire time it's just skill development and you're learning so much about the process and the industry in a very hands-on way.

Sally Coleman

Yeah, totally. So I think the thought of stopping just never really occurred to me. Like I thought, oh, I really love the music industry. I started working at a record label, doing a bunch more stuff in that space as well, sort of outside of the band, broadcasting on radio, like, I just got more and more involved in that industry and really loved the people in it and the way that you got to connect with audiences.

Michele Ong

Yeah, and that's a cool way of approaching it as well, because you're able to learn and experience different aspects of the industry because I know there are a few people out there who go, ah, I'm a musician, I'm just gonna create music. This is the thing. I'm gonna stick to that. But there are a lot of other people, like you, where you know, they'll be working in other areas of production, or process, or media getting more experience in that space while they're also doing all the creation stuff, which, you know, I think it gives you a much broader base and understanding of the space and what you can achieve.

Competition, exploitation, and compromise when the lines are blurred between passion and occupation.

Sally Coleman

Yeah, it definitely does. And there's like pros and cons, I guess, with the arts industry because so many arts workers, and by that I mean the people behind the scenes who make things happen you know, on a bigger scale, systemic level, a lot of those people are artists themselves.

So sometimes that's really good because you have this inherent understanding with people in those roles, but at the same time, it can sometimes lead to situations where there's a lot of very low-paid volunteer roles or entry-level roles that, I'm not gonna say take advantage, but there's a lot of competition for those roles despite the pay because people love what they do in the artistic side.

And I think that's something sometimes all arts industries struggle with is just 'cause you love what you do doesn't mean that you don't need to get paid fairly for it. So I think the older I get and the more I'm like, ah, I wish there was a way to address this.

Michele Ong

Yeah.

Sally Coleman

Yeah loving what you do is awesome, and it does give you that really rounded approach, but it puts the arts in this difficult space sometimes of being people's passion project and their paid project, and there's occasionally like conflicts of interests that come along there.

And it's a little bit different, I think, to some other industries, you know, in STEM or STEAM, where there might be more of a distinction between work and play or there's different expectations of what you get out of a workplace.

Michele Ong

Absolutely. There's always gonna be a little bit of, I guess, uh, let's not, you know, be too delicate about this– exploitation because yeah, it, it's a kind of industry where people will 'cause, even in other aspects of STEM, a lot of it is driven by passion as well. And people, even if they're not creating in the same way, are still creating.

And that passion will drive them to want to get that work out there and they will compromise on compensation, whether that's in terms of monetary value or in kind or whatever it is, but it does lend itself, unfortunately to people undervaluing what they do.

Sally Coleman

Yeah, and I don't wanna make out like there's some nasty person at the top who's like taking advantage of arts workers because it's more just an industry-wide thing. Everyone's so passionate about it that they're willing to do it for low pay or poor hours or things like that. That, that ends up being industry standard.

And I, I think everyone across the industry needs to be like, how are we gonna navigate this ourselves?

Michele Ong

Yeah, and there's a lot of pushback as well now too, which is good 'cause there are a lot of people saying, actually, I do want to be paid better for this. I would like to make a living wage while being able to do these things. And, you know, it's slow progress, but we are seeing a lot more change in the way that people are approaching this.

And actually not just calling out, but at least raising awareness of this should not be the way that we are compensating our artists or our creators, or the people who are contributing to our projects, and that's a good thing.

Remembering that you're a business as well.

Sally Coleman

I think one area that I found has really been helpful for my mindset is starting to apply for arts grants and things like that to do projects and whether you get the grant or not, they really encourage you to be like, what are you paying yourself? Like, how are you doing? Like, how are you living when you do this project? And formalising that as part of your budget, as like your actual personal expenses has like kind of helped just shift my mentality.

And then it does also make it easier when you're getting hired and in a freelance capacity to have those conversations as well, I think.

Michele Ong

Yeah, and that's a really great perspective to have on that too, because you are a business and you are running your own business and you need to approach it that way. And I think a lot of people forget that in the midst of the passion of it, that this is your job and you have to treat it in the same way as you would if you were running a conventional, 'a conventional' in quotes, business where you're a startup or, you know, you're a bricks and mortar store, whatever it is, you still have to pay yourself and you still have to figure out what your budget's gonna be and where are you gonna spend things like stuff for promotion, or service or all that kind of stuff.

Sally Coleman

I think it becomes tricky when it does start as a hobby and like at what point does that transition happen where it is a business. Because in some ways it didn't start as a business. It started as something like you did on your weekends or in evenings, and then it spirals and you're like, okay, I need to make that mental shift of if I'm giving up other work for this, then how do I, you know, do that.

So that's always been something that I've kind of struggled with and had to learn to do.

Michele Ong

Ah, good times. Because you know, no one wants to deal with bookkeeping and all that.

Exploring virtualised bands and alternative methods of audience engagement.

Michele Ong

So you've been creating music for 10 years or so. What made you decide to create a virtual band with motion capture and changing the way that you approach creation and engagement with the audience?

Sally Coleman

Well, I think it was one of those, like, 2020 —big year— everyone had a lot of like, what am I doing with my life?

I got to the point where I was like, look, I wanna start a new music project. I wanna take everything I've learned over the last, you know, however long, nearly a decade, start something new. But I don't just wanna jump through all the same hoops again. I don't just wanna go back and reinvent the wheel. I just wanna try something that's new but still takes in everything that I've learned.

I really love science-fiction and fantasy novels. That's one of my, just like downtime hobbies. Is that, that's a lot of what I read. And I guess the concept for it was really can I take this one thing I love, which is sci-fi fantasy world-building, and bring it to this other thing I love, which is contemporary Australian music and bring them together and just see what happens.

So the beginning of it was just the narrative and the story. And then the animation side was really just me being like, okay, so how do I get this idea in front of an audience? How do I put it on social media or show it to them? Am I gonna draw things? I'm a terrible drawer, like, I can't afford to commission someone for every Instagram story I wanna do.

So animation using a game engine just sort of ended up being the solution to that very practical problem of how I could affordably make a lot of digital content.

Michele Ong

That's cool. But did you have any programming experience before that?

Sally Coleman

No, like zero. I think the last time I animated something I was probably like six and I used PowerPoint to make a flip-book animation by setting all the slide lengths to like 0.01 and then doing something in stop motion on PowerPoint when I was a kid. So I, yeah, never really animated. I hadn't even used a lot of video editing software.

And then jumping into a game engine was a real–

Michele Ong

Steep learning curve.

Sally Coleman

I realised very quickly I had bitten off way more than I can chew. I'd also bought a motion capture suit off the internet being like, how hard can this be?

Michele Ong

Go all in. Why not?

Sally Coleman

So it was fully– it was very hard, but again, in for a penny for a pound. And that was just sort of like a year of watching a lot of YouTube tutorials and spending a lot of time pulling my hair out.

But I got there.

Michele Ong

That's awesome.

So part of what you're doing at the moment is also a PhD in doing this as an actual project. So which came first, like the PhD or the actual idea for creating Big Sand?

Discovering that live performance might be possible in this medium.

Sally Coleman

Definitely the project and the PhD was more me– I, I'd been doing Big Sand for like maybe two years, and I'd spent a lot of time and mental energy on developing the live show, which is something I haven't really talked about. But I guess after I kind of went through all the hoops of learning motion capture and using a game engine and animated characters–

Originally when I came up with a band, I just assumed they wouldn't be able to perform live. I was like, oh, they're, you know, they're digital, they can't perform live. But the more I did it, the more I was like, oh, this is a real time animation tool. Like it's happening in the moment. You don't have to sort of render it out and then show it as a video later.

And that kind of started me thinking of like, what if I did perform as the animated character, like, using the motion capture kind of technique.

So I started looking at how to do that. I reached out to university in South Australia, Flinders University, who has a motion capture stage, and they let me come in and work with them on building a prototype.

And I think the idea for the PhD just came off like, I was in that space, I was working with a lot of them and I realised what I was doing was kind of–

Michele Ong

New.

Sally Coleman

Yeah, that was like further ahead than almost everything else that was going on even in the academic space. And I was like, oh, this is research. So part of it was that I wanted to just spend more time on the research and doing it as a PhD let me do that. It let me spend a lot more time developing the show. But secondly, it was just like a practical problem of how do I fund, again, like treating this as business, how do I fund the amount of time I'm spending on research and development?

Am I gonna be able to monetise this? Is that even why I'm doing it? It's more like knowledge for knowledge's sake. So PhD was kind of a way just to get a small stipend that would help me at least justify some of the time I was sinking into like learning how to do this real time live motion capture animation stuff that I'd been working on.

Michele Ong

That's cool. It's a great way of being able to do it because so much of what we do, especially when it's new, it is pretty much sinking all of your own money and your time into it when you have no idea whether it's a viable product at the end, and yeah.

Sally Coleman

I would say like academia, doing it as a formal PhD is not for everybody, but just kind of looking at what I was already doing and the fact that I'mrelatively confident with that sort of writing. Like I knew I was gonna have to do a lot of writing, but I was like, ah, like that's okay. That's something I'm okay with.

So that kind of worked as a solution for me.

Finding community with others who are also doing their form of R&D.

Sally Coleman

Yeah, not for everyone, but I think there's more people I've spoken to since then who are like, oh, I guess I'm doing research. And I'm like, yeah, like you are.

I know there's all sorts of interesting people I've bumped into in my PhD who are like trying to use new materials to make fashion projects or trying to run businesses in different ways, or trying to market things creatively or like, all of those things are forms of research and if it's new enough and you've spent enough sort of energy in developing it, then yeah, it is a research project if you wanted it to be.

Michele Ong

Yeah, absolutely. And yeah, it's another avenue for promotion, marketing, development. Also getting in touch with other people in related spaces who are doing similar research or even industry people who would be doing similar things or working on adjacent projects which might have overlap. It's such a great way of being able to expand your own research with whatever else might be happening in the space.

Sally Coleman

Yeah, having other, um, I mean, collaborators is the wrong word 'cause everyone's doing their own PhD, but having other people who are asking similar questions and thinking about similar things and using similar technology has been really great. Like I've got sort of a cohort at UTS of other artists and musicians using emerging technology and doing research around that.

So. It's been really nice, I guess, having that kind of–

Michele Ong

Community.

Sally Coleman

Community. Yeah. Which I guess I didn't, well, I'd hoped for, but I wasn't sure I was gonna get out of UTS, but that's been really positive.

Avant Hard Drive, a community of creatives in emerging technologies.

Michele Ong

Yeah. And because there's so many other artists also working on emerging technologies in different ways, like, it can be very isolating if you're doing something new and you don't think anyone else is doing that as well. But other people, even if they're not working on the same things, giving you that support, or at least knowing that they're in that same kind of journey as you are, it's all always very helpful as well.

Sally Coleman

Totally. And I guess on that note, separate, but related, I, I found that exactly like that sense of like feeling a bit isolated because you're working on something that no one else is doing. So I decided last year to start a community night of just getting speakers or artists to come in and share their projects, speak about it for 10 minutes each, and then do like a big Q and A that's very open with the rest of the room.

So I've done three of those. Three, four of those now. They're called Avant Hard Drive. They're in Sydney. And it's just for like, yeah, artists are creatives using emerging technology in some way, and I'll curate a little lineup of people who wanna share their work.

Michele Ong

That's amazing. You should put that online. Seriously, I think a lot of people will be super interested in watching that.

Sally Coleman

. Yeah, I think a big part of it though is having the chance to actually come and meet people in person as well, because it's not just about what happens on stage, that's kind of just the attraction. It's more about like who you then get to talk to in the room and form connections with and share your own project as well. So, yeah, that's something I'm hoping to bring back as well in 2025.

Michele Ong

Yeah. Being able to build that community of like-minded and like-valued creators. Who, yeah, they're just going to keep bringing everyone up together. 'cause you're all keen, you're all curious, and you're all exploring, and there's a lot of collaborative, not necessarily direct collaboration, but being able to go, oh, they're doing it that way. That's a cool way of looking at that.

Sally Coleman

Yeah, totally.

Prior art in virtual concept bands like Gorillaz and Hatsune Miku.

Michele Ong

Talking about you not being sure whether Big Sand could be done in a live show format.

As soon as I heard you talk in your panel at South by Southwest about this, I was thinking about Gorillaz, which is a band, old now, from '98, I think, which is a concept band, and it was pretty much animated. But I've been to one of their live shows and it was, they would've done all the pre-recordings for the animations, but they had a few live performers on stage in front while the animations were going.

And it was an interesting way of being able to kind of blend the two together, although it wouldn't have been like, they wouldn't have been able to do it live the way that you've done Big Sand, which is very cool. But it's, yeah, it's one of those things where every now and then someone gives it a shot and it kind of becomes a contained sort of project with things coming out every now and then.

But yeah, the idea of being able to do these live shows with Big Sand and being able to convey your narrative live with a full interactive thing, it's– to me, it feels extremely viable.

Sally Coleman

Yeah, I mean there's a few things to jump into in that. One is like references and obviously Gorillaz is really helpful as a cultural reference 'cause like a lot of people know them and love them. They're very successful.

I think the way I started explaining my project to people who are familiar with Gorillaz is like, imagine Gorillaz with no money. The reason that I would bring that up is 'cause like Damon Albarn, who's a, you know, lead artist behind Gorillaz, a, had already had like global number one hits with–

Michele Ong

Yep.

Sally Coleman

his band. He had multimillion dollar record deals. He was sort of able to walk into conversations with Sony and be like, give me a million dollars to do this thing.

He also lived with a comic book artist who was very good at what he did. So there was like, already a creative partnership there with a visual person. There was already money, a global network, there are a lot of resources that, I mean I think Gorillaz is awesome, but I think part of the reason it was so amazing is 'cause these [inaudible] managed to really convince the industry to give them the resources they needed to do this crazy cool art project.

Michele Ong

It pretty much was just art project.

Sally Coleman

Yeah, and it was like, it's still is amazing. Like it's so innovative and interesting and the animation style of it is still really just awesome. But that can really only be achieved with a lot of resources, which they luckily have access to. So it's sort of like, how can you be Gorillaz if you are not the lead singer of Blur?

You know, the more new technology comes along, the more those sorts of things do become accessible to, you know, DIY artists and people working in small groups. So, yeah, Gorillaz a great reference. I'm trying to do it on a budget. And yeah, there's sort of other references there as well.

Hatsune Miku is another one that comes up a lot. Who's a vocaloid character, a Japanese like J-Pop star. And I went to one of her concerts as well recently. And again, the animation doesn't happen in real time, so it's not live in that sense. But you're right it's the same as Gorillaz, like the energy of the audience definitely makes it feel live.

The challenges of developing with emerging technologies and building cars for roads that don't exist yet.

Sally Coleman

So I don't wanna say that like when things happen in real time is the only way that you can be live. And what I really wanna do with Big Sand is actually bring together that crowd energy and interaction that you do get at Hatsune Miku, and then add this extra element of like, but then the character can see you, talk to you, you can interact with them and kind of like level it up in some way.

So yeah, in terms of being viable, it possibly could be, but again, it comes down to, like, working on a budget and also because it's emerging technology there's a lot of things that I don't always have control over. Like, you know, how do I find venues that have the screens and walls when I don't have the budget to go and build that for myself every time I do a show?

Michele Ong

Yeah, which is interesting thing, like you mentioned that in the panel or somewhere as well, where the only venues that really would support this were eSports because they're ones with access to high-speed internet and computers and big screens. Which changes the nature of your audience already, just because of the style of venue that it is.

Sally Coleman

Totally. And for example, the first venues I used have been also like museums, which is great, they've being so supportive, but there's a different, when you walk into a museum, you're almost–

Michele Ong

The vibe.

Sally Coleman

In a different mind frame. The vibe different. There's not a bar, it's not loud. So part of it is trying to also choose spaces that set people up for the behaviour I want from the audience as well, which is like dancing and singing and, you know, not sort of quietly standing there being quiet.

Michele Ong

Or it's very one directional when you're in a museum because it's the performance or it's the art being conveyed to you, but you're not directly engaging with it, which is–

Sally Coleman

Most of the time, yeah. There's a kind of a new wave of museums doing like interactive exhibits for kids especially. And they're quite at the front of some of those digital immersive experience things that are very cool. But again, it's very different to live music, which is totally different energy.

So yeah, a lot of it's like figuring out what spaces have access to the technology, the human expertise as well. But then what vibe it's gonna give, so yeah, it's almost like building a car and no one's built the road yet.

Michele Ong

It's a great analogy.

Sally Coleman

Yeah. And like, I've kind of gotta help find the road or build the road as part of the project.

I can't just be like, Hey, I've got this great thing. Who's gonna drive it? So yeah, that's another thing around viability that I've really had to think about.

Feeling authentic in your own medium.

Michele Ong

Yeah. Have you considered Metaverse stuff? Because people like Snoop have been doing things like live concerts in the Metaverse for a few years now, and those apparently sell out ridiculously, in part because he has Snoop Power, but it means that that is an avenue, to take it fully virtual, and I don't know if that's a thing that you can translate your work.

Sally Coleman

I could translate my work. The question is whether I wanna translate my work. And I've kind of grappled with that as well in terms of everything I do is digital.

So in some ways it would be much easier to do a digital first performance, andI use Unreal Engine to make my stuff.

Unreal Engine makes Fortnite. They've got a new version of Unreal that's specifically designed to make content in Fortnite, including motion capture and things like that. So heaps of concerts are happening in that space.

Two things. Like one, they're generally artists that already have a large following. So like Travis Scott, and Justin Bieber, and Snoop Dogg, and you know, people who, the name alone will attract an audience. That's why they do these really, you know, high level partnerships.

And two is like, okay, even if I was an independent artist and there are cool independent artists doing that stuff, it's not really where I hang out. And I don't know if it's– I feel like I'd feel like a bit of an imposter into this world–

Michele Ong

Tourist.

Sally Coleman

Exactly. And I like, I do think that's important of well, if you don't know the space, you can't just barge in and be like, I'm here, everyone. Come to my shows.

I think I would need to get time–

Michele Ong

To understand the culture of that different virtualised space.

Sally Coleman

Yeah. Which I'm open to, but right now the culture that I know is what it feels like to be a live physical music gig and like, I spent a decade doing that and interacting with audiences in that way, and that's what I know and love, so it makes sense for me in my head to gravitate towards, you know, physical concerts. I don't know many other people doing that.

Whereas I think if I was like a digital native and I spent a lot of time in 3D virtual spaces, or that it's Roblox, or Fortnite, or VR chat, or whatever, and I would feel like that was an extension of my existing community rather than, yeah, being a tourist and like, hello everybody.

But it's totally feasible and like maybe one day in the future, it would be looking at doing a hybrid, you know, I do a show, that's physical, but there is a way that you can engage with it, you know, digitally or remotely as well.

Michele Ong

That's really cool. That there are so many opportunities there to explore, especially because of the technologies that we now have that are kind of maybe not fully accessible to everyone, but it's at least getting to the point where the average consumer has now more ability to get access to it than they did before. So that's very cool. And you know, you might be the one who builds that road.

Sally Coleman

I think it's gonna be team effort. So if anyone else is out there doing um, interesting animated performance projects, then get in touch 'cause I do think it is gonna require collaboration and artists and venues, and all sorts of arts organisations working together to make it feasible.

The changing landscape of events and recreation.

Michele Ong

Yeah. No, that's cool. Just the opportunities available for people to also expand, I guess, when we're looking now at a generation who are apparently less likely to go out and club and less likely to go drink. Having venues, which traditionally only cater to that sort of environment, expanding their infrastructure to be able to support alternative events would also support cultivating an industry of alternate types of recreation, which is very cool.

Sally Coleman

Yeah, I think that needs to happen and it, it is tricky in Australia 'cause we have a relatively small population, so risk-taking can be harder. Like I do think Australia's probably a little bit behind the curve when it comes to taking risks 'cause we stick to what we know works.

Michele Ong

And we wait for other people to get it wrong, to figure out.

Sally Coleman

Exactly. And like I don't blame artists or venues for that because it's so tough as it is even doing what you know works. Why would you go out on a limb not knowing if there's an audience there, not knowing if people want it, not knowing if you can afford it.

There's a lot to lose, so, yeah. But I do think it's important to be able to experiment with new kinds of events.

And on that note of younger people drinking less and engaging with entertainment differently. I think one other big thing that maybe hasn't been talked about yet is this, is the social media ban, which is also directing people away, young people away from, traditional platforms where you might find out about bands and live music, and into 3D virtual spaces like Minecraft or Roblox, where you might go to a concert and that your first concert might be in that space.

Michele Ong

And that's an interesting point as well, because that's a primary advertising and promotional avenue for so many industries at the moment. And even things like Roblox, at South by there was another session talking about the marketing power of places like Roblox and the number of traditional brands starting to insert themselves into that space just because of the multimillion, even billion dollar, capacity for earnings in that area.

So yeah, it's definitely a changing space for the way that we do engage with content.

Sally Coleman

Yeah. And young people are really, they're gonna be graduating and that's gonna be their first access to artists and performances and theatre. Like it will be in 3D online spaces. And then if you do want young people to start coming to physical theatre and music, you gotta sell it to them.

Michele Ong

You need the pitch.

Sally Coleman

Yeah, I think we've gotta think creatively of how that transition's gonna happen with that generation.

Michele Ong

Yeah. Lots of fascinating research avenues for you and other people.

Unreal Engine and virtual production.

Michele Ong

So back to Unreal Tournament, sorry, Unreal Engine. There are a couple of other platforms available to do that. So why did you choose Unreal Engine specifically to use for your platform?

Sally Coleman

I mean, at the end of the day, it probably would've been a choice between Unreal and Unity, which are the two larger players.

I think Unity, honestly, was doing some of that motion capture, well, virtual production stuff first. For anyone listening who doesn't know what virtual production is, it's mostly used in film and TV rather than what I'm doing for live performance.

But all the stuff you see that's replacing green screens with LED walls and things like that, I do feel like Unreal Engine has come out hard in that space and they've invested a lot of energy into tools for virtual production. So it's become a little bit of industry standard, I think in film and TV.

Whereas I think games, probably, Unity still dominates in some indie game spaces. But yeah, for me, I felt, I just saw Unreal popping up a lot more in the sorts of examples of things I wanted to do. I found more resources for it, I knew more other people who were using it.

Yeah, a lot of it just comes down to like, can I find someone who can explain this to me? And what do they use and, yeah. It wasn't any ideological choice. It was just okay, it seems to have the tools that I need and I can find more other people using it for similar purposes. It's really what it came down to.

Michele Ong

Yeah, and it's so important as well as when you're approaching a new space. The community support and access to resources is so important, and there's so many new platforms that pop up for such a wide variety of things where the community's poor, the support is either hostile, or negligent, or just not there at all, and there's just not enough documentation for you to even find your own way there or your own path.

So being able to have access to tools and to communities where the tools are well supported, it's the only thing that's gonna make things easier when you're trying something new.

Sally Coleman

Yeah, totally. The other thing that did appeal to me, like after trying Unreal, and the reason I probably stuck with it is I have no coding background at all, and the thought of having to learn to code on top of having to learn to animate, use motion capture, et cetera, et cetera, I think was just gonna be a bridge too far, like learning too many things at once.

So the fact that you can really easily use visual coding instead in Unreal Engine and that Blueprint system is very– well, there's just a lot of tutorials around it that are very visual and easy to follow as a beginner, and it me to stick at it.

Whereas like having to do things in code on top of having all this other stuff, it would've just, I think, maybe pushed me over the edge a little bit in terms of brain capacity.

The technology is more than just a tool, it's now part of the creative process.

Michele Ong

Which is entirely fair. So in this entire process so far of learning all these new things and entering these new spaces, what are some of the most surprising things that you've learned?

Sally Coleman

I don't know. Like I am blown away by the capabilities of game engines and like, what you can do with them in physical spaces as well. The more I lean into it, the more I'm like, oh, I could do, like, literally the possibilities are endless when you combine it with things like projection mapping, or tracking things in spaces, combining it with AI depth cameras and whatever.

You can literally take any input from the world, whether it's the volume you're speaking at or where you move in your kitchen or like anything you can think of, and you can turn that into data that has some kind of digital representation. And I know that sounds very abstract, but as an artist I'm like, I could take any input instead of just like pressing a button on a keyboard or something that's kind of a simple input. You can take anything physically in the world and then turn it into a piece of art.

Michele Ong

So that's really exciting and then seeing the ways that other artists use that for dance, music, visual arts, you know, these kind of immersive installation works that are coming out now has been really cool, and like I think, yeah, maybe the surprise is how much I've actually enjoyed using the tools and they've become part of, instead of just a means to an end, it's actually kind of become, part of what I do. Is involved in your creative process now.

Sally Coleman

Yeah, way more. And originally it was a way to get from A to B, but now it kind of is– it's A, and it's B. Like, using game engines has become kind of part of what I'm creating and I couldn't do it without that medium.

So I– that sounds a bit abstract and vague, but I think that surprised me the most is I've fully become an Unreal Engine girlie, and likeyeah, I really enjoy it.

Technology facilitating immersive art.

Michele Ong

That's very cool. And yeah, talking about any kind of data inputs being able to use to have creative outputs, there are a lot of very cool installations doing that. And I think there's a company in Sydney, Uncanny Valley, well, I think they're international, doing that sort of work as well.

And taking environmental data, temperature, air quality and all of that, and turning that into live art installations where, when people walk through the space, the art will shift around them based on the people and the other environmentals that are there. And I think that's– you become the art, and that's very cool.

Sally Coleman

Totally. There's two I also would shout out in Sydney. One is Stalker Theatre. I saw, they had a thing at Carriage Box in Sydney the other day where it could track people around the walls, but it meant like kids would go up and they were like, you know, pressing birds that would fly away from them and chasing things around the walls. And that was very cool.

The other one who really helped me put together my very first full-length live show. They would like, let me come into their office and just sit there and work. And then every time I got stuck, which was literally every 15 minutes, I would be like, why is this not working? Or How do I do this? How do I do that?

One of their programmers would come over and just sit with me, and rather than doing it for me, would like show me how to do it. So that was really invaluable. It was kind of like a mentorship that I could just come in and work out of their office and they would give me support on everything from like version control to coding Blueprints to documenting things properly.

So that's S1T2, sorry. I dunno if I said that, but they recently did a big collaboration with Melbourne Museum called Antopia, which I haven't seen yet, but you get to go inside the world of an ant and they have all these interactive, immersive elements around things like you mentioned, like being followed through the space.

Sorry, another one they did where you could clap and it created this sonar-like effect where it lit up the things around you based on the volume of your clapping.

So yeah, they're doing very cool things as well in that installation immersive theatre space.

Big Sand as a live performance with audience engagement in a virtualised environment.

Michele Ong

Very cool.

So we'll kind of cut back through 'cause directly with Big Sand, like you've done now two live shows. The most recent one that was at South by Southwest Sydney at Fortress, which was very awesome. And considering– considering some of the events afterwards where they had so many technical difficulties, you did amazing.

Like that show was incredible. And you did that from UTS across the street, streamed that into Fortress, and that was basically put up on the big screens. You had your helpers around you, kind of, keeping an eye on making sure that things were doing the things that they were supposed to, and you had your avatar, in plushy form, interacting with us as well, which was very awesome.

Sally Coleman

This is the first show where, so there's four members of the band. I kind of perform as the lead singer who, conveniently, it has two arms and two legs. Makes it much easier to animate. But another character, Kiz-OO, who's this little robot, I got a puppet made of Kiz-OO for the first time.

And this, I literally got the puppet two days beforehand. I hadn't seen it. It was very last minute decision, like a week out. I think it was two weeks. I gave the person who made it two weeks notice.

Michele Ong

That's good.

Sally Coleman

We didn't even know it was gonna be a puppet. I was like, maybe a helmet, or like a costume, I don't know. And then she hand stitched this amazing felt puppet that I– it was like one of the highlights of the show.

And yeah, it kind of built the idea of bringing physical things into a digital space and vice versa is what I kind of wanted to play with there.

But I'm glad it looks like it went smoothly from your end, because in the lead up to it, it was like 10 hours of just things kind of digitally exploding before the show went on, but–

Michele Ong

Yeah, but even the issues that you had, the way that you integrated that into audience participation, it was also a really great way of doing it. So, you know, clipping issues, people always know about that in games, but you made that part of the audience engagement, and that was fun. And I guess because people were expecting this type of performance and this type of engagement, they were equally prepared to engage with it.

'Cause you know, a lot of bands, a lot of shows at the moment, which is very sad, when you see them try to get audience participation, it's kind of half-hearted or people don't really wanna stand or people don't really wanna cheer or clap. But the entire room was prepared to kind of wave their arms and do silly things and participate with this virtualised space.

And I think that made it really exciting. But I guess for a few people who didn't know what they were walking into, it's like, you guys are weird.

Sally Coleman

Yeah. Oh, it definitely is. And like every time I've done– even the prototype, which was years ago, which was 10 minutes, the thing I was testing wasn't so much the tech, it was like, if an animated character starts talking to you and telling you to do things, will anyone do it? Or will it be like a movie where they sort of just sit there in the cinema being like, okay, interesting.

So I still didn't know like why exactly people do respond so enthusiastically, but it's been pretty consistent throughout performances that audiences really do step forward and they wanna get involved and there's a lot of big grins and people– I think 'cause the whole thing's so silly people don't seem to mind being silly as well. It doesn't feel too confronting.

And obviously some people do like they, you know, would sit upstairs or prefer to stand at the back, totally fine. But it's, yeah, it was really nice in that performance especially, I was surprised by just how engaged everyone was.

Michele Ong

Yeah, I think it helps to know that you're actually there to be engaged with because it's different, like a lot of kids TV shows, they'll do the thing and they'll say, Hey kids, so what's the answer to this question? And then there's this silence and you know, like you don't even need uncanny valley for that, you know that this isn't a real engagement. And even the kids sort of know, they'll shout it out, but they know this isn't real and they know that they're not actually being talked at.

Whereas in this live environment where you are getting feedback from you and the avatar, you know that this is an actual engagement. You are actually having a real in per– well 'in person', but real-time interaction. And I think that helps to promote that engagement knowing that you are going to get the response.

Sally Coleman

Yeah, I think it's like the audience is just as much a part of the performance as I am. Like they've got a job to do. Always amazed how willing people are to like step up and do the, the thing, whether it's clapping or dancing or whatever their role is in a live performance situation. A lot of people just kind of jump in and then they just do it, which is great.

And that's part of my PhD is really. It's less about like the technical side of how do you, you know, do an animated performance. A lot of it's about that, like cultural–

Michele Ong

Engagement.

Sally Coleman

Yeah, like how do you get people a, to understand what's going on in emerging technology. A lot of the time the audience is like, I don't know what's happening. I don't know what I'm supposed to do, so like, yeah. How can you communicate that to people?

But also, yeah, like what sort of things do they react to and how do they react and how can you anticipate and curate that a little bit as a performer. You obviously can't control it completely, and that's part of the magic is the audience will always surprise you.

But you can sort of try to funnel it in different directions based on the outcomes you want.

The future for Big Sand.

Michele Ong

That's very cool. So what's next for Big Sand?

Sally Coleman

I'm still figuring it out. Like I've got a lot of pots on the boil. I would love to do more live shows. So, you know, there'll definitely be another live show at some point. But again, there's a lot things to figure out around like where do I do it? How do I pay everyone to do it when it's such a early stages project, all those sorts of questions.

How do I get audiences to come to it? So live show is one of them. I'm also working on a video game, which is cool. I don't think it'll necessarily be finished anytime soon, but that's been a really great project, and it's also let me work with game developers a bit more and I've learned a lot from them about the games side of Unreal Engine, whereas previously I'd just been doing more of like the animation side of things. So using a lot more logic and inputs and variables and things like that.

So that's in the works. And then, I don't know, it is just a lot of projects happening, things up in the air. So yeah, a little bit of it is like, keep putting myself out there and just see where the world takes me.

Michele Ong

No, I like that. That's a great way of doing it because yeah, in emerging spaces, you really don't know where the path could go.

Sally Coleman

Totally. So yeah, well I'm trying to collaborate with people, reach out to people whose projects I love and be like, I'd love to help if I can. Yeah, and just interesting to see how all this technology sort of keeps progressing in the performing arts, especially.

What advice would you give someone who would like to do what you do, and what advice should they ignore?

Michele Ong

Absolutely. Okay. And last question for you. What advice would you give to someone who'd like to do what you do and what advice should they ignore?

Sally Coleman

And my advice is like Nike, like just do it. I know that's something that's easier said than done. Like there are hurdles in terms of like, do you own a computer? Like that would make it really difficult if you don't. But the advice to ignore is you really don't need to know as much as you think you do.

Like, you don't need to have studied game design. You are the one variable you can control in your ability to learn. I would just believe in that.

Michele Ong

And everyone has to start somewhere.

Sally Coleman

Yeah. Just start. Like, it doesn't matter how bad it looks, or how mediocre it is, just get in there, give it a go. And also one thing that's really helped me is getting things in front of audiences.

Even if they're not perfect, you have to let the audience see it at some point because the show doesn't exist without an audience. So yeah, just get started and whatever you do, just start getting it in front of people and see what happens.

Michele Ong

That's amazing advice.

Thank you so much, Sally, for speaking with me today. It's been so cool talking with you about your journey and Big Sand and the way that you've just immersed yourself in this environment to help create immersive environments for everyone else. It's very cool.

Sally Coleman

Thank you. It's been a lot of fun and yeah, thanks for the chat.

Michele Ong

My pleasure. Hope you have a great rest of your day.

Sally Coleman

Thank you.

Michele Ong

If you enjoyed this conversation, please let me know. Subscribe to this show, leave us a rating, and share this with your geeky or geek-curious friends. You can also support STEAM Powered on Patreon under steampoweredshow, the link for which will also be in the show notes. Thanks for tuning in, and we'll see you next time.