Episode 2: XJ music theory & artist collaboration

Transcript

(music: intro)

Charney Kaye
We got Dave Cole here.

David Cole
Hey.

Charney Kaye
Mark Stewart.

Mark Stewart
Hey.

Charney Kaye
Ian Hersey.

Ian Hersey
Hey.

Charney Kaye
And Jamal Whitaker,

Jamal Whitaker
What’s going on?

Charney Kaye
Let’s talk XJ music theory. As far as that road stretches on into the future, it’s incredible how much we’ve accomplished. Because we had big ideas, and we had big plans, but we had to start somewhere. And I started with just the sequencing of audio and adding it into a final mix. Having that work continuously having it fabricated in the cloud, and having a collaborative interface where we can compose music together and listen to the stream change in real time.

Mark Stewart
For a while, the drums themselves were following the chord changes that I wrote.

Ian Hersey
Oh, that was a special zone. The pitch shifted drums edition.

Mark Stewart
Yes, yes.

Charney Kaye
It was based on my excitement about the technology. And I thought, oh, it’s unlike any house beat you’ve ever heard before, because it’s actually changing with the chords. And like, even when I heard it, I thought like, oh, well, it’s like kind of sick. And Dave, when he heard it, he was like, Look, the thing about drums is that they don’t do that for a reason. These things already have time honored traditions and roles for different reasons, like the drums are a tonal, even when they are slightly tonal. Even though you might tune a drum kit, in order to get certain effects, you’re deliberately letting that sit in a specific place and be a foundation, maybe you want to deliberately change it up on the bridge or something. But you don’t just let that move around harmonically with the rest of the stuff, it actually provides a foundation so that you can hear the other harmonic motion.

David Cole
I totally get what you’re saying about whatever I said about, you know, these traditions and time honored things, it would have been kind of interesting to hear those drums, shifting with the harmony, but having that harmony actually present, you could play a drum like you would play a melody like you could play a marimba, which is, you know, a percussion instrument, right? That is a melodic percussion instrument. But if its role functionally, is to just establish like a groove, it’s really hard to understand what this pitch shifted, changes would be doing or how they would translate. But it would be really interesting to hear it with some keyboard instrument, actually playing those chords, and there maybe being a melody, it would be kind of wild. Actually, that would be pretty interesting.

Charney Kaye
So it comes full circle, that maybe in the future, we actually will do some of that work,

David Cole
Maybe there’s like a dance between having some far flung idea. And then through that experimentation of trying to go as far as you can with that, reining it back in and making that idea sort of accessible or palatable or give it enough context for the effect of that idea to really take hold in the listener.

Charney Kaye
It’s been this really long, interesting journey for me in terms of learning about music.

Mark Stewart
I have also learned about music from working on this, I play keyboard. And I know what makes sense to me when I’m playing it, like what would be an appropriate bass notes substitution for this group of notes that I’m playing up higher, or whatever, but I had to actually kind of systematize it like making these lists of notes for like the various types of instruments. There are all sorts of pretty fine properties of harmony that I had to find out that I knew or discover there’s like low interval limits, you don’t want a certain chord tone. That’s like a color note, like not super foundational to occur too low in the chord. I’ve definitely established values in an explicit way that I think before were just sort of autopilot, you know, something without consciously knowing why it’s right.

Charney Kaye
And that leads us all the way up to like two weeks ago, when Ian hands me over this exhaustive spreadsheet of chord definitions.

Charney Kaye
It was a long road to arrive at a usable interface to compose music for XJ.

Jamal Whitaker
What you’re doing essentially is taking these years of experience looking at grids and looking at the placement of sound in grids, whether that be in you know, Pro Tools, Ableton reason, whatever, and kind of doing an exercise in what it’s like to make music as a deaf person. If you can imagine sitting down to do something like music production, you’re just doing it with like, the most important sense kind of missing. You get to hear it, you know, a minute later and then you get to be like, oh shit, let me you know, pull that back up. And obviously I did that wrong, you know, where you have those moments. It just relies so heavily on your experience to say, okay, well I know what a pattern of a kick and snare should look like, just visually. Yeah, it’s not so much the lack of one sense. It’s the reliance on like, spatially, what do you see here? What plays in your mind, you’re kind of reading music, right? I had spent some time taking piano lessons in like the fourth grade and all that stuff just kind of went in one ear and out the other. I play by ear. For these guys, I imagine it’s pretty easy to look at sheet music and just hear what’s written down on the sheet.

Mark Stewart
Once that Stepmatic thing happened, there were many more ways it could possibly go wrong. A lot of iterations of like, grid step size and Phantom bugs have you plot a note here on the grid, and it pops up three and a half beats later, that sort of thing. We hacked at it for a long time.

Jamal Whitaker
Oh, yeah.

Mark Stewart
And we have like, a viable user interface now.

Jamal Whitaker
Being the tool that I designed. In months prior, there was aspects of it that were like, okay, cool garden, man, there was so many just little things that were just various bugs that had to get ironed out and did over time. And of course, without being like my main focus, every time one of those things got ironed out, it was just like, “oh, man, this is amazing,” like, “this is the greatest thing ever,” you know.

Charney Kaye
It’s been really surprising where the most effort has gone into to just pull this off.

Charney Kaye
Building stuff is fun, but then what is also fun is not building stuff.

Charney Kaye
Parallel to the music composition, there’s been this journey of research and development of the entire paradigm of an algorithmic medium.

Ian Hersey
How would you, like, succinctly describe an algorithmic medium?

Narrator
A medium is any means of recording and transmitting information. And music is information. What makes it an algorithmic medium is that instead of directly writing on the medium, verbatim, you’re providing instructions. And the medium, instead of just being static, is alive in some way,

Charney Kaye
The living aspect of the medium is not consciousness. I mean, just to be clear, like, it’s just not, it’s not an AI, it’s not a conscious entity. It’s more like the individual cells of your body, following the instructions in your DNA. The broad mission of XJ music is to cultivate artists work in an algorithmic media, and try to find this future in a way that is compelling for artists and audiences, focusing on making sure that the human aspect of this is not lost, all the way through to you sitting there having this experience, you know, you still know that this is music made by people, and you know, who made it and you know, what their intention was when they made it. And the platform and app that we’ve built is our particular approach to that.

Mark Stewart
Before XJ was fabricating notes when it was just drums. There were much fewer things that could go wrong. It was a simpler interface. Basically, there was just like this pretty simple graphical thing by which I would input chord names at times only. So every once in a while There would be some error. But by and large, it seemed to work correctly. I mean, it worked for our purposes, which at that time was not yet actually thinking about instruments or anything like that. In just a couple of months before we had note fabrication, we had to take it seriously that how do we finally get it making pitched sound? And so there needed to be like lanes for the different instruments and the viable notes. And we also needed to figure out how to write programs, detailed programs, so riffs, melodies, and things like that, how do you represent them graphically. So just like sets of chord changes, there might be multiple sections of a song that that are, have different looping sets of chord changes, and then also furnishing, like, what the acceptable range of notes for any given instrument in any given time are, as well as writing detailed programs, like melodic and rhythmic shapes, to be followed by any of the instruments which get kind of combined together with the voicing of any given time and turned into musical parts.

Ian Hersey
I would say that maybe a quarter of those sounds made it till today. I mean, the next time that I did, like a major batch of work on instrument sounds was to do the Showhouse string, I had already learned so much about what would fly and what wouldn’t. That was like, you know, near 100%. okay, that’s cool. We had also like, not established, like whether there was going to be sort of an effects engine going on. And what we’ve settled on now is that like, for the time being at least affects EQ, time based effects, any of that stuff is printed into the waves that we’re uploading for the multi sampled instruments, that still takes trial and error, like, I’ll upload an instrument via like, fuck, the filter needs to be down like a little bit, it’s too bright, and then just have to go redo it and then re upload all of them. So much more was in place, whenever I didn’t show how sounds like I knew that it was going to be 120 beats per minute, just straight. So I could put written like effects into audience that I could make progressive loop audios, they were going to work, no matter what.

Charney Kaye
And as I climbed each one of these technical hurdles, you know, on the one hand, building, the finished product is the most expensive way to find out whether or not something is going to work. And on the other hand, this is one hell of a complex MVP. I wonder if we could actually have tested this out any simpler or faster.

Charney Kaye
What have been some of the most challenging aspects of this process?

Mark Stewart
Excluding things that don’t work together is like the trickiest part of all of this, I feel like on my end. Yeah, I use memes to like categorize different song types by like how fast the harmonic motion is.

Narrator
The word meme isn’t just the latest viral TikTok. A meme is a unit of culture, or a behavioral system passed from one individual to another. An XJ artist breathes life into each template with a unique set of memes.

Ian Hersey
Part of like teaching that algorithmic medium, how to do things that makes sense is just telling it “don’t do this.” There’s a meme tag that I have for like, “don’t combine these two things.”

Mark Stewart
If something has chord changes that are too fast, then that should exclude like certain types of bass lines that are written. Things like that.

Jamal Whitaker
Yeah, that’s bleach and ammonia.

Ian Hersey
That’s bad. Don’t do it.

Jamal Whitaker
Yeah.

Mark Stewart
Or this type of thing should only come up in context of this one specific thing. There was a lot of random choice. There was like too much. It was extremely chaotic. I loved it, but it was not really viable music. I remember, there was that one vocal like, “ah.”

Ian Hersey
Yeah, “doh doh doh”

Jamal Whitaker
Yeah, as a sticky instrument. And I remember that came out like right when stickies were like brought online for the first time, and those programs were like, way too busy, and there were no sticky buns. So it’s just continuously random note choices.

(music: too much randomness)

Mark Stewart
And as fun as that was and as much as it technically worked, in that there weren’t, you know, bad harmonic clashes, it really needed to get reined in.

Charney Kaye
In this process, we learned to find a balance between randomness and patterns, logic and nonsense. One tool we developed to work in this space we call the sticky bun.

Narrator
The term sticky bun is unique to XJ music, like a cinnamon roll baked by the dozen, a sticky bun is a delightful treat served as a similar looking group.

Charney Kaye
XJ starts with some randomness and then bakes a dozen sticky buns with variations on the central riff.

Mark Stewart
And that’s where the idea for sticky buns where, like it needed to kind of copy a part from those from one round of those random choices and build a loop.

Charney Kaye
The frameworks we’re building out in the different channels are laying the groundwork for us to bring artists into a recording studio and add more performances into the channels.

Narrator
In XJ, a performance is a recording of a human artist’s rendering of the programs in audio form, XJ will remix these recordings to seek many variations of the original version.

Mark Stewart
Like say you have a trumpet player hanging out for two hours and just playing a lot over a couple of chord progressions. And someone can curate that right and cut it up into like, a little two bar segments or whatever.

Ian Hersey
And those can be like freely combined into a new melody whenever.

Mark Stewart
Yeah. Dave and I were talking about this a lot. And I think it’s gonna be cool when we can get some awesome keyboard player in there for an evening recording session. And then just sort of have him blow over a lot of these tunes. And then that’s just up to people snipping it up, and like categorizing what stuff it could work over. And then basically making like some kind of sticky bun part out of that. If we’re talking about house music, and also Lo Fi Hip Hop, that’s pretty true to what it is. And listening to house mixes, there’s like one weird vocal thing.

Ian Hersey
In house music, the melody is most often like some sort of vocal sample. The inflections of that are, you know, pretty crucial to why it’s like evocative.

Mark Stewart
Two syllables in between like, two two-syllable words, so that what you actually hear is like, verbally meaningless. So it’s just an instrument, but it is, you know, harmonically and melodically like, relevant and compelling.

Ian Hersey
If the melody is going to be generated like a sticky bun, it’s going to be a one shot kind of note-for-note thing, and that’s going to really limit our ability to inflect and make it natural.

Mark Stewart
As far as Lo Fi is concerned, we already have hook audio, you know, we can just take a performance, and then just snip it up and make hooks out of it. Technically, there’s nothing keeping us from, like, if we wanted to actually have in play like a 32-bar solo over a particular tune, we can just make that happen. Yeah,

Jamal Whitaker
It could even be a live performance that mimics a four-bar loop. But still, within that live performance, you have that added layer of you know, human variation and organic flow to it.

Mark Stewart
Yeah, I feel like we can have one or two of these sessions with some good players, or even ourselves doing it and end up with a ton of new material.

(music: new performances)

Charney Kaye
Also, by that time, Ian had gotten started on the ambient music project. And he was using the instrument in some interesting new ways.

Ian Hersey
The idea with space was just like factor tempo out of the equation of making a stream. I think originally, I’d just put the tempo like 15 BPM, so slow as to render tempo basically like a non-issue. That really opened it up. And then if I designed the main programs in the right way, the end of any main program could overlap with the beginning of any other one, and there wouldn’t be dissonance, if I just gamed it right. Getting like a little bit frustrated by how building a thing out of entirely like, one shot instruments could kind of make it feel a little like, robotic or flat. If that’s what you have to work with, then all your like variants and texture and sort of interest in sounds has to come within each audio file for each sound. And so from there, yeah, just made a bunch of sounds that kind of had movement within them, and then eventually have like rhythmic schemes within them, and then had like phasing going on between notes of the same sound. Each note has an individual audio that’s playing like, say an eighth note rhythm on that note, except they’re all at different tempos. So when you combine two of them, they will naturally phase with one another, no one will be slightly faster will be slightly slower. And then you’ll have this kind of never totally lining up kind of rikey phase thing.

Mark Stewart
Slaps is more about looping audio. So what I’ve been doing is creating like environments that have like a few different possible for change loops, where you have hooks and cords that can be basically used interchangeably. So it’s sort of alien story. But it’s also less generative than pretty old, detailed programs method that we’ve had.

Charney Kaye
I’ve thought of a lot of ways over time that I could have done fake versions of this. But we wouldn’t actually know if it was going to work or how it was going to work. And we would just be behind the ball if the idea caught on ahead of that. So instead, now we’re poised to deliver this thing that we’ve been promising and talking about and working on, it’s ready to go. If we build enough of a platform, there becomes like a new kind of live performer, to use XJ to perform all the stuff that’s been composed by other people. The live music output is one of the most challenging things to represent in the user interface, XJ runs continuously making nonstop music, but under the hood, that music is divided into segments.

Narrator
A segment is a portion of a program that can be loaded and executed independently of other pieces. XJ is a machine operating on a continuous series of segments of music, each segment begins as a set of ideas then fabricated and published as audio.

Charney Kaye
The foundation of the live music generation is the creation of one segment after another and the repetition and variation between Segments. Dave put the idea in my head years back, that musically, if you do anything, and then repeat it, you’ve done a third higher thing. You’ve actually drawn the audience’s attention to the repetition, put the ball in their court to interpret why you just repeated that thing you just did. Philosophically, that’s underpinning a lot of what extra is doing these days where sometimes I’ll hear it, like it just randomly cut into a new segment of chill house and it goes, okay, you know what I’m gonna do, I’m gonna play a bass line and then stop and hit you with like this other thing that’s like some pads, but the bass stopped, you know, we’re just going back and forth today. And like, I know, the moment that I hear it, okay, that’s actually some weird intersection of things that causing chunks of music to just be missing for, like, half the segment. But then the moment that second segment hits, and it just does the exact same shit again-

Ian Hersey
Then it’s like music.

Charney Kaye
Yeah, it has an effect.

Mark Stewart
Yeah, repetition legitimizes, yeah.

Ian Hersey
You can say that again.

Charney Kaye
Yeah, then it’s music.

Ian Hersey
You can say that again.

Charney Kaye
Yeah, then it’s music.

Mark Stewart
But whatever.

Charney Kaye
Having a sense of humor has been essential at every stage of this process, like I would have died otherwise.

Mark Stewart
I agree.

Mark Stewart
Yeah, you heard that thing where it’s like, okay, three of these bass notes are not there. The chord instrument just stops for like this weird set location of of time, every loop. Yeah, I always sort of interpreted it as like, xj just like, not being able to compute all of those notes at once. So it was sort of like direct its attention to where it could. I still don’t know what it is. We were like greatly expanding voicing stuff in a pretty short amount of time, despite it being an unintended thing, and a thing that would eventually need to get worked out, right. I’m fond of it. When I heard it. It also almost felt like it was choosing to fabricate musical parts based on their interest. If you would hear some of one instrument that was doing something and then a chord would change and you would hear a different instrument a while. And it almost sounded like it was choosing these things like we still don’t know what it was right?

Charney Kaye
There is an uncanny sense that something is influencing it. And I think we have inadvertently built a mind through layers and layers of stochastic decision-making. But it’s all based on curated cultural input from humans. I learned a ton from Douglas Hofstadter theorems in “Godel, Escher, Bach,” which is considered to be a foundational work in AI. But I go back to the old school media theory like Marshall McLuhan, the medium is the message, we’re a straightforward evolution of media in terms of being an instrument and using it in a live broadcast.

Jamal Whitaker
As I’m going through and creating this new genre that is solely me within XJ, I’m kind of cataloging the process, and figuring out what would be interesting to not only other artists, but to a broader audience as well. So a lot of that is, you know, when I’m sitting there working, it could be two in the morning, I’m just sitting there clicking, dragging sounds in from splice, chopping them up in Pro Tools. And on one hand, it’s, to me that is mundane. But there’s so many people that will tune in and watch somebody do that for hours on end on Twitch or YouTube, in a live stream, what have you. And why not just throw that out there? Yeah, put some beats on in the background and get people at least something interesting to listen to, and get a taste of what I’m creating live right there in front of their eyes. But uh, but yeah, I think it could be pretty interesting.

David Cole
If XJ could create an environment that felt welcoming and exciting to artists, here’s a new playground to play in for artists and musicians that is going to be appealing to artists, I think if the user interface for XJ is intuitive enough, or easy enough to use, and the learning curve is not too steep. And they can see some sort of demonstration of it, where they can see the possibilities and implications of the medium. Because the process of making music has become such a, an interesting medium in and of itself, documenting the process of what you guys were talking about could be so interesting, and I think actually really exciting, and could be attractive to other musicians.

Jamal Whitaker
Yeah.

Charney Kaye
When you’re building out a Voodoo Lion channel, that’s already very personal. It’s already music that you’ve thought of in terms of songs and brought some amount of finality to, you know, how does that translate in terms of your identity as Voodoo as a producer into XJ channels.

Jamal Whitaker
So one of the key things I’ve been looking to do with the Voodoo Lion channel is one, you’re starting with songs and songs that some were released, some are unreleased, they’re pretty much going to go on the same album I’m imagining, but the way that I’ve been constructing it, is by using XJ’s meme system. With trap, we’ve been going back over having memes be genres, and in some circumstances and channels beforehand, we’ve used different terms, meats, cities, colors, elements. And in this case, I want to go straight for songs. So the very first song that I’ve actually put into the Voodoo line channel is a song that I did called Hey Toni. And the way that I’ve started out is by defining the rhythmic pattern that Hey Toni uses, literally importing those drums of course, and recreating that pattern within XJ and basically defining Hey Toni as a meme, and letting it ideally go from there and be able to populate the drop in the same bass tones the same. Other percussion Everything basically, but the sample as an experiment for this because you don’t want to be, obviously running around with copyrighted material. And this, that has also presented a challenge too, because a lot of my own production is sample based. And so it’s kind of had me going outside of the box and really sifting through my work and picking out things that aren’t sample based, and also tapping some of the other XJ music members on on the shoulder and getting them to, Hey, you want to collaborate and we play these samples, or just see where your mind goes, I’m imagining it as being like a stone, okay, we’re dropping the stone in this river, the ripples that come off of that are going to be where somebody like Mark or Ian would say, Hey, I like these drums, I’m hearing them. I want to see what this sounds like with some Oregon over it. And I’m going to tag that as Hey Toni and have XJ be able to oscillate between, hey, we’re going to take the original sample that Mark replayed, and we’re going to add some Oregon that in, played over it, that matches and, and just see where it goes. If I’m just scoping it out, it would have to be 12 songs, and 12 main programs to really be able to get that sense of variety. And as a listener keep you engaged.

Charney Kaye
So it starts off as a Voodoo song becomes a taxonomy meme, like a high level meme?

Jamal Whitaker
Yeah, cuz if I think back to the early days of, of when Mark and I were going through Lo Fi beats, we were starting with the idea of a city, the different cities of Lo Fi are Aberdeen, Boulder, Nagoya and Luxor. And then that city could then be broken off. Aberdeen has somewhere around like eight different programs that are associated with one meet. So I’m just kind of taking that and running with it. Because it’s like, hey, the most recent thing we were doing the most recent way we were operating and operating well and getting sounds into Lo Fi. So keep that rolling.

Charney Kaye
What is the audience experience going to be listening to the Voodoo channel?

Jamal Whitaker
It’s going to be twofold. One, if you’re familiar with my work, you’re going to be able to hear bits and pieces of what you know. And see it reimagine constantly, every time you open the Voodoo line channel and extra music, you’re gonna hear something that you know, with something that you never know. And that’s dope to me. Like that’s that’s the whole idea behind collaborating with artists is taking a project that you’ve put all that love into all that work into and seeing it reimagined through lenses that you know, you never would have even thought it would be capable of sounding lines. And so that brings me to two in which you’re not familiar with voodoo lines, music, which you should be you’re hearing this brand new blend of funky ass hip hop that is percussion based. And it oscillates between different genres of hip hop, we can dip into trap we can dip into Lo Fi we can dip into boom bap, boy can we dip into boom bap, I’ve got rack and ass now listen, if I put this next day’s hands, the heat that I’ve got some of these baselines, some of these kicks and snares is going to be harmful.

(music: outro)

Charney Kaye
If you’re a musician out there hearing this and all this turns you on, get at us. Come on in the water’s fine.

Jamal Whitaker
And this isn’t a swimming pool or a lake. This is an ocean of possibilities we’re talking about. We’re getting at something that artists couldn’t even dream of 10-15 years ago, and now is at our fingertips.