ActionScript.org Flash, Flex and ActionScript Resources - http://www.actionscript.org/resources
From Fruity Loops to Actionscript
http://www.actionscript.org/resources/articles/1056/1/From-Fruity-Loops-to-Actionscript/Page1.html
Alex Nino

My bio... hmmm... what to say... I arrived on this crazy world clearly by mistake on March 81, in a far, far forgotten small town with more problems than a whole nation, close to the pacific ocean in the Colombian territory, it is called Cali. Early I began playing with some gadgets like "telebolito", "intellivision", "Atari", and it was here when I started feeling interested in games and how they are being done but before start programming I've learned to play piano a guitar (I love music, can't live without it). Then I learned some programming languages such pascal with Borland Delphi, and C++ but I have to confess that it was Macromedia Flash who changed my life once we met each other in 98. I'm working with it since it didn't have support of mp3 audio format files, yes the version 3. And then together (Flash and I) started travelling around south america looking for a better job opportunity and professional development than my hometown could offer us at that moment. Special thanks to: Brazil, Costa Rica and Colombia for being a great school. At this moment I'm happily living and paying taxes in London, UK but like a salmon fish one day I will be back to procreate :)

 
By Alex Nino
Published on December 5, 2010
 
About 3 years ago, when actionscript 3 was born, Andre Michelle came with the idea of injecting audio data into a ByteArray (binary SWF structure) for generating dynamic audio, I did have a go with this approach for creating my own music dynamically (on the fly) based on pulses previously stored in a MIDI file, which it worked fine but it wasn't too good in terms of performance and synchronization, especially creating long sequences by linking audio files where easily all of them get out of synch in slow machines. So I forgot about all this till Adobe has introduced into the flash player platform (FP10) the very famous SampleDataEvent.SAMPLE_DATA, giving us lots of ways for generating dynamic audio. I did some experiments exporting music from FruityLoops to Actionscript.

Working with MIDI files and Flash (actionscript)

About 3 years ago, when actionscript 3 was born, Andre Michelle came with the idea of injecting audio data into a ByteArray (binary SWF structure) for generating dynamic audio, I did have a go with this approach for creating my own music dynamically (on the fly) based on pulses previously stored in a MIDI file, which it worked fine but it wasn't too good in terms of performance and synchronization, especially creating long sequences by linking audio files where easily all of them get out of synch in slow machines. So I forgot about all this till Adobe has introduced into the flash player platform (FP10) the very famous SampleDataEvent.SAMPLE_DATA, giving us lots of ways for generating dynamic audio.

Dynamic audio in Flash

In my previous article speaking about "interpolation" I have made a Piano Keyboard of 6 octaves as proof of concept of reusing one single piano key by changing its pitch for generating the rest of the keys. Well, that was a good practice but it is a bit useless if we cannot have any control on it apart of clicking the keys with the mouse. A friend of mine said to me, you should enable the keyboard and map the keys, say A,S,D,F,G,H,J and use the W,E,T,Y… for semitones but that was still useless for me, I mean, the approach of it was just explaining how the hermite interpolation works and not making a synthesizer that everybody can play and feel great with.

Actually there are some online audio applications made in flash like the hobnox (audiotool) which contains many crazy synthesizers modules made natively in actionscript plus a very complex sequencer and a huge set of different effects. The first time I saw it, I said Wooooooooooowwwww but I have to confess that once I've started using it, I didn't stick to it for long and I jumped back to Cubase, Logic, Reason, FruityLoops. Forgive me if I'm making a very wrong comparison; I am not saying that it is a bad tool, No! It is perhaps the most complicated online flash software I have ever seen in my entire life which is great for the portfolio of the people who have been involved in that project (especial thanks to Andre Michelle, Joa Ebert and the people who made it, it is really brilliant, a great idea… I particularly think that the SampleDataEvent.SAMPLE_DATA exists thanks to you guys). The reason why I said that is because of I am so used to desktop audio applications where we have audio hardware acceleration ASIO, DSP, midi interfaces, very fast UI response, plus lots, lots of resources, thousands of people making plugins, effects, synths, etc. That's why I found difficult to feel familiarized with the audiotool, I think it is still in progress and perhaps in the future it will become an online audio tool as good as any other desktop audio software, the last time I saw it was promising a lot, I can imagine how complicated this flash application is.

From Fruity Loops to ActionScript

Well, unfortunately there is not an easy way of doing this transparently, import an audio project into actionscript from an audio software is something very complicated, apart of generating lots of WAVs of course. Especially all the audio that is being processed dynamically by effects, synthesizers modules, etc. Most of these audio programs used hardware acceleration and against that we can't do much. I have tried many different ways to import my audio projects into Flash but I always end up with exporting lots of WAVs which is not good in terms of swf-file-size, and unfortunately It seems to me that this step is unavoidable unless you want to make your own synthesizers in actionscript natively, similar they did it in the audiotool (crazy).

The middle term

Well the middle term is reusing as much as we can waveforms (.wav) and apply some easy (non expensive in terms of CPU) effects on top of them, so we can have at the end a good sequence filled of good samples, change the pitch of them on the fly, with some filters, etc… it is not too bad! And it is still CD quality and it sounds good! And the best part of all this is that it will sound the same in all computers, in all platforms (the flash player platform magic).

Playing Midi Files

Well , some people don't know exactly what midi is, and some of them are very confused about General Midi and MIDI itself. Midi means pulses (messages), it is a communication protocol used in many different areas (not only in music), did you know that even some medical equipments at hospitals use midi? MIDI is a very light, old and solid protocol for communicating computers, mixers, consoles, musical instruments, etc. find out more about it at www.midi.org

Now, General MIDI is something 100% related to music, it consists in a set of musical instruments (128 in total) which includes some traditional instruments such as piano, guitar, flute, trombone, trumpet, etc… and also it includes a Helicopter sound, a gunshot, stadium crowd, etc… don't ask me why. First time I saw General Midi was when I bought a workstation keyboard earlier on 90's (KORG X3). I used General Midi by that time and then I was a bit disappointed when I tried to play my songs later on in another keyboard (Roland XP50). It sounded completely different, a different thing.

General midi is a good approach if what you want to do is show an idea about how a song sounds like, it uses the MIDI messaging protocol for playing each of the set of 128 instruments, MIDI lack of sounds (it is only messages), all these instruments are implemented internally by the device (machine) and played when processing MIDI messages. It has a big disadvantage and it is that it doesn't guarantee that your music will play and sound the same in all devices (platforms), so there is not fidelity at all by using General Midi format. It is like HTML earlier on 90's, it looks different between browsers, General MIDI sounds different between devices and all this is sometimes annoying.

So, let's forget a bit General MIDI and focus in the important part, the MIDI communication protocol. Please take a look on the following video for understanding a bit more how all this works in Flash

and then, let's jump to the examples at the second page


Too much talk (examples)

Well, as I said before I've been working in a keyboard piano where I change the pitch of one note (A440) for generating the rest of the keys (please read the interpolation article to understand more about it). So, in the Flash SWF below I did a similar keyboard, this time using 5 different waveforms playing .mid files, using up to 5 midi channels, each channel per waveform. Don't misunderstand it, This is not a General MIDI reader, it only has 5 waveforms used in 5 channels when reading .mid files. It was a very good practice, here I developed my own MIDI files reader (it was a pain, bad documentation out there) and I mixed it with some Bitmap particles animation, the entire thing taught me a lot of synchronizing audio with animation using MIDI files, check it out! feel encouraged to play your own midi files by using the last option "load locally a midi file".

Can you see how all this could be applied in the real world?

Well, some people who I work with asked to me, what is that for? Maybe this is not clear enough. This is a proof of concept; it can be used in many different scenarios where you have to deal with audio and you require control on it. For example, have you ever wonder what's the easiest way of making a guitar hero game in flash? Well using this approach it could be done very, very easily. Let's get into the code, in the example below I'm explaning how to read a midi file (one single channel) and play it back by using your own waveform sample. In this case I'm using the same waveform I used in the interpolation piano keyboard article:

Yeap, here is the source code of it.

An imported song from Fruity Loops

This is a bit more interesting approach. In this example I have done a quick song in Fruity Loops as proof of concept; the total song length is 3 minutes and 15 seconds (at the end it doesn't really matter how long it is). The important thing is how many instruments (including voices) your song uses and how complicated those instruments are; there are some VSTi and DXi instruments highly detailed that makes everything harder when exporting their samples, you have to do it manually one by one.

How does it work?

Basically you make your own song (music arrangement) by using any kind of audio software (fruity loops, cubase, logic, reason, etc...) as you do them normally and then, at the end you have to analyze and detect which parts of your song are you reusing the most, being the most specific possible for example: in a drums set how many times in your song did you use the Kick, HitHats, Toms, etc...? Everything consists in export them separately as waveforms (.wav) and assign to them a specific note in a specific MIDI channel. The same for the harmonics; piano, bass, synths... what you have to export is only one note A4 (A440) and then compose the chords through MIDI notes.

Once you have done this, you will notice that your song will sound similar to the original one (based on VSTi and DXi intrumsnets), try to include all effects applied on your instruments like reverbs, flanger, chorus, etc, when exporting waveforms. The more waveforms you use the nicer your song sounds, but also the bigger your .swf file will be at the end, so this approach may not work well on very complicated songs, normally when using acoustic instruments I prefer just use the full song as a mp3 file and forget about all this.

It really helps to save time if while you're creating a song you keep in mind that you will have to split it into small tiny parts (waveforms).

A song 3 minutes long (400KB) High Quality, Stereo

This is a very good example of exporting a song from Fruity Loops, I could make this song even 10 minutes long and it won't affect the size of my .swf file, again the important thing are the samples.

So, here we are. I made a quick song and I exported its samples with a MIDI file that plays these samples in certain order, into my SWF are (all of them 44.1KHz Stereo 112 kbps MP3)

  • synth.wav, 28.47KB
  • rhodesAMinor.wav, 39.79KB
  • rhodesAMajor.wav, 37.52KB
  • bass.wav, 7.45KB
  • flute.wav, 226.71KB this is the most expensive instrument in terms of file size
  • Kick.wav, 4.21KB
  • snare.wav, 4.54KB
  • hithat2.wav, 1.62KB
  • hithat1.wav, 2.27KB
  • clap.wav, 5.83KB

Put you headphones on and play it!

And here is the source code of it.


Guitar Hero concept using MIDI

Guitar Hero concept using MIDI

Hehe! This is very interesting, in this example I'm not generating any audio at all, it is just a Mp3 file playing in the background (synchronized with a MIDI file). This example works completely in the other way round, the sound rely totally on the animation and the animation rely on FPS (frame rate), If the animation get stack the sound stops completely. In this case what I did was just placed some notes in fruity loops on top of the mp3 and them generate a MIDI file to control the animation synchronized with the music.

This demo didn't take long, no more than couple of hours, it so easy to build a Guitar Hero logic based on Midi files. Each finger is a note and each instrument is a MIDI channel, that's all! Take a look.

Yeap, here is the guitar hero concept source code.

Well, I think it is all for now, feel free to contact me if you would like to clarify any doubts you have about MIDI and Flash, see you!

Resources

Finally I got the chance for writing some documentation about all this MIDI files research in flash.

check the online detailed documentation Midi Files Reader
some demos midifiles examples, feel free to ask if you have any doubt.