Skip to content

Categories:

URSI Symposium Poster

Summer research is coming to an end. Next thursday I leave to go back home, away from campus for a few weeks before the beginning of the Fall semester. Below is a thumbnail of our research group’s poster, to be presented at the URSI Symposium on September 31 in the Villard Room, Main Building, Vassar College.

Click on the thumbnail below to view the PDF of the poster to explore in full detail. All design, writing, and diagrams, were done by myself. Sheet music generated through Rosegarden, all diagrams drawn in Inkscape, image editing done in Adobe Photoshop, and overall design and layout done in Adobe Illustrator.

2009 URSI Symposium Poster Thumbnail

Posted in Reflections, Theory. Tagged with .

By Anthony
July 24, 2009

No comments

Preparing a MIDI File for Analysis

In my experience thus far working with MIDI files, I have coined a recursive acronym: MIDI Is Drivel, Indubitably. MIDI files are very messy internally, which make them poor candidates for musical analysis and algorithmic generation of music. However, the MIDI format is an industry standard for both performance and composition. JFugue attempts to make working with MIDI files relatively easy with a Java API, but still, when we look at the output of converting a MIDI file called FourBarMelody.mid into a JFugue pattern using:

Pattern pat = player.loadMidi(new File("FourBarMelody.mid"));

Here is our output:

V0 @0 T585 V1 @0 V0 I[Piano] @0 V0 C5/0.3076923076923077a100d127 @720 V0 G5/0.10256410256410256a100d127 @960 V0 G5/0.20512820512820512a100d127 @1440 V0 C5/0.20512820512820512a100d127 @1920 V0 E5/0.20512820512820512a100d127 @2400 V0 F5/0.20512820512820512a100d127 @2880 V0 G5/0.41025641025641024a100d127 @3840 V0 C6/0.3076923076923077a100d127 @4560 V0 B5/0.10256410256410256a100d127 @4800 V0 A5/0.10256410256410256a100d127 @5040 V0 B5/0.3076923076923077a100d127 @5760 V0 A5/0.10256410256410256a100d127 @6000 V0 G5/0.10256410256410256a100d127 @6240 V0 D6/0.20512820512820512a100d127 @6720 V0 E6/0.20512820512820512a100d127 @7200 V0 C6/0.20512820512820512a100d127

Using the above output for analysis would be extremely difficult. It is also highly un-readable. After each note we are presented with a duration that is part string, part decimal. We are given attack and velocity information (represented by the a100 and d127 in each token), which makes converting this pattern into a binary rhythmic pattern impossible.

However, note the MIDI time signals represented by an @-sign followed by a number. Using some arithmetic and some clever coding implementing JFugue’s ParserListener interface, we’re able to turn the above String into this:

C5/0.375 G5/0.125 G5/0.25 C5/0.25 E5/0.25 F5/0.25 G5/0.5 C6/0.375 B5/0.125 A5/0.125 B5/0.375 A5/0.125 G5/0.125 D6/0.25 E6/0.25 C6/0.25

Now how did we get from a total mess to the elegant string presented above? The JFugue API provides both ParserListener and ParserListenerAdapter classes to help create parsers for any purpose. All we had to do was override methods in the interfaces.

public class TimeNoteListener extends ParserListenerAdapter {
 
        /* Instance Variables */
        long lastNoteTime = 0;
        long currentTime = 0;
        String output = "";
 
        /* Overrides the default: 
         * Takes the time from the parser and sets
         * the global variable current Time
         */
        public void timeEvent(Time time){
            currentTime = time.getTime();
        }
 
        /* Overrides the default:
         * Takes the noteValue from the parser, and
         * using the currentTime finds the duration of the
         * previous note.
         */
    @Override
        public void noteEvent(Note note){
            byte noteValue = note.getValue();
            String stringForNote = Note.getStringForNote(noteValue);
            // ignores calculations for previous note for first case
            if(output.length() != 0){
            double duration = (currentTime - lastNoteTime)/1920.0;
            output += duration + " ";
            }
 
            //updates lastNoteTime
            lastNoteTime = currentTime;
 
            output += stringForNote + "/";
        }
 
    public void printParsedString() throws IOException {
 
        MusicStringParser parser = new MusicStringParser();
        Player player = new Player();
        parser.addParserListener(this);
        Pattern simplePattern;
 
        try {
            simplePattern = player.loadMidi(new File("fourBarMelody.mid"));
            parser.parse(simplePattern);
        } catch (InvalidMidiDataException ex) {
            Logger.getLogger(TimeNoteListener.class.getName()).log(Level.SEVERE, null, ex);
 
           }
 
        // calculations of last note duration to complete the last measure
        double remainder = currentTime % 1920.0;
        double lastNoteDuration = (1920 - remainder)/1920;
 
        if(lastNoteDuration != 0){
            System.out.println(this.output + lastNoteDuration);
        } else {
              System.out.println(this.output + "1.0");
        }
    }      
}

So whenever JFugue encounters a timeEvent or noteEvent in the string, it responds accordingly, rendering a Pattern that is easy to manipulate and analyze.

Posted in Code. Tagged with , , .

By Anthony
July 13, 2009

1 comment

Fuzzy Matching

Last week we created a pattern matching algorithm which we later instantiated as the “FuzzyMatcher” class. The FuzzyMatcher works as follows.

Let’s say you perform a musical phrase on a MIDI keyboard, and I have a database of known rhythm patterns (generated algorithmically, taken from Schillinger’s Encyclopedia of Rhythms). How closely does what you play match the patterns in the database? Using the MusicStringToBinaryPattern converter you obtain a rhythm pattern of:

1001010010

Now match the above pattern with a BinaryPattern from the Schillinger rhythm database. For example, the resultant of 5:4 in 5/8 time.

10001100101010011000

Now look at them, one on top of the other:

1001010010
10001100101010011000

Look at the shorter pattern. Every time we get a 1 with a 1 or a 0 with a 0, we have a match in the pattern.
We can also test the pattern at different offsets. For example, with an offset of three, we start comparing at the third index of the Schillinger pattern:

   1001010010
10001100101010011000

Below is the code for finding the number of hits given two arrays and an offset.

  public int findHits(int[] longer, int[] shorter, int offset) { 
        int hitCount = 0;
 
        for (int i = 0; i < shorter.length; i++) {
            if (longer[i + offset] == shorter[i]) {
                hitCount++;
            }
        }

We can extend this method with another method, offsetMatch, that will return an array with indices representing various offsets, and values of percent matches:

 
 public double[] offsetMatch(int[] testArray, int[] schillingerPattern, int patternIndex) {
 
        this.patternIndex = patternIndex;
        int[] longer;
        int[] shorter;
 
        if (testArray.length > schillingerPattern.length) {
            longer = testArray;
            shorter = schillingerPattern;
        } else {
            longer = schillingerPattern;
            shorter = testArray;
        }
 
        double[] offsetArray = new double[longer.length - shorter.length + 1];
 
        for (int offset = 0; offset < (longer.length - shorter.length + 1);offset++) {
            offsetArray[offset] = ((double)findHits(longer, shorter,offset))/shorter.length;
        }
 
        return offsetArray;
 
    }

From this we can use a findMax method to find the best match within the pattern at the corresponding offset:

  public Match findMax(double[] t) {
 
        double maximum = t[0];
        int offset = 0;
 
        // start with the first value
        for (int i = 1; i < t.length; i++) {
            if (t[i] > maximum) {
                maximum = t[i];   // new maximum
                offset = i;
            }
        }
 
        Match match = new Match(maximum,offset,patternIndex);
 
        return match;
 
    }

We’ve tested a few different patterns taken from real music and had some surprising results. We had several perfect matches within the patterns at various offsets, and otherwise finding large percentages of hits within the patterns. Keep posted.

Posted in Code. Tagged with , .

By Anthony
June 23, 2009

No comments

Converting a JFugue MusicString to a Binary Rhythmic Pattern

To construct rhythmic patterns from a JFugue MusicString, I have created a class called MusicStringToBinaryPattern. The purpose of the class is to look at each note within a MusicString, analyze its duration, and subsequently generate an array of 1’s and 0’s that indicate what Schillinger refers to as “points of attack” (indicated by 1’s). Let me illustrate with an example.

A test string is constructed with a 4:3 rhythmic pattern, represented by:

"C5/0.375 C5/0.125 C5/0.25 C5/0.25"

The duration of each note object is represented by a decimal value: 0.125 = an 1/8th note. First, each note is examined individually, returning an array of Double valued constants:

   public Double[] getDurations(String s) {
 
        Pattern pat = new Pattern(s);
        MusicStringParser parser = new MusicStringParser();
 
        String[] tokens = pat.getTokens();
        Double[] durationArray = new Double[tokens.length];
 
        // fill array with decimal durations
        for (int i = 0; i < tokens.length; i++) {
            durationArray[i] = (parser.getNote(tokens[i])).getDecimalDuration();
        }
 
        return durationArray;
 
    }

The output of this function for the example string would then be:

0.375
0.125
0.25
0.25

Because the patterns we are looking for are related to the ratios of the durations and not the durations itself, we create a findMin() function to search for the smallest value within the array:

public double findMin(Double[] nums) {
        // Use curMin to keep track of the smallest value
        // that has been seen so far...
        double curMin = nums[0];
 
        // Examine each number in nums in turn and see
        // if it is smaller than the smallest number seen
        // so far.
        for (int i = 1; i < nums.length; i++) {
            if (nums[i] < curMin) {
                curMin = nums[i];
            }
        }
 
        return curMin;
    }

The smallest value of this array being, obviously, 0.125. This minimum value becomes what we will call the “multiplier” value. Each element within the array is divided by the multiplier value, and an array of integers is returned. Dividing each element by 0.125, we obtain an array of integers:

3
1
2
2

So we have three eighth notes (a dotted quarter note), followed by one eighth note, and two two-eighth note durations (or two quarter notes). Subsequently we construct an array of 1’s and 0’s taking on the pattern:

public int[] convertMusicStringToBinaryPattern(String s) {
 
        Double[] arr = this.getDurations(s);
        Double multiplier = this.findMin(arr);
        int[] intArr = new int[arr.length];
        int patternLength = 0;
 
        // convert decimal duration to integer ratios
        for (int i = 0; i < arr.length; i++) {
            intArr[i] = (int) (arr[i]/multiplier);
        }
 
        // create an array with a length equal to the sum 
        // of all the elements of intArr
        for (int i = 0; i < intArr.length; i++) {
            patternLength += intArr[i];
        }
        int[] patternArr = new int[patternLength];
 
        // fill array with 1's and 0's
        int index = 0;
        for (int i = 0; i < intArr.length; i++) {
            patternArr[index] = 1;
            index++;
            for (int j = 1; j < intArr[i]; j++) {
                patternArr[index] = 0;
                index++;
            }
        }
 
        return patternArr;
    }

And the array output looks like the following:

1
0
0
1
1
0
1
0

Posted in Code. Tagged with , .

By Anthony
June 3, 2009

No comments

Reversing a Musical Pattern

Before I can begin work on analyzing and rearranging the rhythmic phrases of a composition, I am going to need to write a lot of simpler tests before more complex work can begin.

One example of a pattern transformation is the reversal of a pattern. This is simple in JFugue, and especially so because the following snippet is given in the guide:

public class ReversePatternTransformer extends PatternTransformer {
  public ReversePatternTransformer() {
    super();
  }
  public void voiceEvent(Voice voice) {
    insert(voice.getMusicString(), " ");
  }
  // Most of the other events follow the same steps as
  // voiceEvent: insert(element.getMusicString()," ");
  public void noteEvent(Note note) {
    insert(note.getMusicString(), " ");
  }
  public void sequentialNoteEvent(Note note) {
    insert(note.getMusicString().substring(1, note.getMusicString().length()), "_");
  }
  public void parallelNoteEvent(Note note) {
    insert(note.getMusicString().substring(1, note.getMusicString().length()), "+");
  }
  private void insert(String string, String connector) {
    StringBuilder buddy = new StringBuilder();
    buddy.append(string);
    buddy.append(connector);
    buddy.append(getReturnPattern().getMusicString());
    getReturnPattern().setMusicString(buddy.toString());
  }
}

Using the snippet above, I wrote a program that takes a predefined pattern, finds its reverse, adds the two together, and plays the result:

public class NoteTransformerTester {  
  public static void main(String[] args) {
    Player player = new Player();
    ReversePatternTransformer rpt = new ReversePatternTransformer();
    Pattern pattern = new Pattern("C5q C5q G5q G5q A5q A5q Gq");
    Pattern transformedPattern = rpt.transform(pattern);
    pattern.add(transformedPattern);
    player.play(pattern);       
  }
}

Not too impressive, but it might be a step in the right direction.

Posted in Code. Tagged with , .

By Anthony
January 20, 2009

No comments

JFugue for Musical Pattern Analysis

A few days ago, I found what could possibly be the ideal platform for implementing the Schillinger pattern analyzers and generators needed to develop the Composer’s Intelligent Assistant. It’s called JFugue, which, as it claims on its home page: “is an open-source Java API for programming music without the complexities of MIDI.”

MIDI files are not easily human-readable. Upon opening a MIDI file in emacs, instead of easily recognizable notation, we are given mostly MIDI time signals that consist only of “on” or “off” signals. Imagine trying to explain in plain language how the snippet below is supposed to somehow represent an excerpt from Beethoven’s String Quartet No.1 in F, Op.18 No.1:

1
2
3

\224;^@@\22495 \2249^@@\22477 \2247^@@\22469 \\246^@@\2247;
\2247^@@\2244= \2244^@@\2246? \2246^@@\2247A \2247^@@\2249C
\2249^@@\2247E \2247^@@\2246G \2246^@@\2244I \2244^@@\2242-\201

Now, imagine teaching a computer to read that, trying to write a piece of software that could parse the above three lines of data for rhythmic patterns (see previous post), intervals, chord changes, voicing, etc.

The verdict is clear: MIDI files in plain text are not well suited to represent written music and furthermore, present challenges to analyzing a MIDI file for patterns.

JFugue, on the other hand, handles MIDI as pure data and reads musical events as objects in memory that each can be analyzed, combined, transformed, and repeated in a human-readable format: MusicStrings. More generally, JFugue provides a high-level Java API for musical analysis and programming.

The following snippet from the enormously helpful Complete Guide to JFugue, written by JFugue author David Koelle is one example of how to extract musical data from a pattern:

1
2
3
4
5
6

 public List<Instrument> getInstrumentsUsed(Pattern pattern) {
        MusicStringParser parser = new MusicStringParser();
        parser.addParserListener(this);
        parser.parse(pattern);
        return instruments;
    }

Using this function in a separate class, I wrote a short program that listed the names of the instruments used, which proved slightly buggy, but nonetheless a very intuitive interface to an otherwise inevitably beastly MIDI file (which is thousands of lines longer than the three line excerpt, which (my conjecture) probably represents a second or two of music at most (the file I opened in emacs was far longer and this particular midi file is only the first movement of a string quartet, so probably no more than a few minutes of music in the file).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

public class instrumentLister {   
  public static void main(String[] args) { 
    GetInstrumentsUsedTool fetcher = new GetInstrumentsUsedTool();    
    try {
      Pattern myPattern = patternFetcher();
      List instrumentList = fetcher.getInstrumentsUsed(myPattern);
      for (Iterator iter = instrumentList.iterator(); iter.hasNext(); ) {
        Instrument instr = (Instrument) (iter.next());
        String name = instr.getInstrumentName();
        System.out.println(name);
      }
    } catch (IOException e) {}
    catch (InvalidMidiDataException e) {}
  }  
  public static Pattern patternFetcher() 
    throws IOException, InvalidMidiDataException {
    Player player = new Player();
    Pattern pattern = player.loadMidi(new File("trio.mid"));
    return pattern;
  } 
}

I am still very much new to JFugue, and have only a semester of Java under my belt, so I have yet to program detection of rhythmic patterns, voicing etc. in JFugue, but as I spend more time with the API and complete guide I see many possibilities for what can be done with this library.

Posted in Code, Reflections. Tagged with , , .

By Anthony
January 14, 2009

No comments

Schillinger Rhythm Patterns

The goal of this research project, in part, is to develop a Composer’s Intelligent Assistant using the Schillinger System of Musical Composition as a theoretical backbone. The system is a method of composing music using mathematical processes, or, in other words, patterns.

Rhythm serves as the basis for the Schillinger system. Traditional musical notation tells us that the formation of rhythms is a process of subdivision, with its basis in “the beat.” However, Schillinger calls for us to revise our notion of rhythm to think in terms of multiplicity and interference. These notions are best explained by example.

Suppose, in a given piece of music, the finest granularity of rhythm we can detect is the eigth note. Let us then quantize the eighth note to give it a value of 1.

(Think: eighth note = 1)

We can extend this notation to say that a quarter note would then be 2, a dotted quarter note would be 3, a half note 4, etc.

Schillinger tells us that we can express any musical rhythm using repeating patterns of quantized rhythmic values. We do this initially by picking two relative prime numbers, for example, 4 and 3 (meaning they each is indivisible by the other)

Then, we need to find the least common multiple of these numbers to find out when these two rhythmic values will exactly intersect, and therefore form a pattern, so we multiply them.

4 x 3 = 12.

Ok, now we have twelve points of attack, let’s draw them on a grid, as Schillinger does:

twelve

Now, draw two lines just below, for every 4 points of attack and every 3. A picture tells a thousand words:

simplepattern3

Now, what pattern do we have? The result shows that we have a point of attack for the entire first three boxes, then a point at the fourth box, then another separate point for the fifth and six, etc. More explicitly, in traditional notation the rhythm would look like:

simplepatternnotation

May not look like much for now, but if you start listing pairs of primes, multiplying and doing this out by hand, more complicated patterns do emerge. Additionally, Schillinger goes on in his theory of rhythm to explain many more techniques for pattern making than described here. I’m still learning them!

(Schillinger diagrams drawn in Inkscape, notation done through TuxGuitar)

Posted in Theory. Tagged with , , , .

By Anthony
January 12, 2009

1 comment

The Composers Intelligent Assistant

Yesterday Prof. Jones and I had our first meeting. We outlined the main goals for the research project. It is our goal to begin the building of a framework for developing a Composer’s Intelligent Assistant (CIA). The CIA will serve as an AI enhancement to a composer’s output. The basic idea is that a composer would input a musical phrase via midi, and the CIA would first analyze the phrase for special patterns of rhythm, harmony, etc. This information would form the basis in determining the style of the composer. More phrases of a composers own output serving as input to the CIA would be further analyzed and synthesized in order to give a better idea of the stylistic traits of a particular composer, as given by the pattern matching. This could then be used to form stylistic generators that could assist the composer in performing the mundane task of repetitiously varying phrase input to fill in the gaps (so to speak) in the composition that s/he wants the assistant’s help on. This is not to show the limitations of the composer, but rather to open up all the possibilities of composition, possibilities that otherwise are masked by simple fatigue. Composers of music (at least as regards music that relies on tonality and rhythm, the works of John Cage would probably not be able to be stylistically imitated by an intelligent assistant) are often bogged down in the hum-drum task of ‘trying something else’ when it comes to writing their compositions. The Composer’s Intelligent Assistant seeks to liberate the composer from the mundane task of combination and permutation, so that all the composer has to do when looking for a means to phrase completion or structural gap-filling is to listen to the possibilities and choose which one s/he likes best.

Posted in Reflections.

By Anthony
January 9, 2009

No comments

What the heck is “musical artificial intelligence” anyway?

To ask ‘what is musical artificial intelligence?’ is to simultaneously pose several other questions. What is artificial intelligence? What does it even mean to be intelligent? But most of all, what does it mean to be musically intelligent? I think to ask this last question is to give a sense of what researchers in “musical artificial intelligence” are trying to grapple with.  Musical artificial intelligence might be said to be the field of inquiry uniformly spanned by a question of musical intelligence: how can we engage the practice of musicmaking from the standpoint of the musicmaker?  How do we compose music? How can we compose music to engage the soul? What then should separate musical AI from musicology, anthropology, and music theory, is its approach in engaging these questions.

Posted in Reflections. Tagged with , , .

By Anthony
January 1, 2009

No comments