evan.musing << current

Syntax errors are a part of life for programmers. The language of the computer, no matter how flexible the language, is very picky.

And thus how the language communicates back to the user about what it didn’t understand is important, because time is spent in this phase, no matter the skill level of the programmer.

In Ruby specifically, MRI’s parser (and by extension the melbourne parser Rubinius uses) use yacc, and thus suffer from syntax errors which can be particularly difficult to understand. One particular syntax error that commonly occurs is when there is an ‘end’ missing from an expression. This results in the dreaded syntax error, unexpected $end, expecting kEND message.

Here is a quick example:

class Spaghetti
  class Sause
     def add(plate)
        while more?
           plate << self
     end
  end
end

Now, this is a short example and so spotting the error is fairly easy. But this error typically occurs when you’re working on a 600 line file with multiple classes inside classes and with complicated logic, making it quite difficult to find.

This evening, I decided to try and at least help make this easier to find. So now in Rubinius, rather than

syntax error, unexpected $end, expecting kEND

you get

missing ‘end’ for ‘class’ started on line 1.

Thats a big improvement, because now first off, it’s fairly clearly communicated what is wrong, i.e. that you’ve forgotten an ‘end’. In addition, it tells you what element still required a ‘end’, in this case, a ‘class’ on line 1.

Now, this is far from perfect. It’s pointing you to the element that was unclosed, rather than the one that you actually forgot the ‘end’ on. But it at least is now pointing you to the chunk of code that is the offending code. In practice, this can be a big help.

In the future, it might be possible to try and use indentation to try and narrow down where the missing ‘end’ should be. But for now, every little bit helps.

Got a new harddrive for my wifes MacBook. Couldn’t find her 10.5 upgrade DVD, so I tried to use the install DVD for my new MacBoor Pro. The DVD promptly told me it would not install (Apple cripples the OEM DVDs to only reinstall on the same make of machine), but it did not exit. So I select Restore from Time Machine Backup from the Utilities menu.

Nothing happens.

I try a few times, nothing. I give up for a while. Upon returning and seeing it still doesn’t work, I open the Install Log under Window.

I notice there are number of entries listed: Unable to load XIPanel_RestoreIntroduction nib file. Ok, well something is wrong with the Installer.

Terminal to the rescue. Open it up, do find / -name “XIPanel*” and find a number of them in a Resource folder. Ok, so it’s there, this must be a bug in the installer. Now we get serious. If you’ve gotten this, here’s what you do to get the Time Machine Restore to launch:

• Open Terminal
• Run ps ax and find the listing for Mac OS X Installer.app, note the number of the far left for it (the Pid)
• Run kill pid_of_installer
• You should now get a dark grey background, you’re doing great.
• Run export LANG=en_US.UTF-8. I’m not sure if this matters, but I did it, so you should too
• Run cd “/System/Installation/CDIS/Mac OS X Installer.app/Contents/MacOS/”
• Run ./Mac\ OS\ X\ Installer “/System/Installation/Packages/OSUpgrade.pkg”
• The installer should popup and say Welcome and such (likely with no graphics, thats ok)
• Click Yes/Continue enough for the Utilities Menu to appear at the top and select Restore from Time Machine Backup
• With luck, it will load! You can now do the restore!

Thanks for playing! Booo to Apple for having bugs in their Installer, Yay to Apple for leaving Terminal available in the Installer!

NOTE: Be sure to leave the Installer in the foreground while it restores, otherwise it will stall!

I have to gush about my favorite feature of the Apple TV upgrade. It’s a feature I’ve wanted since Apple TV came out. You can now use an Apple TV as remote AirTunes speakers!!

I love it.

There have been some amazing things happening in the rubinius world in the last week.

• compiler2 (aka. c2) became stable enough that we began using it for everyday usage. This is big in my mind because c2 is vastly superior to compiler1. Also, it shows the power of the approach we’re taking. To my knowledge, we’re the only project to write 2 completely separate compilers for our project. We were able to do this primarily because both compilers have been written entirely in ruby.
• Springboarding off c2 being stable, Kernel#eval() got implemented in really an incredibly small number of lines and in a short amount of time. I began working on it at midnight one night. Worked til 3am that night, then again the rest of the next day. So in about 12 hours, we had a eval that passes the craziest of eval specs. Most of the reason for that was again the approach. Having MethodContext’s be first class objects made getting all the proper info for eval simple. The architecture for c2 made it super easy to add in the custom logic to make eval operate properly. Again, all done 100% in ruby.
• Again, coming off eval working, irb became functional. Getting it running flushed out a few bugs (5 I think), and it runs perfectly. This is big for anyone that has looked at the irb codebase. It contains a lot of meta-magic as well as a large, complicated ruby tokenizer for parsing the code the user types in. This is giant news for me because it means that the project has finally built the codebase and momentum to run real world ruby code. I almost wet my pants when I fired up irb and it worked.
• Galvanized by the success of irb running, I started in on rake. Rake contains a great set of tests to use to make sure rubinius is actually working properly, so I started in on those. I first found that rake’s test use flexmock, another library that jim wrote. So I started on flexmock’s tests instead. That exposed a few bugs, but after about an hour, it was running at 100%. Flexmock uses a TON of meta-magic, so having it work validates a lot of work we’ve been doing. On to rake’s test, again, flushing out bugs. I managed to get everything working except for a few things that depend on our currently broken Dir.glob.
  But in the end, I was able to have rake process our Rakefile properly and spit out the list of tasks and run a few of them. So awesome.
• Lastly, we removed compiler1 completely and renamed compiler2 to compiler. The compiler is dead! Long live the compiler!

Man, it’s been a busy last 7 days. I’m so excited about these developments because as I said earlier, it means that the momentum on the project has built up the point that we’re making real progress running real world code. This is a seminal moment for the project. Up to now, we’ve been running uphill, slogging through specs and trying to get enough code in place. We’ve finally reached the top of the first hill and now we’re running downhill, headed for 1.0. There are a lot more hills in front of us, but clearing the first one means that there is no going back.

My hat is off to every single rubinius committer and person that has helped out on the project, most of all EngineYard. We’d never have reached this milestone in the project this early if it weren’t for all of you. Find me on the street and I’ll buy you a coffee, or in a bar and I’ll buy you a beer.

Enjoy!

So, I got fed up trying to get wordpress to behave nicely on my server, so I’ve moved my blog to wordpress.com, where they’ve got everything already setup nicely.

Let me know if anything isn’t working.

Some of you will notice that blog.zenspider.com has been down for a few days, as have the QuickRef and the rest of the services Ryan Davis (zenspider) runs.

He’s having massive problems with his ISP and all the services are currently hosted from his home. Hopefully they’ll get it ironed out soon, otherwise I’m worried about reading how a Covad/Qwest/Speakeasy support person was dragged into the street and beaten to a pulp.

We all wish him good luck!

Something is screwed up with how I’ve got wordpress installed. I’ve switched it backing to using the ugly permalinks for now, because the nice looking ones cause it to serve up RSS everytime. Still investigating, but at least things work now.

If anyone has had this problem, please comment!

Last night I finished up support for the Task class, which will form a core part of the system.

The Task class provides essentially the same functionality as getcontext/setcontext and setjmp/longjmp in C, with a little more icing to make them more useful. The main functional difference is that when a new Task is created, it gets a new stack to operate on. This allows Tasks to be switched in and out with any other Task at any time.

The other main feature is the ability to associate a Task with a block. When the block is activated, it will begin executing the block.

You can probably see where this is going… thats right, greenthreads. Tasks are going to form the basis for green threading. The main thing that Threads add on top of Tasks is a scheduler, ie, which Task should run when.

Now, A small example:

ts = Task.current

t2 = Task.new do
  puts "2: wee! new task! switching back.."
  t2.swap(ts)
  puts "2: you're returned!"
end

puts "1: activating new task.."
ts.swap(t2)
puts "1: back to the future!"
ts.swap(t2)
puts "1: and we're done."

Who’s output is:

1: activating new task..
2: wee! new task! switching back..
1: back to the future!
2: you're returned!
1: and we're done.

This is a trivial example, where Task#new creates a new Task and associates the given block with this new task. But you can see how as Tasks are swapped back and forth, execution of the task continues exactly where left off.

I’m going to begin work on the Thread scheduler shortly, and I’m expecting to be able to implement it and the Thread class itself directly in Ruby (ie, no C).

The biggest thing missing is synchronization mechanisms. My plan is to add a Channel class which will allow for objects to be passed between Tasks, with some synchronization built-in.

Too long since the last posts, I need to figure out how to make sure I do at least one post a week.

Anyway, since last time, rubinius has evolved more is really beginning to grow into it’s own. The denizens of #rubinius have been working hard, building up the core library. In the meantime, I’ve continued to work on the VM backend.

Performance

I’ve made a number of performance changes in the last month. The first was speeding up how stack frames (MethodContext) were created. They’re now opaque objects so that the VM can manipulate them much more quickly. There is still some work to do in that department as well, mainly in cache contexts rather than just throwing them away everytime.

Next, I went through and made a bunch of changes to how the C code is generated by tweaking the GCC options. That alone sped the VM up by about 200%. GCC now properly inlines the critical VM sections, which is a big boost.

I also added 2 method selector caches. MSC’s speed up method dispatch a lot because they save you have having to traverse all around any number of classes to find the proper method to call. The first cache is a global cache like 1.8 sports currently. It boosted performance by another 100% probably. I’ve now begun experimenting with inline caching. Inline caches stash the last method that was called at a call site so it can be used again. I’ve implemented inline caches for both method calling and access of constants. The inline caches are currently make a small performance improvement over the global cache, but I need to write some good benchmarks to get some decent numbers for comparison.

My current task is to get the interface for calling back into C code finished and commited. It’s going well and is going to use C/Invoke to speed up calling these functions. I’m particularly proud of the design, it’s going to allow the VM to remain stackless (ie, doesn’t use C stack to call methods) yet the C code can call back into ruby code.

Lastly, i’ve created a tumblr. Check it out!

Today I merged the archive loading code from a branch into trunk. The archive loading allows rubinius to load code directly from a .rba (ruby archive). .rba’s are really just zip files with recompiled .rbc inside them. Archive loading is really important for stuff like the kernel and compiler because it allows all the code for them to be contained in one file and used all together.

I’ve also cleaned up the argument’s to ./shotgun/rubinius. The kernel is no longer specified on as the first argument, it’s either detected in a specific directory (currenly just runtime/kernel.rba) or is specified directly in the KERNEL environment variable.

Thirdly, I’ve checked in a kernel.rba and compiler.rba which people can use without going through the hoops of generating their own. I’ve modified the Rakefile so that build:all now only builds ./shotgun/rubinius and therefore uses the checked in version of kernel.rba and compiler.rba.

The upshot of all this change is that it should be a lot easier for people to out of the box get rubinius running simple code.

shell$ svn co http://code.fallingsnow.net/svn/rubinius/trunk rbx
shell$ cd rbx
shell$ rake build:all
shell$ ./shotgun/rubinius -e "puts 'hello rubinius'"

I’ll be restructuring the kernel code in rubinius in the next few days to clean up things further. This cleanup should allow people to work on the core classes and methods a bit more easily.

Thanks to everyone who’s submitted patchs and commited code to the project!