Dynamic languages tools

Link: http://blogs.activestate.com/ericp/2007/01/kid_adding_a_ne.html

I got a CD with Komodo Edit on it at XTech, but haven't tried it until now. The recent open-sourcing of the code, and then this post appeared on Planet Mozilla.

It looks like it would be good for DSL development, and playing with it Komodo Edit seems to be very usable. My current editor of choice is SciTE, which Komodo Edit embeds, but Komodo Edit gives projects, and language aware autocompletion and navigation (go to the definition of a function). I haven't found out if it's possible to go to definitions in other files, which would be very useful - I spend quite a lot of time doing Ctrl-C,Ctrl-O,[filename],Ctrl-F,Ctrl-V,Enter to jump around a JavaScript project. It also does polyglot programming - a simple XUL file with a script element switches nicely from XML highlighting to JavaScript highlighting in the script element's content.

The user defined language tool is lexer + state machine based; I'd have preferred it to be a packrat based one, which I've played with before. But I don't want to have to write a whole IDE, I like using the Scintilla editor control, and Komodo's approach seems to get the job done.

As a side note, del.icio.us seems awfully slow these days. You'd have though that Yahoo would be able to give it a couple more servers. I wonder if the competitors are any better.

(N+1)^2 = N^2 + 2N + 1

My mind drifted when sitting on the loo at work the other day, making patterns on the cubicle tiles...



(changed title to more clearly reflect the idea that adding a row to two adjacent edges then one tile creates a new square, which the animation shows if you're viewing in a SVG enabled browser, such as Firefox or Opera.)

XTech 2007 Summary

A short post with the best things -

• Joost demo - I need them to open source their sprite code as well as using the XULRunner base, or I'll have to write the same thing at work. It also would make a better platform for the UML modelling and SAM meta-tool stuff I used to do. Maybe they'll turn up on Mozpad.


• XMPP pushing - Several demos of making interactive applications on top of XMPP, and a cute demo of SamePlaceSuite, which I got my sister to install this evening; Strange Attractor have a fuller write up of the Jabber and ubi-comp talks.


• Arduino hacking - I first got into computers after messing around with analogue electronics, light sensors and making motors go on and off. This is the sort of thing I could make cool stuff for my nieces with, who might get tempted to take things apart. There seems to be a generation of programmers today who don't have any idea of the hardware below, because there's so many layers in modern systems; I don't know how many CS courses still have obligatory hex microprocessor programming sections now that microprocessor instructions are several times as long and have a multitude of addressing modes. 6800 was much simpler than modern cores, Arduino seems simpler still.


• Joins on open data - several talks mentioned using joins across open data sets, for example tags on photos with geolocation data can be used to infer tags for the locations, and also many cases of joins across datasets for space and time visualisation in stamen.com's talk.


• The eponymous stamen.com talk on interactive visualisations - but with very little actual interaction. They create beautiful, information rich applications, many simply play through a time selection and give a video scrub control, and you watch a population of events evolve in a space. The information-software as branch of movie making metaphor.


• Interaction embodiments - Nabaztag and Matt Webb and others, using simple metaphors and littoral devices which span between physical and web spaces. I like physical things that do one thing; gestures and simple control. I hate trying to do anything detailed with a mouse, though a trackpad is ok - something like drawing with your fingertips, and stylus or touch screen is much better. That most of the physical interaction we have with computers is with a soap-on-a-rope always strikes me as odd (they also hurt my wrists if I do use them, but that's another issue). I'm still waiting for the nanotech to get to the state that I can write code in my moleskine and it will execute and report the result on the next line of paper.
  


• Together-apart-together-apart flows in information/interaction embodiments and mediated spaces/private spaces. The dumber the network - and xmpp is pretty dumb - the more diverse the messages; getting information software as rich as print media and interaction as easy as a pocket book is hard. But fun hard.

Pete

Rich Application Platforms

Well, there's been a bit of a kerfuffle about whether or not Silverlight (nee WPF/E) supports binding, widgets and XMLHttpRequest gubbins. And Adobe has open sourced Flex, as well as having had its Apollo platform out for a bit now. XULRunner is still going, being used for SongBird and Joost, and freshly the DVD version of Wikipedia, though whether or not Mozilla is promoting it well as a platform is under discussion.

Having done an awful lot of XULRunner programming in the last few months, there's still a lot to be done before its an easy to use platform. Quite a lot of undocumented interactions cause crashes - particularly race conditions between XBL bindings being attached and scripts in the body of the document being run, and any manipulation of the tree in script during an XBL constructor. But I don't have any idea how well Silverlight or Apollo handle such things, or even if they support anything as flexible as XBL. There needs to be an XULRunner 2.0 that is a lot more predictable in its behaviour, and less arcane in its invocations.

There's also a big question for the sort of system's I'm interested in - can a platform do something like the ClusterBall WikiPedia visualisation? XULRunner SVG grinds to a halt at a thousand or so arcs, and rotated font anti-aliasing is a bit off. Flash, using Haxe, can render ten thousand arcs, but getting text to rotate is a hassle - it requires embedded fonts, and that means more tools, you can't just set the rotation property. There's quite a lot of weirdness in Flash, half of it is object view graph based, the rest procedural, not everything works and it doesn't play nicely with the browser, for example capturing keyboard input and not being part of selections. As far as I know, there isn't an open high-quality 2D graphics library other than anti-grain 2.4 which will scale and isn't, well, a bit weird. But anti-grain has moved to GPL, and isn't hardware accelerated at 2.4, so what will become of it I don't know. I'd really like a good, object oriented language, with a mix of dynamic and static typing, support for mesh tuple spaces, event queues, first class functions and relations, orthogonal persistence, which runs fast enough, and generates portable, high quality visualisations and information displays. None of the options are quite there yet, but XULRunner's JavaScript and SVG seem to be the most sane.


Pete

Prunes. Thursdays. Fortress. Locks.

I'm going to try and get a bit more regular with my blogging. Like every Thursday.

This week, I've got my hands on an iRex Iliad from work so I can read the 400 odd page Fortress language specification (1.0 beta) without killing a small tree.

I like Fortress' traits system, it seems to have enough information for the sort of semantic compiler optimisations and automatic parallelism I'm interested in. Although the language doesn't expose these, it's very similar to some of the sorts of thing I was trying to do at BAE with code generation from a maths model, which led me to start playing with kin. Though kin is relational rather than functional. The reference interpreter isn't a high performance system; if I get some time and the kin interpreter gets anywhere, I may use Fortress as an input language, though I do want to do something more with the relational aspect. I like that its definition of an integer is based on rings and fields rather than bit-string operations. My fiancée is in Romania from May to September, so, instead of just pining away, I may actually stop talking and do something on kin rather than a sequence of 400 line micro-benchmarks.

Both for work and kin I've been trying to get up to speed with these guy's work; the publish-subscribe idiom is used heavily within the virtual machine for the system I'm creating at work, and it seems to spend an inordinate proportion of its time waiting for locks on the properties it's published (I'm still getting to grips with interpreting VTune's output for multi-threaded apps).

kin related I'm trying to think of a nice way of packing R32 and R64 vectors so you can both look up type information and operate on them using SSE. For objects with a reference to their type, if each type is aligned to a cache line, it gives more state bits in the word - 4 rather than 2 for a 32bit aligned pointer, which may be useful. For a primitive field in an object, the field type can be associated with the object's value rather than the field's value, as it can't be aliases, so all code which accesses it has to go through the object's traits first. The cases I want to optimise, such as Array[/ (R32)^4, (ZZ32,ZZ32) /] (a 2D array of 4D 32 bit vectors, which has a very efficient contiguous representation) I imagine would also have access to the type of the array and functions would operate primarily on the array. But mapping a magnitude function onto the Array would need to operate on the type of the element, and I was imagining a fully dynamic dispatch (which then solidifies and inlines the commonest branches). Maybe pushing the type onto the function argument stack would work in those cases, rather than making every value carry its type around. In that case, the array member case and the object field or method local variable cases are all the same - the context you got the value from can give the type of the value, which is either the global ref to the type of a primitive, or the reference to the metaclass of a reference type. That should work for both Fortress and ECMAScript type objects constructed from traits. My current model for running ECMAScript is that each time you set a field on an ECMAScript object to a value of a different type to the type you create a new reference type trait object to describe it; these anonymous traits are reference counted in addition to any memory management so may modify rather than copy previous traits if they are only used by the one object; not original but amenable to having a common runtime for all the sorts of languages I tend to use - hpc, dynamic UI, and relational logic.


TME

Current reading

Following the announcement of Mozilla Tamarin - the adaptation of Adobe's ActionScript virtual machine to give JIT compilation of JavaScript in Gecko applications, and the starting up of the SmallTalk.rb project to get Ruby running on a better VM, I've been doing a bit of reading:

StrongTalk - fast, typed smalltalk
- interestingly, unlike ActionScript, the typing is orthogonal to the speed optimisations
- everything is implemented as mixins, functions which add features to classes

Mirrors - a mechanism for separating reflection and conventional object APIs

JS Typechecking via JS-Sourcerer or a more formal approach which I don't know half the notation of
- JS-Sourcerer doesn't have a model for the interfaces to XUL, and doesn't accept IDL to define it (though it should be possible to create an IDL to its proprietary type definition language)
- there's also a more complete modal type model; I need to investigate more of the theory here, and here.
- at the least you can infer types at any execution point and then assume that changes to object's or prototypes relaxes assumptions
- I need to find whether most type inference systems for dynamic languages are Prolog-like, and whether a Rete-like system is more suitable for dynamic languages
- assuming the cost of the inference actually is worth it in execution terms; it may be better to do some pattern matching then allow constraint violations to push execution back to the interpreter. So choosing an inference engine which is suited to the pattern matching may be more important.
- Type Feedback vs. Concrete Type Inference - maybe TI in the bytecode generator, then TF in the VM?

And, in other news, there's the re-branding of the Semantic Web as Web3.0. Personally, I've been using Web3 for community driven 3D on the web. Let's see if RDF hacking get as popular as World of Warcraft machinima or Virtual Life bartering.


TME

Getting annoyed again

Lately I've been rationalising and refactoring the code base of the large-ish XUL application which I've spent about half my work time this year prototyping.

Which naturally means I'm annoyed with JavaScript this week.

• I really need a type checking JS lint. We're doing demos tomorrow, but after that I'll give JS Sorcerer a whirl. Last Friday I wrote my ToDo list, which currently has 117 items on it, my line manager is talking quite seriously about getting two or three extra staff to share my workload.
  


• Managing packages and dependencies is a little clunky. This is true of all languages I've yet seen, and possibly Java is least worse (I've not done anything real in Common Lisp so don't know, though it seems neither better nor worse). I don't want to have to use a script to build the XUL files to ensure that everything is included; part of using XUL was to make it easily built.
  


• I'm using the observer pattern a lot in the UI. This means virtually entity in the UI uses __defineSetter__ to map each of the object's properties to a structure containing the value and an observer list. I also am getting performance hits where multiple values are changed within a function, so observers are called repeatedly rather than firing them only one the values are all consistent. I really would like a language with transactions, concurrency and observers built in. (I must re-read the Fortress spec to see if it includes observers, or at least hooks on its equivalent of (setf (slot-value foo 'bar) baz) to allow them to be added).
  


• There's nice JS syntax for initialising objects, and mechanisms for prototype inheritance, but there still seems to be repetition in creating constructors for objects with prototypes, lots of function Foo(a, b, c) { this.a = a; this.b = b; this.c = c; } Foo.prototype = new Bar();, where I'd like to get rid of all the this.'s. Maybe some type of prototype + merge/copy rather than classic prototype + constructor.
  


• Probably my own fault for not using E4X, but creating SVG using the DOM is a pain.
  


• Every now and again I still get bitten by XBL's asynchronous loading.

On the other hand, R. is back from paternity leave and so we now have subversion installed, so supporting the users who are using earlier branches should be less stressful.


TME

Better GUIs

Every now and then I go and have a look back at Java land. Yesterday I found this blog entry
John O'Conner's Blog: Better GUIs are one step closer.

It's very painful making good layouts in Java. For my last large Java UI project, which was based on porting a large mainframe ISPF application to run as a desktop application on PCs, I ended up implementing a layout and look-and-feel with most of the CSS box model on top of swing. I would have used XUL for it, but it was a Java shop and adding another platform was too political. There already is CSS look-and-feels in Java, so it shouldn't still be an issue getting things lined up right.

Anyway, I tried to do the same thing in XUL, so that it looks like this on a Mac:





source
css

Now, this took a little while while I remembered that you have to specify widths to ensure each flexible box ends up the same size, but the declarative syntax means you don't really need an IDE and a graphical editor, and the full CSS support means you can skin things if you like.

But that's not the real problem. Better guis are not just better aligned guis - they are concerned with user experience. And if you have to produce that amount of code just to align your fields, then you won't be agile enough to respond to your users.


TME

Back to the Rats

Since it came up again in LtU, and I was thinking about it after Burning Chrome mentioned parsers for syntax highlighting, I was at Henry S. Thompson's seminar on XML schema validation (which is essentially a similar problem to parser design), then Brenden Eich mentioned Narcissus in his closing session at XTech I've been playing with a packrat parser written in JavaScript.

I started again 7pm on Thursday during XTech*, and spent most the night on it, then did more on it on Saturday at BarCamp Amsterdam. Current work in progress is in this directory, with a simple demo which parses a definition of the grammar used to define grammars, then creates divs to render it prettily in the iframe. The grammar below the iframe is the grammar used to parse what's in the iframe, and below it are debug messages. Editing what's in the iframe causes events to mark it dirty, but the incremental reparsing doesn't work yet.

The next stage is to add the incremental reparsing, so when you do edit it it tries to reparse, and to link that to the divs so that you don't have to throw away all of the content of a div when you reparse it, as at present.

The Triep stuff is a combination of Trie with predicates that was another idea for incremental parsing taylored for interactive development. I'm going to add some form of trie to the parse states for autocompletion, though I may go back to using a standard trie as the calculus for the predicates is tricky, and reusing the packrat grammar may mean it's not needed.

One thing about using PackRat using the visitor pattern (or any other tree-walking recursive descent), is that you don't need to convert the grammar into a finite state machine, which means that while Henry S. Thompson's handling of numerical constraints is an elegant solution, the problem simply doen't exist - you can simply put the counter in a for-loop in the visitor's method, and you're done. The parser doesn't do that since most text grammar don't require it - it only handles zeroOrMore, oneOrMore or zeroOrOne, but nToM would be handled in much the same as those, and I don't know in principal why you couldn't use tree-walking descent for XML - my ASN XER to Java data bindings used exactly that approach, and that's a superset of WXD.


TME

Art, state of, one year on from last year (the).

Previous frustrations with XMLHttpRequest, and more recently finding DeltaV didn't appear to be supported even in Firefox at work may be changed if a bit of sensible flexibility gets the W3C spec to conform to the HTTP rfc's extension-method = token rather than a vendor specific white-list.

I'd still really like a browser based, version controlled, graph drawing tool for modelling and knowledge capture, but with the WhatWG's canvas and support for SVG in Firefox stable enough that I'm writing production code based on it, and the real possibility of single page applications such as this wiki using Amazon Simple Storage Solution, I'm thinking of retiring the Java based, serverside image code of my LapisBlue, which I never got round to connecting to a versioned store anyway.

So I'm thinking of retiring LapisBlue, since I'm paying monthly for a full featured server solution that's not getting any use, whereas I can pay for a tiny amount of data storage and get the clients to do the rendering work now. Though proper version control would be nice, saving deltas or labelled versions to S3 should also be possible, more fun that configuring a tomcat installation that pulls in a thousand or so libraries, and not reliant on extension methods as subversion's DeltaV implementation is. What you lose is a queryable database, but I'm thinking of using it for a pattern wiki rather than anything else.

In other news, I got rather exited over the weekend thinking about using SSE for a faster 'byte' code interpreter, and resurrecting kin - my toy language for graph matching based code generators to generate simulation models defined generically on traits, which I'd partly implemented on the JVM - as a scripting language plugin for the gecko platform. If you can SIMD the graph matching, and maybe also either SIMD the bytecode scripting, or (since kin uses pure visitor functions a lot) use SIMD optimised blocks with scripting, you may get close to Java performance without having to track Sun's generics cruft.

It's still easier for me to write Java than C++, especially when you need to use libraries - each library having its own code conventions and memory management model - or Lisp for that matter (since I've done far more Java than Lisp in anger), but for many things JavaScript's good enough. The only things I've found this year that I've written in Java have been something to test an algorithm for work, which could have been written in anything really, and an annealing based graph layout, which ran too slow in JS to be useable. But annealing graphs is exactly what kin would be suited to, and be designed to parallelise it, so it may be that the Java world gets even smaller for me.

I'm not sure how useful web-based simulation tools would be, and suspect a good enough interpreter + a really, really good code generator would be a better match to a lot of the problems I like thinking about than trying to do anything like Sun's Hotspot, brilliant though it is.

Third point of this summary - I'm also excited about building distributed clusters of collaborating applications and services on top of xmpp. It's something I've been pushing at work, and I've got enough of the infrastructure there that the rest of my team are starting to play with it, building models and connecting them to XUL UI's with xmpp pub-sub. I've got till mid June to build it out to a full distributed system with service discovery, which means a mix of quite easy xml binding and doing some fairly hard concurrency work to get the simulators' execution model and the pubsub code working well without excessive threads or critical sections.

Oh, and I'm going to XTech2006 next month. It's nice to be working for somewhere that's not too stingy to send it's people away again.


TME