CF8: AHAH!

In my last few posts, I looked at the mechanisms CF8 provides to get data into the browser. Just getting data is not enough, of course – you still need to plug that data into elements in the browser’s HTML DOM tree so that is is made visible to users, allowing them to view and interact with it. This calls for JavaScript skills, and possibly the use of toolkits like jQuery or Prototype to ease the task. Such applications can be extremely powerful and interactive, but do require a lot of custom JavaScript code.

Your average Web 1.0 application performs tasks like this on the server – hence the plethora of templating languages, designed to allow programmers to plug data into HTML markup. ColdFusion was the first, of course, and as we like to modestly claim, still the best! 😉

It seems inevitable that someone would come up with the idea to combine these two approaches, asynchronously fetching markup from the server, and plugging that markup into the HTML DOM tree in the browser. It’s a pretty neat idea – let the server process the template for a given fragment of HTML, which allows programmers to code in a familiar server-side templating language, use AJAX to fetch that markup fragment, and push it into the browser’s HTML DOM tree. Given the need for a catchy Web 2.0 acronym, the technique has come to be called AHAH, short for Asynchronous HTML And HTTP, pronounced “a-ha!” or perhaps “ah! ah!” – I’ll leave the choice to you, dear reader.

CF8 provides support for AHAH in a number of different ways – with the ColdFusion.navigate() JavaScript API, the various AJAX “container” tags (CFDIV, CFLAYOUTAREA, CFWINDOW and CFPOD), and the AjaxLink() and AjaxOnLoad() built-in functions. The “container” tags are so called since they act as containers for HTML markup, allowing the contained markup to swapped out for new markup fragments fetched from the server. In my next few posts, I’ll talk about the functionality these tags and functions offer, and how you can use them in your CF8 AJAX applications.

So far, I’ve discussed templating on the server, and using custom JavaScript to plug data into the HTML DOM tree on the browser. There is also a third approach – templating on the browser. To date, the only framework that I know of which accomplishes this is Spry. So here’s another way to build your AJAX applications – use Spry JSON datasets in conjunction with CF8 JSON web services to pull data into the browser. But more on that later…

CF8: Of Star Wars, Layer Cake and CFAJAXPROXY

Now that I’ve covered the server-side JSON support we have in place, it’s time to talk about something a little different – the client-side tools that CF8 provides to leverage the stuff we’ve put in on the server.CF Developer Encased in Carbonite

If you’re a CF developer, you’d have to be encased in carbonite not to have heard that CF8 adds a slew of new tags and functions to help you build AJAX applications. The first of these to be built was the CFAJAXPROXY tag.

Once automatic JSON<->CFML serialization and deserialization was built into CF’s remote CFC invocation layer, the next logical step was to provide a mechanism that would allow invocation of CFC functions remotely from a web browser in the simplest manner possible. The easy way out would have been to just write a JavaScript function that works like CFINVOKE, perhaps something like this:

ColdFusion.invoke(<Path to CFC>, <Function>, <Arguments>)

We did consider it, but it just didn’t feel natural enough… Continue reading “CF8: Of Star Wars, Layer Cake and CFAJAXPROXY”

CF8: Evolving JSON Support, Part 2

In my last post, I looked at the new JSON functions available in ColdFusion 8, and showed how they could be used to consume JSON web services. Today I’ll look at how CF8 lets you expose your applications as JSON web services.

Even with just the CF8 JSON functions, it is easy to create JSON web services – consume arguments from the caller (in the URL or FORM scopes) in JSON format, deserializeJSON() those to CFML types, run your logic on the CFML objects, then serializeJSON() your return type to get the data back to the caller in JSON format. Easy to do, from a CFC or a CFM.

But with our credo of making hard things easy (and easier!) that just wasn’t good enough.

Lost In Translation?

I’ve always liked the notion that the transformation from and to data formats for communication should be independent of application logic. Taking that a step further, I like to think that application designers should not need to worry about data formats (well, not too much, anyway!) – the client of a web service should be able to decide the data format in which it wishes to communicate, depending on its capabilities. Perhaps one client likes to talk SOAP web services, another prefers XML, and, perhaps, is willing to play with WDDX, and yet another chooses to chatter away in JSON. Once an application designer decides that a particular bit of logic should be exposed as a web service, why should it matter what format the data is exposed as?

RESTful Web Services

Before I get into what we’ve done with CF8 in this space, a little background first – did you know you could invoke CFC functions over HTTP? You should – to my mind, it’s one of the coolest things about CF, which I’ve written about before. Here’s how it works:

http://myhost/my/application/callme.cfc?method=echo&input1=yodel&input2=hey

Simple – form a URL to the location of your CFC, and pass it the function to invoke in the method request parameter (URL or FORM scope). Other request parameters are provided to the CFC as arguments. Of course, you have to be sure that the CFC function you’re invoking is marked access=remote, or CF will not allow you to call it remotely, as a web service. By default, the return format is WDDX – CF will automatically serialize the return value from your CFC to WDDX and return that to the caller. Arguments can be provided to the CFC in complex nested CFML types using the special request parameter argumentCollection, which can take a WDDX-formatted string, which must represent a CFML struct, and is deserialized to form the arguments scope for the CFC function to be invoked.

In the currency of the dynamic web, CF makes it simple to expose CFCs as REST web services. If you are planning to design web services into you applications, and don’t know what REST is, you should – REST, or REpresentational State Transfer, outlines a way of thinking about web services architecture using the mechanisms built into HTTP, rather than reinventing the wheel a la SOAP web services. For more see Roy Fielding’s Ph.D. dissertation (Roy Fielding originated the term, relevant chapter here), and for an easier, more fun, introduction, see Duncan Cragg’s REST dialogues. Personally, I’m a huge fan of REST web services – while SOAP has it’s place, I think it really is needlessly bloated and overly complex for the vast majority of use cases it is applied to.

Talking Tongues

Getting back on topic, CF8 introduces the new returnFormat attribute for HTTP invocations of CFC functions, which allows a caller to specify the format of data they’d like to get back. returnFormat supports the following options:

  • wddx – the default, return data as WDDX
  • json – serialize the return value from the function to JSON and return it
  • plain – do nothing with the return value from the function, return it as-is. Note that the plain return format works only for simple types or XML documents. If you try to return a complex type (struct, array, query) with returnFormat=plain, CF will throw an error, since complex types cannot be returned as-is; they must be serialized to a string representation (WDDX or JSON) to transmit over HTTP. Also note that if you’re returning a XML document, you should remember to set the HTTP response content type appropriately – text/xml for most cases, or as appropriate for the particular XML dialect that you’re returning.

Taking my example above, a user who wants a JSON response would modify the URL like so:

http://myhost/my/application/callme.cfc?method=echo&input1=yodel&input2=hey&returnFormat=json

To flesh things out, CF8 can also accept JSON in the argumentCollection request parameter. For instance:

http://myhost/my/application/callme.cfc?method=echo&argumentCollection={"input1":"yodel","input2":"hey"}&returnFormat=json

For completeness, returnFormat is also available as a CFFUNCTION attribute – if you would rather your remote functions had a default other than WDDX you can set it on your functions, and save callers from have to specify it in their requests.

So what’s it good for?

For starters, you can now invoke CFCs directly from AJAX applications – there any any number of AJAX frameworks, including Spry, jQuery and Prototype, which provide support for consuming data from JSON web services. Oh, and lest we forget, CF8 offers up the cfajaxproxy tag to aid in this effort – but more on that later.

Or, if you buy into the “web as a platform” meme (I know I do!), you could choose to expose public JSON web services from your applications, as do del.icio.us, and many others. I particularly like the fact that services such as del.icio.us, built on the backs of their users’ tag gardening activity, open up the data for others to access and do with as they will. Then there’s the it-makes-commercial-sense approach, with companies such as Amazon leading the way – Amazon Web Services provide an astonishing array of functionality (XML rather than JSON, but that’s besides the point), providing pay-as-you-go services, such as S3, and the eCommerce Service, which allows developers to build their own storefronts on top of Amazon’s catalogues.

Think you could be the next Amazon? CF8 makes the technology end of things easier – by a lot – and it certainly doesn’t hurt that CF8 ups the ante on performance to let you operate at Amazon-scale that much more easily.

CF8: Evolving JSON Support

When we first decided to include AJAX functionality in CF8, well, that was just about all that we had decided! Which was fun, because it gave me a broad mandate to go out and come up with some crazy ideas, and scary, because I didn’t have a clue what to do about it.

So we started small, and built native support for JSON into the server, with a fast Java JSON parser of our own, which provided the basis for the new JSON functions in CF8:

serializeJSON(<CFML object>, <serializeQueryByColumns>)

deserializeJSON(<JSON string>, <strictTypes>)

isJSON(<JSON string>)

I’m not going to get into the specifics of these functions here – Ben Nadel’s done a great job covering them on his blog. Also, a note, these function signatures are not final – we’re tweaking them a bit (just a bit, applications you write now will not be broken!) for the final release. But more on that later.

JSON has been gaining currency as an alternative to XML for data interchange for a variety of reasons: it’s lighter, easier to read and write, maps well to basic constructs in many languages (such as arrays and structs), and, perhaps most importantly for the dynamic web, plays well with yer average browser’s language of choice, JavaScript.

With just these basic functions, it becomes easy to consume JSON-powered web services, mash them up as you will, and spit them back out again. For instance, here’s a tasty morsel of a CFC I wrote to get data from del.icio.us JSON web services:

<cfcomponent output="false">

	<!---
	Gets del.icio.us tags for a user.
	The web service returns a JSON object, mapping tag names to
	numbers of posts with that tag, which deserializes to a CFML Struct.
	More details of the web service available at:
	http://del.icio.us/help/json/tags
	--->
	<cffunction name="getTags" access="remote" output="false">
		<cfargument name="userid">
		<cfset tagsURL = "http://del.icio.us/feeds/json/tags/#userid#?raw">
		<cfhttp url="#tagsURL#">
		<cfreturn deserializeJSON(cfhttp.fileContent)>
	</cffunction>

	<!---
	Gets posts for a user, optionally for a specified set of tags.
	The web service returns a JSON array of objects, one object per
	post. Each post object has the keys:
	u - URL
	d - description
	t - tags
	This deserializes to a CFML Array of Structs, with one Struct per post.
	More details of the web service available at:
	http://del.icio.us/help/json/posts
	--->
	<cffunction name="getPosts" access="remote" output="false">
		<cfargument name="userid">
		<cfargument name="tags">
		<cfset var tagList = replace(tags, ",", "+")>
		<cfset tagsURL = "http://del.icio.us/feeds/json/#userid#/#tagList#?raw&count=30">
		<cfhttp url="#tagsURL#">
		<cfreturn deserializeJSON(cfhttp.fileContent)>
	</cffunction>

</cfcomponent>

CFHTTP to the service URL, deserializeJSON on the response, and, all of a sudden, you have native CFML types to extract data from. Doesn’t get much easier than that! I’ll show off an AJAX application that I wrote with this CFC in a little while.

Given CF’s ability to consume data from a variety of sources (databases, LDAP servers, FTP, MS Exchange, HTTP, etc.), along with native support for common data formats (XML, and now JSON), and the large variety of web services out there, you’re now limited only by your own imagination in the mashup applications you can build. Go forth and multiply!

New Beginnings

So it’s that time of year again – all the cool kids are out at CFUnited. I would have loved to have been there, but it is not to be.

Why? Well, with much reluctance, I am leaving Adobe, for entirely personal reasons. If I could, I would have stayed on here for many years to come, and seen ColdFusion through many more releases. The team is, without a doubt, the finest I’ve ever worked with, it’s been quite an experience putting ColdFusion 8 out, working on everything from language design to server scalability, and it has been a pleasure getting to know all the great folks in the ColdFusion community, with whom I will continue to be involved. As Damon likes to say, you can join the ColdFusion team, but you can never leave… Sinister and megalomaniacal, that’s our Damon! 😉

Anasuya and I will be moving to California, where she’ll get down to the arduous task of writing her D.Phil. thesis, and I’ll enroll in the Master’s at the UC Berkeley School of Information, a.k.a. the iSchool. In recent years, I’ve found myself fascinated by the impact that the technologies we build have upon society. The iSchool’s faculty and roster of courses, which meld computer science and the social sciences, should help shape these interests, and give me space to think about what I want to do next. I’m really excited to be studying at the iSchool – this will be a whole new set of challenges and possibilities.

It’s been a lovely 7 years in Bangalore – I met Anasuya and got married here, studied classical guitar at the Bangalore School of Music, jammed in various bands with weird names, circumnavigated the city on a bicycle, did the Koshy’s Sunday brunch (many times), drank too much beer at Peco’s, and so much more. Sadly, I’ve also watched the city decline in this time. Just as the software and services boom drew engineers to Bangalore, so it also drew land sharks, and their ilk. Couple that with an unstable political environment, and you have a recipe for disaster. I can only hope that the citizen’s groups working to better the city will find success in their efforts.

This blog will continue to be, of course, and while I’ll still be writing about technology, I’ll also be talking about the social impact of computing (intelligently, I hope!). And there will be lots more CF8 in the days to come – I’ve been rather insanely busy helping mop things up for the final release build, but I should have more time free to blog CF8 features from now onwards, especially the AJAX stuff, of which I am particularly proud.

New ColdFusion blogs

A couple of new ColdFusion team blogs for you to watch – as though there isn’t enough Scorpio buzz out there already!

Rakshith, a great guy who worked with me on the AJAX stuff, has got his blog up – he’ll be writing up new Scorpio features that he worked on, as will I, in the days to come.

Ahmad Patan, a member of our QA team, has started the CFPDF blog, to talk about new PDF manipulation and PDF form processing features in Scorpio. The QA guys pair up with development team members right from design time, so they end up knowing the features inside out, upside down, sideways, backward, and any other way you’d like to write your CFML code.

Dude, where’s my class?

Have you ever been stuck in a situation where you needed to find out where a particular Java class was loaded from? While this may not be the kind of problem the average CFer faces, I’m sure there are many gurus who can attest that they’ve been burned with issues of this sort. Robi Sen recently posted a bit of Java code to help solve problems like this, and reminded me of a time just a few months ago when I had to chase down a bizarre bug that turned out to be rooted in exactly this kind of JAR file hell.

On the ColdFusion team, we have a fairly rigorous checkin/build/test process. Every time anyone checks something in to the source control system, the build machine gets it, attempts to do a full build and runs a series of automated regression tests against it, just to make sure that nothing obvious has been broken. Those who break the build have the dubious honour of being named beer boy/girl, until someone else breaks the build, and claims the crown!

This has happened to me several times and, on most occasions, the problem was obvious, embarrassing, and easily fixed. However, in this particular case, I just could not figure out what was going on – I’d made changes to a couple of classes, with class A relying on a new method that I’d introduced in class B, but the build machine would constantly fail, with an error message which made it appear as though the new method on class B just did not exist. The build worked on my machine, and on other engineers’ machines, but constantly failed on the build machine.

It was Rupesh who eventually suggested that, perhaps, class B was getting loaded from somewhere other than it was supposed to be. As matters turned out, he was right. There was a dead JAR file that someone had placed on the build machine with a very old version of class B; the build machine was trying to build against that, rather than the newer version which I had checked in. Once the dead JAR file was removed, everything went through OK, and my beer boy crown passed on to the next hapless build-breaking engineer.

This bit of detective work was made possible by some interesting, albeit not very well known, Java APIs. All Java classes are defined, self-referentially, as instances of the class java.lang.Class. The java.lang.Class instance for a specific Java class can be loaded using java.lang.Class’s forName method. Every class maintains an object called ProtectionDomain, which defines the security access rules for the domain which that class belongs to. ProtectionDomain, in turn, maintains a CodeSource object that points to the location from which the code for that domain was loaded. The one gotcha to keep in mind is that the CodeSource object will be null for code which is loaded by Java’s bootstrap* classloader.

So here’s a bit of CF code which uses these APIs:

<cfset codeSource = createObject("java", "java.lang.Class").forName(url.classname).getProtectionDomain().getCodeSource()>
<cfif not isDefined("codeSource")>
<cfoutput>Code source is not defined; <b>#url.classname#</b> is probably being loaded by Java's bootstrap classloader.</cfoutput>
<cfelse>
<cfoutput>The class <b>#url.classname#</b> was loaded from <b>#codeSource.getLocation().toString()#</b></cfoutput>
</cfif>

Just pass your classname in (be sure to get the uppercase/lowercase characters right! Java’s not as forgiving as CF) with the classname URL parameter, and this little code snippet will tell you exactly where the application server loaded the class from.

There is one caveat to keep in mind, which can be good or bad, depending on where you’re coming from. The example that I’ve provided will show where a class is loaded from, from the perspective of the calling code’s classloader. 99% of the time, this is exactly the behaviour you’ll want, but for those with more advanced requirements, you may want to drop this code into a more appropriate location. This is an important consideration, since Java does allow parallel classloaders, which may load the same class from different locations. All application servers adopt this approach, since it is mandated by the JEE specifications. Sounds abstruse? Ignore this little detail then, it probably doesn’t matter to you anyway.

For those strange times when you have to debug a problem on a production system, or want to make sure that you’re using exactly the same classpath as your application server, you may want to try the approach that I’ve outlined. I hope that it will help you as much as it helped me.

Update: As Dan points out, Robi’s intention was quite different from mine – his code was written to list all classes in a given package, not to find where a class was loaded from.

* The Java bootstrap classloader is the mother of all classloaders, which loads up the Java core libraries. All application servers establish their own byzantine classloader hierarchies under the bootstrap classloader; I’ll navigate that labyrinth another time. For the curious, this theserverside.com article, although a bit dated, will give you an idea of what’s involved.

Sneak Peek: Scorpio Server Monitoring

Just for all you folks who couldn’t make it out to MAX – here’s a peek at what Scorpio may be offering up for server monitoring. This is the presentation I gave at MAX, with voice over, but missing, alas, the live demos. It may take a little time to load because of all the audio, so do be patient.

Unlocking the ColdFusion Server Black Box

Take a look, and leave comments here in case you think there’s anything big that we’re missing.

Disclaimer: This feature is still under development and is subject to change. Consider this presentation to be a just a preview of something that may make it into Scorpio.