The Racket Blog

Racket v5.0.1

2010-08-03T22:36:00.000-04:00

Racket version 5.0.1 is now available from

  http://racket-lang.org/

Datalog is a lightweight deductive database system with Racket integration. It is now available in the datalog collection and with #lang datalog.
Racklog provides Prolog-style logic programming in Racket, adapted from Dorai Sitaram's Schelog package. It is available in the racklog collection and now as #lang racklog.
By default make install and raco setup compile collections in parallel on all available processors. (Use raco setup -j 1 to disable, if necessary.)
Changes (as part of 5.0) in the racket language compared to the scheme language: constructor-style printing, a struct alternative to define-struct that fits more naturally with match and constructor-style printing, bytecode-dependency management via SHA-1 hashes instead of just timestamps (where the openssl/sha1 library provides the SHA-1 hash function), a reorganization of scheme/foreign into ffi/unsafe and associated libraries, and new printing functions eprintf and displayln. Also, a generator from racket/generator is required to have the form (generator () body ...), which supports a planned extension to let a generator accept arguments.
Changes to the racket language (since 5.0): internal-definition positions allow mixing expressions with definitions, full continuations can escape past a continuation barrier, custodians can attempt to terminate subprocesses and subprocess groups (see current-subprocess-custodian-mode, subprocess-group-enabled), the JIT supports additional unboxing flonum operations and unsafe variants, ffi/unsafe provides an asychronous-call mechanism to deal with foreign threads, a new "." modifier for format string directives (e.g., "~.s" and "~.a") limits the respective output to (error-print-width) characters.
The core type system of Typed Racket has been substantially revised. In particular, Typed Racket can now follow significantly more sophisticated reasoning about the relationships between predicates. Additionally, Typed Racket now allows variable arity types in more places, allowing programmers to specify variable-arity lists.
We are working on an optimizing version of Typed Racket that takes advantage of type information for certain classes of programs. This project is a work in progress. For those interested, see the documentation for #:optimized.
The web-server/formlets library adds a formlet* form that allows dynamic formlet construction, as opposed to formlet which requires syntactic Xexprs and static formlets. Several new library formlets are added.
The syntax/parse library has new support for matching literals at different phases using the #:phase argument for literals and literal sets.
RackUnit now includes a GUI test runner as rackunit/gui.
The 2htdp/image library now includes flip-vertical and flip-horizontal operations that mirror images (vertically and horizontally).

Racket

2010-06-07T14:50:00.004-04:00

PLT is happy to announce the release of Racket, available from

  http://racket-lang.org/

With Racket, you can script command shells and web servers; you can quickly prototype animations and complex GUIs; regexps and threads are here to serve you. To organize your systems, you can mix and match classes, modules or components. Best of all, you start without writing down types. If you later wish to turn your script into a program, equip your Racket modules with explicit type declarations as you wish. And Racket doesn't just come as a typed variant; you can also write your modules in a purely functional and lazy dialect.

Racket comes in so many flavors because Racket is much more than a standard scripting language or a plain programming language. Racket supports language extensibility to an unequaled degree. A Racket programmer knows that making up a new language is as easy as writing a new library.

To help you start quickly, Racket includes batteries in all shapes and sizes, most importantly, extensive documentation and all kinds of libraries.

Racket occupies a unique position between research and practice. It inherits many major ideas from language research, among them type safety (when the type system says that x is a number, then at runtime it always is a number) and memory safety (when some memory is reclaimed by the garbage collector it is impossible to still have a reference to it). At the same time, user demand governs rigid adherence to purely theoretical principles.

Racket, formerly PLT Scheme, is a product of over 15 years of development. Although Racket starts with a mature software base and an established user community, its new name reflects our view that this is just the beginning of Racket's evolution.

PLT Scheme v4.2.5

2010-04-02T21:51:00.001-04:00

PLT Scheme version 4.2.5 is now available from

  http://plt-scheme.org/

PLT now supports multi-core parallelism via futures. Futures create tasks that run in parallel, as long as the tasks stay in the "fast path" of the runtime system. For more information, see the guide.
Our unit testing framework, schemeunit, is now included in the distribution. A graphical test runner is available via schemeunit/gui.
The support languages for the "Programming Languages: Application and Interpretation" textbook by Shriram Krishnamurthi are now part of PLT Scheme. In addition the PLAI GC language comes with a random mutator generator (to help test collectors) and an improved heap visualizer.
New Russian and Ukranian translations, thanks to Sergey Semerikov.
A number of improvements to Redex's typesetting facilities.
Typed Scheme users can now automatically generate predicates from types with define-predicate. Typed code can be inserted in untyped modules by requiring with-type from typed/scheme.
The scheme/class library now provides contract combinators for classes (class/c) and objects (object/c). See the Reference and Guide for details. Also, a backwards-compatible object-contract version of object/c has replaced the old object-contract combinator.
Writing new kinds of contracts is now easier with keyword-based constructors (make-contract and make-flat-contract), a simpler set of structure properties (prop:contract and prop:flat-contract), and the introduction of blame objects for tracking contract metadata.
The Scheme-implemented bytecode reader fails less often. This is used by "mzc --decompile". The Scheme-implemented bytecode writer uses the compact bytecode format and fails less often. This may be used in the future for Scheme-implement bytecode processors.
The language dialog now suggests using "#lang" more strongly as the default language. DrScheme no longer uses the term `Module language'.

Talk at Flourish

2010-03-08T11:35:00.006-05:00

The image in this post shows a tree where the interior nodes represent directories and the leaf nodes represent files in the PLT source code. The leaves are colored based on the programming language used. (To avoid clutter, if there is more than one file in a given directory written in a particular language, that language only gets a single dot.)

Some highlights: the blues are Scheme-like languages, the reds are langauges we use to write documentation (see Scribble for more about them), the greens are teaching languages, orange is the language we use to bootstrap new languages, and yellow is a language for metadata about nearby files.

Curious about how we managed to write and use so many different languages? I'll be giving a talk at Flourish 2010 next week (3/19 @11am, UIC in Chicago) explaining how. Come to learn more!

DAGs vs Trees

2010-02-28T20:16:00.006-05:00

As I wondering whether or not there is a better layout algorithm for the module browser window, I looked into tree maps. Of course, the modules in a program form a DAG, not a tree, so I wondered just how big the tree would get if all of the shared structure in the DAG were replicated. Hey, I figured, if a tree map can handle showing me my entire filesystem, maybe that could work.

... yeah, no. Turns out to be hopeless. In the spirit of a geeky take off on a jelly bean counting contest, lets see if you can guess just how big these things get. Consider the module graph from the program #lang scheme (ie, the graph that just contains an empty program). This program loads 170 modules with 917 connections between modules (counting the main file that just contains the #lang scheme).

So, the question: how many nodes are there in the unsharified tree? First one to come within 1 billion of the right answer gets all of the fame and glory that this blog brings to bear (har har). I'll post the answer in the comments in a few days (and no fair cheating, those of you that know enough to be able to get your hands on the DAG).

Benchmarks

2010-01-29T12:57:00.005-05:00

First, the usual disclaimer:

That said, I've run the latest version of PLT Scheme on two sets of benchmarks:

Benchmarks in the PLT sources – vs. Bigloo, Chicken, Gambit, Guile, Ikarus, Larceny, MIT Scheme, and Scheme48; safe operations and generic arithmetic only

Benchmarks in the Gambit sources – vs. Bigloo and Gambit; generic vs. fixnum-/flonum-specific arithmetic, safe vs. unsafe operations

The second set is why I started running benchmarks. Fixnum-/flonum-specific arithmetic and unsafe operations are new in PLT Scheme 4.2.4. The benchmark results suggest that the new operations in PLT Scheme offer roughly the same performance benefits as in Bigloo and Gambit. There's room for improvement, but it's a good first cut. For the other results: PLT Scheme is rarely the fastest implementation on a given benchmark. For most purposes, though, it's in the same ballpark – except for programs that spend all their time capturing and invoking continuations. It's fun to run benchmarks occasionally. Now, back to working on language design, libraries, documentation, usability...

PLT Scheme v4.2.4

2010-01-28T21:40:00.001-05:00

PLT Scheme version 4.2.4 is now available from

  http://plt-scheme.org/

The scheme/flonum and scheme/fixnum libraries provide flonum- and fixnum-specific operations. In the case of flonum-specific operations, the JIT compiler can recognize combinations of operations (including local bindings) and improve performance by "unboxing" intermediate results.
The scheme/unsafe/ops library provides arithmetic and other operations that are implemented without dynamic checks. Avoiding checks can sometimes improve performance, but at the expense of safety.
2htdp/universe: We have severed the connection between universe and an image library and made a few other, minor changes. Most programs will now have to change to require the htdp/image library explicitly. For the full details, see the new Porting World Programs section of the documentation.
The 2htdp/image library continues to grow. In this release, it is supported by 2htdp/universe, equality changed to be based on how the images are drawn, cropping and curves were added, and support for more kinds of pens were added.
htdp/world: The old world teachpack remains deprecated. HtDP/2e exclusively uses the new 2htdp/universe library. For backwards compatibility, the world teachpack will remain in the distribution until the coming summer.
The scheme/class library now provides this%, which refers to the class of the current object (i.e. this).
scheme/generator has convenient functions for infinite generators, and for converting a generator to a sequence for iteration.
PLT Scheme's add-on directory can be customized by the $PLTADDONDIR environment variable or --addon/-A command-line flags. This controls where downloaded Planet packages and their compiled Scribble documentation are installed.
Additional extensions include: saving errno in foreign calls, much improved sort speed, normalized results from procedure-arity, and more.

Scheme Videos (Lectures and Talks)

2010-01-02T11:40:00.005-05:00

Scheme Videos (Lectures and Talks)

(thanks to Geoffrey Knauth and Hari)

Following a mailing-list request, it turns out that there are quite a lot of Scheme-related lectures and talks floating around out there in video format. The following list was compiled by Geoffrey Knauth, with contributions from Hari and Michael Sperber.

There's the SICP course Abelson & Sussman gave to [HP, I think] in the mid-1980s: http://groups.csail.mit.edu/mac/classes/6.001/abelson-sussman-lectures/

MIT OCW / 6.001 using SICP, Spring 2005: http://ocw.mit.edu/OcwWeb/Electrical-Engineering-and-Computer-Science/6-001Spring-2005/VideoLectures/index.htm

All the ICFP 2009 videos (man this made my day!!): http://vidiowiki .com/feature/list/fnu /ICFP_2009

Daniel P Friedman - A Celebration (this too!): http://www.cs.indiana.edu/dfried_celebration.html

DrScheme v4.0 Tour: http://www.youtube.com/watch?v=vgQO_kHl39g&fmt=18

Similar, if you understand Russian:
http://www.youtube.com/watch?v=wECY7s9k-V0
http://www.youtube.com/watch?v=2CVJjqOT6WM

Matthias Felleisen - Programming at Northeastern: http://www.savevid.com/video/matthias-felleisen-programming-at-northeastern-university.html

Matthew Flatt - Processes without Partitions: http://www.researchchannel.org/prog/displayevent.aspx?rID=3892

Shriram Krishnamurthi on WeScheme: http://vidiowiki.com/watch/cydr9yk/

Robby Findler - Macros Matter: http://www.mefeedia.com/video/26348171

Using PLT Scheme in the Game Industry: http://www.youtube.com/watch?v=2CVJjqOT6WM

Stanford Lecture (Kawa): http://www.youtube.com/watch?v=_cV8NWQCxnE

Bluetooth communication using PLT Scheme: http://www.youtube.com/watch?v=pmR_dIXm6sY

SICP at UCB: http://webcast.berkeley.edu/course_details.php?seriesid=1906978454

http://www.aduni.org/courses/sicp/ from ADUni by Holly Yanco. It comes with pretty good lecture notes and problem sets.

Michael Sperber's DMdA lectures (in German, natch): http://timms.uni-tuebingen.de/List/List01.aspx?rpattern=UT_200[89]_____00[12]_info1_000_

Futures: Fine Grained Parallelism in PLT

2009-12-07T13:18:00.002-05:00

We're pleased to announce the initial release of parallel futures, a construct for fine-grained parallelism in PLT. Roughly speaking, a programmer passes a thunk to 'future' and it gets run in parallel. That "roughly" holds a few gotchas, partly because we're just getting started and partly due to the technique we're using. See the documentation for more details:

http://pre.plt-scheme.org/docs/html/futures/

If you've got a multicore machine where you can't keep the cores busy or your office/machine room is a bit cold, try this program:

#lang scheme
(require scheme/future)
(define (loop) (loop))
(for-each
 touch
 (for/list ([i (in-range 0 (processor-count))])
  (future loop)))

Note that you have to build mzscheme with futures; it isn't enabled by default, but see the docs above for how to do that. Beyond the above, we've also gotten a few parallel kernels going and are seeing good scalability up to 8 cores (the biggest machine we have around for the time being).

PLT Scheme v4.2.3

2009-12-01T06:03:00.001-05:00

PLT Scheme version 4.2.3 is now available from

  http://plt-scheme.org/

The unit test framework for the teaching languages provides check-member-of and check-range for checking "random functions", i.e., "functions" that may produce several different results for one and the same argument.
Added a new image library, 2htdp/image. Significant changes from htdp/image:
- copying and pasting does not introduce jaggies
- equal? comparisons are more efficient
- added rotation & scaling
- got rid of pinholes (new overlay, beside, above functions based on bounding boxes)
The scheme/vector library provides common vector operations (also reprovided by scheme).
The scheme/promise library provides several new kinds of promises with alternatives execution strategies.
New port-reading utilities: in-port, port->list, file->list.
A new require-macro, path-up, for requiring a file that is higher in the directory tree.

PLT Scheme v4.2.2

2009-10-04T16:29:00.001-04:00

PLT Scheme version 4.2.2 is now available from

  http://plt-scheme.org/

DrScheme now, by default, compiles all of the files that are loaded when it runs a program and saves the compiled files in the filesystem. This should lead to faster load times (not faster runtimes) since it avoids re-compiling files whose dependencies have not changed.
New Scribble libraries and documentation make it easier to get started with Scribble, especially for uses other than PLT documentation. DrScheme now has better indentation and syntax coloring support for Scribble languages (and generally all @-exp based languages).
The new syntax/keyword library provides support for macros with keyword options. A new quick start guide has been added to the documentation for the syntax/parse library.
Added support for abstract contracts via the #:exists keywords. This is an experiment to add support for data hiding to the contract system.
Added in-producer: a sequence expression makes it easy to iterate over producer functions (e.g., read). A new scheme/generator library creates generators that can use a (parameterized) yield function.
HtDP langs: several primitives now consume 0 and 1 arguments in ISL (and up), including append, + and *. In addition, make-list was added to the primitives.
The API to Universe has a number of new constructs. All Universe programs should run unchanged. The most important change is the addition of animate as an alternative name for run-simulation. In addition, adding the clause (state true) to a world description now pretty-prints the state of the world into a separate canvas.
A number of changes were made to the DeinProgramm / DMdA language levels: The check-property and contract forms were added, define-record-procedures-parametric has changed. See the documentation for details.
The test engine in the HtDP languages no longer warns programmers when the Definitions window has no tests.
ProfessorJ (and related code) is no longer included in the PLT distributions. It may re-appear in the future as a PLaneT package.

set! vs set-box! and unbox

2009-09-21T07:23:00.007-04:00

A few weeks ago I was chatting with some PLT folks and was surprised to hear them say that they avoided set! because using set-box! and unbox was easier to see what was going on.

This struck me as wrong since one might pass boxes around and then you can't be sure which box you're mutating, but you cannot pass variable references around and thus which variable you're using is always lexically apparent. (Of course, when you add lambda into the mix that isn't really true, since you can capture a variable in a closure and pass that around.)

Their point seemed to be that you had to write something special at each use of the box, unlike with set! where you simply write a variable reference and it might be getting a changing quantity and it might not be. This made me realize I could do something to help, at least, and so I changed Check Syntax so that it colored set!'d variables in red, like this:

PLT Scheme v4.2.1

2009-07-30T08:24:00.001-04:00

PLT Scheme version 4.2.1 is now available from

  http://plt-scheme.org/

This is the last release that includes ProfessorJ. As of the next release, Kathy Gray who created and maintained the Professor will move the code to planet and maintain only at a minimal level.
Typed Scheme 2.0 extends the type system significantly, making it more expressive. For example, predicates applied to selectors, such as (number? (car x)), are meaningful to the type system.
Faster installation of Planet packages that trigger install of other Planet packages, because the documentation index is updated only once after a group of packages is installed.
The syntax/parse library provides macro writers with an enhanced syntax pattern matcher that reports errors based on the patterns' declared classes of syntax.
Identifier mappings following the v4 dictionary interface and naming conventions are available from the syntax/id-table library.
Redex: added define-relation and generalized patterns that appear in "where" clauses to use the full Redex pattern matcher. (This is a backwards incompatible change, but one often requested; see the Redex release notes for details.)
The Web Server's serializable closures are now available for other purposes through the web-server/lang/serial-lambda library.
Teachpacks: small changes to universe portion of the "universe.ss" API, plus the addition of a form for launching many (communicating) worlds simultaneously. Bug fixes concerning conversion to strings.
It is now possible to create custom scribble readers with a command characters different than @, see make-at-reader/inside and make-at-reader
Note: this is likely to be the last release that includes a solaris distribution. If you need these builds, or if you have access to a (Sparc) Solaris machine than can be used in PLT builds, then please let me know.

Serializable Closures in PLT Scheme

2009-06-23T12:02:00.003-04:00

PLT Scheme supports an extensible serialization system for structures. A structure is serializable if it has a prop:serializable property. There are many properties in PLT Scheme for other extensions, such as applicable structures and custom equality predicates.

The PLT Web application development framework uses these features to provide serializable continuations through a number of source transformations and a serializable closure structure.

Warning: This remainder post refers to features only available in the latest SVN revision of PLT Scheme.

I've recently made these closures more accessible to non-Web programs through web-server/lang/serial-lambda. Here's a demo:

#lang scheme
(require web-server/lang/serial-lambda
         scheme/serialize)

(define f
  (let ([z 5])
    (serial-lambda
     (x y)
     (+ x y z))))

(define (test-it)
  (printf "~S~n" (f 1 2))
  (let ([fs (serialize f)])
    (printf "~S~n" fs)
    (let ([df (deserialize fs)])
      (printf "~S~n" df)
      (printf "~S~n" (df 1 2)))))

> (test-it)
8
((2) 1 ((#"/Users/jay/Dev/svn/plt/collects/web-server/exp/test-serial.ss" . "lifted.6")) 0 () () (0 5))
#(struct:7a410aca70b31e88b4c2f0fe77fa7ffe:0 #)
8

Now, let's see how it is implemented. web-server/lang/serial-lambda is thin wrapper around web-server/lang/closure, which has two syntax transformer functions: define-closure! which defines the closure structure and make-closure which instantiates the closure. (The two tasks are separated to easily provide a user top-level definition syntax for named closures with different free identifires, rather than simply anonymous lambdas with fixed free identifiers.)

make-closure does the following:

Expands the procedure syntax using local-expand, so it can use free-vars to compute the free identifires.
Uses define-closure! to define the structure and get the name for the constructor.
Instantiates the closure with the current values of the free identifiers.

The more interesting work is done by define-closure!. At a high-level, it needs to do the following:

Create a deserialization function.
Create a serialization function that references the deserializer.
Define the closure structure type that references the serializer.
Provide the deserializer from the current module so that arbitrary code can deserialize instances of this closure type.

These tasks are complicated in a few ways:

The deserializer needs the closure structure type definition to create instances and the serializer needs the closure structure type to access their fields.
The serializer needs the syntactic identifier of the deserializer so that scheme/serialize can dynamic-require it during deserialization.
The deserializer must be defined at the top-level, so it may be provided.
All this may occur in a syntactic expression context.

Thankfully, the PLT Scheme macro system is powerful to support all this.

syntax-local-lift-expression allows a syntax transformer to lift an expression to the top-level of a module and returns the identifier it is bound to.
syntax-local-lift-values-expression (added in 4.2.0.3) provides the same for expressions that return multiple values, such as make-struct-type, which is used to define structures.
syntax-local-lift-provide (added in 4.2.0.4) allows a syntax transformer to lift a provide to the top-level.

The only complicated piece is allowing the deserializer and serializer to refer to the closure structure constructor and accessors. This is easily accomplished by first defining lifting boxes that will hold these values and initializing them when the structure type is defined. This is safe because all accesses to the boxes are under lambdas that are guaranteed not to be run before the structure type is defined.

An aside on the closure representation. The closure is represented as a structure with one field: the environment. The environment is represented as a thunk that returns n values, one for each of the free identifiers. This ensures that references that were under lambdas in the original syntax, remain under lambdas in the closure construction, so the serializable closures work correctly inside letrec. This thunk is applied by the serializer and the free values are stored in a vector. The closure also uses the prop:procedure structure property to provide an application function that simply invokes the environment thunk and binds its names, then applys the original procedure syntax to the arguments.

An aside on the serializer. The deserializer is bound to lifted identifier which is represented in PLT Scheme as an unreadable symbol. Version 4.2.0.5 added support for (de)serializing these.

PLT Scheme v4.2

2009-06-01T14:41:00.001-04:00

PLT Scheme version 4.2 is now available from

  http://plt-scheme.org/

Internally, this version includes a conversion from C++ to Scheme for part of the GUI toolbox --- specifically, 25k lines of code that implement the general text and pasteboard editor. This conversion is a start on a larger reimplementation of the GUI toolbox. Although we believe that this change will help make PLT Scheme better in the long run, we must expect bugs in the short term due to porting errors. Users should therefore be aware of the change, even though the new implementation is meant to behave the same as previous versions.

A new statistical profiler is now available; see the "profiler" manual for more information. Currently, the profiler supports only textual output, but future plans include a GUI interface and DrScheme integration.
The world teachpack is now deprecated. Its replacement universe has a new interface that uses strings instead of symbols and characters.
Web-server: Native continuations in the stateless servlet language support capturing continuations from untransformed contexts; soft state library for stateless servlets.
DrScheme's Stepper can now jump to a selected program expression.
New in scheme/base: hash-has-key?, hash-ref!, in-sequences, in-cycle. New in scheme: count, make-list (from scheme/list), const (from scheme/function).
Some performance improvements, including faster start-up for small programs. The latter is a result of delaying module invocations at higher phases (compile time, meta-compile time, etc.) until compilation is demanded at the next lower phase; this on-demand instantiation is per-phase, as opposed to per-module within a phase.

[Note that mirror sites can take a while to catch up with the new downloads.] Feedback Welcome.

Typed Scheme 2.0

2009-05-25T13:47:00.003-04:00

Typed Scheme version 2.0 is now available from SVN.

One persistent limitation of Typed Scheme has been that while this expression works as expected:

(if (number? x) (add1 x) 7)

The simple transformation of making x a part of a structure breaks Typed Scheme's ability to reason about the code. So this expression doesn't typecheck:

(if (number? (car x)) (add1 (car x)) 7)

With the newest version of Typed Scheme, now available in SVN, both of these will now work. In general, Typed Scheme can now follow paths into arbitrary immutable structures, including pairs.

This is part of a more general reworking of underlying mechanisms of the Typed Scheme typechecker, which makes it both simpler and more flexible. I hope that it will be possible, sing this new foundation to add additional features that make more programs easy to express in Typed Scheme.

Of course, these changes mean that Typed Scheme may be more unstable, so if you notice any new bugs, please let us know.

Unfortunately, this won't be available in the upcoming 4.2 release, but it will be in the release after that.

If you have any questions or comments or feature requests for Typed Scheme, please let us know.

Explicit Renaming Macros; Implicitly

2009-05-24T03:17:00.004-04:00

It's been one too many times that I hear respectable Schemers talk about how they like explicit renaming macros — not because they're more powerful, but because using them is close to using simple defmacros. In this post I'll show how you can easily write ER-like macros in PLT, just so I won't need to explain the same thing once again. Disclaimers:

If you're not interested in ER-macros, then you shouldn't read this.
I'm not claiming that ER macros are not respectable, I'm just surprised at the knee jerk reaction to syntax-case.
This is not an attempt at providing some portable library or even a PLT library. The intention is to show that syntax-case-style macros are "as convenient" as ER macros, if you really want to get down to that level.
This is also not an attempt at any kind of formal claim of equivalence in any direction, only a demonstration that you can get the same kind of convenience.
The bottom line here should be just the convenience point, addressed at people who like ER macros for that, and who think that syntax-case macros are somehow magical or that you lose the ability to play with S-expressions.

The important fact here is that while PLT's syntax-case macro system does not give you raw S-expressions, what you get is a simple wrapper holding them. A macro is a syntax transformer: a function that consumes a syntax value and returns one. For example:

  (define-syntax (foo stx)
    #'123)

is a macro that always expands to 123 (with #'123 being the usual shorthand for (syntax 123)). A syntax object in PLT Scheme (the input to macro functions) is an S-expression, with some lexical information added — this includes the lexical context (in an opaque form), source location, and a few more things. To be more precise, a syntax value is a nested structure of wrappers holding lists and pairs, holding more wrappers, with identifiers at the leaves, where an identifier is a wrapper holding a symbol. It's easy to strip off all wrappers using syntax->datum if you like to work with S-expressions, but you don't want to strip it off of identifiers, since that will lose the important gravy. (In fact, the defmacro library works by stripping off all information, even from identifiers, then reconstructing it by trying to match names in the output form with the original input.) Instead of throwing away all information, what we want to do is preserve identifiers. We can use syntax->list for this: this is a function that takes a syntax value that contains a list, and strips off the top-level extra information leaving you with a list of syntaxes for the sub-expressions (returning #f if the input syntax does not hold a list). Once we have such a list, we can do the usual kind of processing with it, and when we're done turn the result back into a syntax using datum->syntax (which "borrows" the original input expression's information). For example,

  (define-syntax (add1 stx)
    (let ([+ #'+])
      (datum->syntax stx `(,+ 1 ,@(cdr (syntax->list stx))) stx)))

That's a very simple example though; if you try something a little more complicated, you quickly find out that all this unwrapping is inconvenient:

  (define-syntax (mylet stx)
    (let ([*lambda #'lambda])
      (datum->syntax
       stx
       `((,*lambda ,(map (lambda (x) (car (syntax->list x)))
                         (syntax->list (cadr (syntax->list stx))))
                   ,@(cddr (syntax->list stx)))
         ,@(map (lambda (x) (cadr (syntax->list x)))
                (syntax->list (cadr (syntax->list stx)))))
       stx)))

(Note also the *lambda that is used to avoid the lambda expressions used in the meta-code.) What can help here is some helper function that receive a syntax value as its input, and turn all wrapped lists into real lists recursively, but will leave identifiers intact:

  (begin-for-syntax
    (define (strip stx)
      (let ([maybe-list (syntax->list stx)])
        ;; syntax->list returns #f if the syntax is not a list
        (if maybe-list (map strip maybe-list) stx))))

But as long as we're writing a syntax utility, we can make it do a litte more work: feed the resulting tree to your transformer, grab its result, and do the necessary datum->syntax voodoo on it:

  (begin-for-syntax
    (define (er-like-transformer transformer)
      (define (strip stx)
        (let ([maybe-list (syntax->list stx)])
          ;; syntax->list returns #f if the syntax is not a list
          (if maybe-list (map strip maybe-list) stx)))
      (lambda (stx)
        (datum->syntax stx (transformer (strip stx)) stx))))

With this utility defined, the above macro becomes much easier to deal with:

  (define-syntax mylet
    (er-like-transformer
     (lambda (exp)
       (let ((vars  (map car (cadr exp)))
             (inits (map cadr (cadr exp)))
             (body  (cddr exp)))
         `((,(syntax lambda) ,vars ,@body)
           ,@inits)))))

...and this is almost identical to the explicit renaming version of the macro; for example, compare it with the sample code in the MIT-Scheme manual. The only change is that (rename 'lambda) is replaced with (syntax lambda). Obviously, this is very close, but doesn't show intentional captures. So I just grabbed the loop example from the same page, and did the same change — only this time I used #'foo instead of (syntax foo) (and I also changed the one-sided if to a when). The resulting macro works fine:

  (define-syntax loop
    (er-like-transformer
     (lambda (x)
       (let ((body (cdr x)))
         `(,#'call-with-current-continuation
           (,#'lambda (exit)
            (,#'let ,#'f () ,@body (,#'f))))))))
  
  (define-syntax while
    (syntax-rules ()
      ((while test body ...)
       (loop (when (not test) (exit #f))
             body ...))))
  
  (let ((x 10))
    (while (> x 0)
      (printf "~s\n" x)
      (set! x (- x 1))))

This is pretty close to a library, and indeed, as I was writing this text I found a post by Andre van Tonder on the Larceny mailing list that uses a very similar approach and does make a library out of it. This is done by adding two arguments to the expected ER-transformation function — one is a rename function (since the above method uses syntax it is limited to immediate identifiers), and the other is always passed as free-identifier=?. Such a compatibility library is, however, not the purpose of this post. Finally, there is still a minor issue with this — PLT has an implicit #%app which is used wherever there are parentheses that stand for a function application — and in this code they are used unhygienically. This is usually not a noticeable problem, and if it is, you can add explicit #%apps. It might also be possible to find a more proper solution (e.g., use a hash table to keep track of lists that were disassembled by the client transformer), but at this point it might be better to just use the more natural syntax-case anyway.

The Two State Solution: Native and Serializable Continuations in the PLT Web Server

2009-05-18T16:48:00.000-04:00

One of the annoyance of the stateless Web application language that comes with the PLT Web Server is that you can't call third-party higher-order library procedures with arguments that try to capture serializable continuations. (I know, you try to do that all the time.) For example:

(build-list
 3
 (lambda (i)
   (call-with-serializable-current-continuation
    (lambda (k) (serialize k)))))

The problem is that the stateless language performs a transformation on your program to extract the continuations into a serializable representation. If you really need to do this, we've developed a compromise called "The Two State Solution": one state on the client and the other on the server. Only the third-party parts of the continuation (in this case, the code inside build-list) are stored on the server; everything else is shipped to the client. You just need to annotate your code slightly to indicate where the transition is:

(serial->native
 (build-list
  3
  (lambda (i)
    (native->serial
     (call-with-serializable-current-continuation
      (lambda (k) (serialize k)))))))

serial->native signals the transition to the third-party and native->serial signals the transition back.

It is still a little annoying to find when you've called these third-party higher-order library procedures with arguments that try to capture serializable continuations, so there's a simple macro that provides a transitioning wrapper for you:

(define-native (build-list/native _ ho) build-list)

expands to:

(define (build-list/native fst snd)
  (serial->native
   (build-list
    fst
    (lambda args
      (native->serial
       (apply snd args))))))

This new feature is documented in the online manual, of course.

Soft State in the PLT Web Server

2009-05-18T16:47:00.000-04:00

Many Web applications and network protocols have values in the continuation that are necessary to complete the computation, but that may be recomputed if they are not available. This is "soft state".

For example, a Web application may cache a user's preferences from a database and deliver it to a Web browser as a hidden value; when the value is returned to the application in subsequent steps, it is used to customize the display. However, if the preferences were not available (or were corrupted in some way), the application could retrieve them from the database.

When using the PLT Web Server's native continuations, this roughly corresponds to the use of a weak box: a box that the GC is allowed to erase the contents of. When using the PLT Web Server's serializable continuations it roughly corresponds to a weak box and a weak hash table (that holds its keys weakly) to give the box a serializable value as an identifier.

This programming pattern is a bit difficult to get right. So, a library that implements it is now provided with PLT Scheme: web-server/lang/soft.

Here's a trivial example:

#lang web-server

(provide interface-version start)
(define interface-version 'stateless)

(define softie
  (soft-state
   (printf "Doing a long computation...~n")
   (sleep 1)))

(define (start req)
  (soft-state-ref softie)
  (printf "Done~n")
  (start
   (send/suspend
    (lambda (k-url)
      `(html (body (a ([href ,k-url]) "Done")))))))

Scheme Workshop: deadline NOT extended!

2009-05-18T13:10:00.002-04:00

We're holding the line on our submission deadline; it's still June 5, so that gives you about three weeks to write something awesome.

Re-posting the entire CfP on a blog seems a bit tacky, so instead I'll just post the link:

http://blog.plt-scheme.org/2009/01/cfp-scheme-workshop-2009.html

We look forward to your submissions!

What is send/suspend?

2009-05-06T08:58:00.001-04:00

I often ponder what send/suspend really is. It is a lot like call/cc, but has the curious property that the continuation escapes in a single way and is only called in a particular context. I often wonder if there is something weaker than call/cc that implements send/suspend.

Today I wrote a little dispatcher that uses threads to implement send/suspend. In this implementation, sending truly suspends the computation.

Here's the code: http://www.copypastecode.com/codes/view/5003

The trick is to have channels for communicating responses and requests. When you run this example, you should be able to add two numbers. But, in contrast to the normal send/suspend, all the URLs are one-shots, because once the computation is resumed, it moves forward... it is never saved.

This implementation technique also precludes clever implementations of send/suspend/dispatch, like:

(define (send/suspend/dispatch mk-page)
  (let/cc k0
    (send/back
     (mk-page
      (lambda (handler)
        (let/ec k1 
          (k0 (handler (send/suspend k1)))))))))

the DrScheme repl isn't the one in Emacs

2009-03-29T13:03:00.004-04:00

Dear old Lisper,

You have found drscheme and it is almost like your old Lisp machine. It is easy to program in it, it has things that nobody else has, and we all love parentheses. But after some initial enthusiasm you are wondering why in the world, we decided not to provide commands for sending individual definitions and expressions from the Definitions window to the Interactions window, aka, REPL.

It wasn't an accident. It was by design after some difficult experiences. I am personally a product of the Emacs world that you are describing below, and my advisor Dan Friedman was called the "Lispman" on his door sign at Indiana.

When I first discovered the idea of sending individual expressions and definitions from a buffer to a repl, it was a near-religious revelation to me. I wanted everyone to know this trick and use it. When I started teaching the freshman course at Rice, I told our chairman so and he asked "why". I was shocked, awed, and I failed to explain to him how it mattered. He was a mathematician and I wrote it off. They don't know.

Then I started watching my sophomores and juniors at Rice in lab. Now that was a true disappointment. Few if any used this trick and when they did, they more often tripped up and got the repl into a state where they didn't know what was going on.

In the mid 90s, I wrote some more Little books with Dan, and boy, time and again, I watched him stumble across the state of the repl. I even watched him re-start the repl and load the whole buffer more often than not.

Why? In the presence of macros and higher-order functions and other beasts, it is difficult for masters of the universe with 30 years of experience to keep track of things. What do you think students with 10 or 20 days worth of experience will do? Is it really such a deep principle of computing to create the objects incrementally in the repl as opposed to thinking systematically through the design of a program?

I decided not and asked Robby to make DrScheme's repl transparent. That is, it re-starts the repl and re-loads the buffer every time. I consider this behavior a suitable compromise: have a repl but don't confuse yourself with send-defs and send-exprs. This is especially true in an age when sending an entire buffer takes as much time as sending an individual expression or definition. Soon we'll get "compilation behind your back" so that only the current buffer is re-interpreted. It'll start things even faster.

Even though I had used the incremental mode for more than a decade when I switched from Emacs to DrScheme in 1998, I have hardly ever looked back. I miss a few other things but the incremental repl is one of those rituals old Lispers acquired and never questioned ... but it isn't fundamental and critical to anything. (Note there is no qualifying clauses, no when no if no but. I really mean this last sentence the way I spelled it.)

PLT Scheme v4.1.5

2009-03-22T01:29:00.001-04:00

PLT Scheme version 4.1.5 is now available from

  http://plt-scheme.org/

Web Server:
- new URL-based dispatching library web-server/dispatch,
- customizable continuation serialization policies for the stateless web language web-server/stuffers,
- abstraction of serve/servlet to build simpler dispatchers web-server/servlet-env,
- HTTP Digest authentication support web-server/http/digest-auth,
- built-in cookie support in web-server/http/cookie and web-server/http/cookie-parse,
- highlighting and pretty-printing of errors in Xexpr constructions,
- load control dispatcher for limit concurrent sessions web-server/dispatchers/limit.
Scribble:
- Literate programming is now available using the new scribble/lp language.
- A new at-exp language makes it convenient to use the scribble reader's @-expressions for general code.
- The scribble/text preprocessor language has been extended to deal with indentation and other formatting issues.
- The "scribble" command-line tool accepts a --pdf flag to render PDFs (via pdflatex).
DrScheme now provides feedback when PLaneT downloads and installs new packages.
Units & Contracts:
- Elements of unit signatures can now be associated with contracts via the contracted signature form.
- A contract combinator for units, unit/c, has been added.
- The new with-contract form provides a nestable submodule contract boundary, protecting a group of definitions from the surrounding code with contracts.
- The define/contract form has been reworked and new define forms have been added: define-struct/contract and define-unit/contract.
Language levels and teachpacks from the DeinProgramm project for the German textbook "Die Macht der Abstraktion" by Herbert Klaeren and Michael Sperber have been added.
Misc:
- Typed Scheme now comes with several more pre-wrapped libraries, found in the typed collection.
- The xml and html collections are now contracted.
- Binding parsing in net/cgi now interacts with net/uri-codec's configuration parameters.
- DrScheme captures logging output.
- Check syntax: it is now possible to tack arrows crossing the currently selected text.
- New bitwise-bit-field function.
The usual pile of bugfixes. (Notable: scheme/package works, deflate compression fixed, DrScheme language dialog issue resolved, match fixes, Windows networking, and much more.)

[Note that mirror sites can take a while to catch up with the new downloads.] Feedback Welcome.

Maintaining self-references in Planet packages

2009-03-12T16:35:00.030-04:00

PLaneT packages may refer to themselves (i.e. include module paths referring to some part of the same package) for a number of reasons. One module may require another. Scribble documentation traces for-label imports to construct hypertext links. DrScheme language levels may provide a module path for an initial namespace.

In each of these cases, we want the module path to refer to the same version of the same package that it occurs in. On the other hand, we do not want to have to manually search and replace the version number every time we update. Before I solved this problem I would often release some new version x.0 of a package, only to find some lingering dependency on y.0 that my search/replace had not caught. Of course, then I had to go back and replace all occurrences of both x.0 and y.0 with x.1 and release again. To avoid this headache, we need a way to express self-referential module paths with precise, implicit version numbers.

The built-in module paths don't quite support this. The relevant forms are PLaneT paths with version numbers, PLaneT paths without version numbers, and relative paths:

(planet my/package:1:0/dir/file)
(planet my/package/dir/file)
"../dir/file.ss"

PLaneT paths with version numbers suffer from the search and replace problem: they become obsolete, and must be changed with every new release.

PLaneT paths without version numbers "upgrade" with a new release: they automatically refer to the latest version of a package. Unfortunately, this means they aren't really "self"-references. As soon as version 2.0 is released, every version-free reference to the package refers to 2.0. Even the references in version 1.0 get implicitly updated, and become forward references rather than self-references.

Relative paths are precise, in that they always refer to the same version of the same package. However, because they implicitly refer to the directory containing the source code, they are only valid within a single file. They cannot be reliably passed to DrScheme for a language level namespace, traced for documentation links by Scribble, or used by other such external tools.

None of these options provides precise, stable, externally comprehensible, self-referential module paths.

To fill this need, I have released (planet cce/scheme:4:1/planet). This module provides PLaneT package authors with several macros that construct references to the current package in require imports, Scribble documentation, and dynamic values. The self-referential modules paths are constructed automatically at compile time based on the source location of the macro use and the local PLaneT package database. Their expanded form always includes an explicit package name and version number (both major and minor). Here I will summarize their use, with (planet my/package:1:0/dir/file) as a running example. For full details, see the online documentation or install the package.

To import a module from within a PLaneT package, use the this-package-in require transformer. To re-export bindings from a module imported this way, use the this-package-out provide transformer, or use require/provide/this-package in place of both.

For example, you might want to import and re-export bindings from dir/file:

(require (planet my/package:1:0/dir/file))
(provide (all-from-out (planet my/package:1:0/dir/file)))

You can leave out the package name and version number, thus making the code invariant across upgrades, by writing:

(require (this-package-in dir/file))
(provide (this-package-out dir/file))

Or, you can further simplify it:

(require/provide/this-package dir/file)

All three of the above are equivalent (in version 1.0).

In Scribble documentation, a module often refers to itself via defmodule, declare-exporting, or schememodname. I provide the extensions defmodule/this-package, declare-exporting/this-package, and schememodname/this-package, respectively. These macros allow the user to supply a path such as dir/file, or to omit one to refer to the package as a whole (or its main module). In positions where the original macros allow a sequence of module paths, these forms allow two sequences, one for self-referential module paths and one for other paths.

To document an entire module, one might first write:

(defmodule (planet my/package:1:0))

The automatic self-reference version is simply:

(defmodule/this-package)

In more complicated cases, one might document a sub-part of a package or present bindings from multiple sources:

(defmodule (planet my/package:1:0/dir/file)
  #:use-sources
  [(planet my/package:1:0/private/impl) somewhere/else])

These self-references can still be automated:

(defmodule/this-package dir/file
  #:use-sources
  [private/impl]
  [somewhere/else])

Finally, I provide this-package-version-symbol for constructing PLaneT package symbols as runtime values. This macro is analogous to this-package-version from the planet/util collection, but it constructs a symbol rather than an s-expression. You can use this symbol to construct a module path for a DrScheme language level, or escape it with unsyntax in Scribble's schemeblock typesetting to create self-referential example code.

This list of utilities may not be complete. Users may need to write new macros for self-referential PLaneT packages. To that end, (planet cce/scheme:4:1/syntax) provides syntax-source-planet-package. This function is analogous to this-package-version, but operates on syntax objects and is designed to be used in macro transformers. There are also -owner, -name, -major, -minor, and -symbol versions following the same analogy.

I find these tools useful for maintaining my PLaneT packages, and I hope other authors will too. If you do give them a try, please send feedback on what does or doesn't work, or what might be improved. I would eventually like to add a refined version to the PLT Scheme collections once we get enough experience to know that these tools are fairly complete and usable.

Web Application Development in PLT Scheme

2008-10-06T15:33:00.006-04:00

Many users often post to the plt-scheme mailing list asking for introductions to Web application development in PLT Scheme. They've heard of the continuation-based PLT Web Server and want a gentle introduction. Unfortunately, there has been a distinct lack of good documentation and tutorials for the server. Taking the cue from two users: Jens Axel Soegaard and David Reynolds, we've written a tutorial with Danny Yoo.

The tutorial is available at http://docs.plt-scheme.org/continue/index.html. It walks through the creation of a blog application, introducing features slowly and culminates in an SQL-backed database for the posts. Of particular interest, is the fast start servlet setup based on the Instaservlet package from Untyped.

Please take a look and pass along this as a pointer to those who may be interested in PLT Scheme.