03 Oct 2010

The Two-State Solution: Native and Serializable Continuations Accord

posted by Jay McCarthy

The Racket Web Server allows an expressive way of writing Web applications using first-class continuations to capture the control-flow of the server while it is waiting for the client to respond. For example:

#lang web-server/insta
(define (get-number p)
  (string->number
   (extract-binding/single
    'num
    (request-bindings
     (send/suspend
      (λ (k-url)
        `(html 
          (body
           (form ([action ,k-url])
                 ,p nbsp (input ([name "num"]))
                 (input ([type "submit"])))))))))))
(define (start req)
  (define how-many
    (get-number "How many numbers to add?"))
  (number->string
   (foldr 
    + 0
    (build-list 
     how-many
     (λ (i)
       (get-number 
        (format "Provide number: ~a" 
                (add1 i))))))))

This application creates a re-usable get-number interaction abstraction and uses it in a number of different contexts. In particular, it uses it in the higher-order context of build-list. This application also reuses useful third-party library functions like foldr, etc.

Such an application would be complicated to write in a traditional Web programming environment because the continuation of each get-number invocation is considerably more complex than is typical. Yet, the first-class continuations in Racket ensure that this continuation is captured exactly, correctly, every time.

Unfortunately, the native first-class continuations of Racket are not serializable, so they impose a per-session resource expenditure on the server. This can be alleviated through expiration policies, but such policies are inherently unsound because continuations URLs are global roots.

In the past, PLT has provided tools that automatically restructure this kind of program into one that uses serializable continuations through an acronym soup of source transformations: CPS, lambda-lifting, defunctionalization, SPS, and so on. These tools effectively create automatically what most Web programmers write manually, except the tools don’t mistakes. But the tools also don’t take into consideration what functions actually contribute to the interaction context and transform library functions like foldr (which is unnecessary in the continuation) the same as functions like build-list (which are necessary.)

Our past work (based on another PLT paper) alleviates this problem by only requiring functions like build-list to be transformed. From the perspective of a programmer, “transformed” is tantamount to “rewritten” because the source code for a third-party library may not be readily available. Programmers would have to program add-many-numbers.com as:

#lang web-server
(require web-server/servlet-env)
(define (get-number p)
  (string->number
   (extract-binding/single
    'num
    (request-bindings
     (send/suspend
      (λ (k-url)
        `(html 
          (body
           (form ([action ,k-url])
                 ,p nbsp (input ([name "num"]))
                 (input ([type "submit"])))))))))))
(define (build-list n f)
  (for/list ([i (in-range n)])
    (f i)))
(define (start req)
  (define how-many
    (get-number "How many numbers to add?"))
  (number->string
   (foldr 
    + 0
    (build-list
     how-many
     (λ (i)
       (get-number 
        (format "Provide number: ~a"
                (add1 i))))))))
; This requires a pre-release version
; to run in an un-named DrRacket buffer
(serve/servlet start #:stateless? #t)

where build-list has been re-implemented, but functions like foldr have not. This application, despite its striking similarity to the first, requires absolutely no per-session server state, so it is considerably more scalable.

Do we need to re-implement build-list? What if the third-party, higher-order function (build-list) that we use with a higher-order argument that causes Web interaction (get-number) is too complicated to re-implement?

Naturally this blog post would not exist if we didn’t solve this problem.

Our new approach, dubbed The Two-State Solution, allows the programmer to transparently use a very small amount of per-session server state to store just the part of the continuation inside functions like build-list while serializing everything else to the client.

The key is to use delimited, composable continuations to isolate the appropriate part of the continuation. The programmer designates this piece of the continuation through the serial->native and native->serial annotations. The programmer can write the application as:

#lang web-server
(require web-server/servlet-env)
(define (get-number p)
  (string->number
   (extract-binding/single
    'num
    (request-bindings
     (send/suspend
      (λ (k-url)
        `(html 
          (body
           (form ([action ,k-url])
                 ,p nbsp (input ([name "num"]))
                 (input ([type "submit"])))))))))))
(define (start req)
  (define how-many
    (get-number "How many numbers to add?"))
  (number->string
   (foldr 
    + 0
    (serial->native
     (build-list
      how-many
      (λ (i)
        (native->serial
         (get-number 
          (format "Provide number: ~a"
                  (add1 i))))))))))
; This requires a pre-release version
; to run in an un-named DrRacket buffer
(serve/servlet start #:stateless? #t)

The important distinction here is that both the build-list and the get-number abstractions do not need to change. We simply mark the context as being a “serial” or “native” context through the annotation forms. This re-written version will be more scalable than a purely native version, but represents an easier to achieve step in the evolution of a program, because third-party, higher-order functions can be used as is.

This work will be presented at OOPSLA 2010. It is also described in a paper with same name this blog post:The Two-State Solution: Native and Serializable Continuations Accord.

more →

15 Sep 2010

Extending Typed Racket, Part 1

posted by Sam Tobin-Hochstadt

The Typed Racket team is pleased to announce a number of new additions to our system. We’ll be writing a few blog posts about them, all of which you can read here.

This post begins with the core of the Typed Racket type system. The fundamental idea at the heart of Typed Racket is called occurrence typing. This is the technique that allows us to typecheck existing Racket programs without requiring rewrites. Here’s a simple example:

(if (number? x) (add1 x) 0)

The typechecker can figure out from the use of number? that the second occurrence of x is always going to be a number. This simple form of occurrence typing is enough to take Typed Racket a long way. But because we want to be able to handle all the sophisticated reasoning that programmers are already using to write their Racket programs, we have been working on extending the system further.

The new design of our system is described in a paper, Logical Types for Untyped Languages, in the upcoming International Conference on Functional Programming. The introduction provides an overview that’s acessible to any Racket programmer, but here’s the key example:

(cond
  [(and (number? x) (string? y))  1 ]
  [(number? x)                    2 ]
  [else                           3 ])

In expression 1, we know that x is a number and y is a string. In 2, we know that x is a number and y is not a string, by the logical properties of and and cond. This form of logical reasoning is enabled by the new foundation of the system, and makes the entire system significantly more expressive.

All of these improvements are available in the current version of Racket.

more →

03 Aug 2010

Racket v5.0.1

posted by Eli Barzilay

Racket version 5.0.1 is now available from http://racket-lang.org/

  • Datalog is a lightweight deductive database system with Racket integration. It is now available in the datalog collection and with #lang datalog.

  • Racklog provides Prolog-style logic programming in Racket, adapted from Dorai Sitaram’s Schelog package. It is available in the racklog collection and now as #lang racklog.

  • By default make install and raco setup compile collections in parallel on all available processors. (Use raco setup -j 1 to disable, if necessary.)

  • Changes (as part of 5.0) in the racket language compared to the scheme language: constructor-style printing, a struct alternative to define-struct that fits more naturally with match and constructor-style printing, bytecode-dependency management via SHA–1 hashes instead of just timestamps (where the openssl/sha1 library provides the SHA–1 hash function), a reorganization of scheme/foreign into ffi/unsafe and associated libraries, and new printing functions eprintf and displayln. Also, a generator from racket/generator is required to have the form (generator () body ...), which supports a planned extension to let a generator accept arguments.

  • Changes to the racket language (since 5.0): internal-definition positions allow mixing expressions with definitions, full continuations can escape past a continuation barrier, custodians can attempt to terminate subprocesses and subprocess groups (see current-subprocess-custodian-mode, subprocess-group-enabled), the JIT supports additional unboxing flonum operations and unsafe variants, ffi/unsafe provides an asychronous-call mechanism to deal with foreign threads, a new "." modifier for format string directives (e.g., "~.s" and "~.a") limits the respective output to (error-print-width) characters.

  • The core type system of Typed Racket has been substantially revised. In particular, Typed Racket can now follow significantly more sophisticated reasoning about the relationships between predicates. Additionally, Typed Racket now allows variable arity types in more places, allowing programmers to specify variable-arity lists.

  • We are working on an optimizing version of Typed Racket that takes advantage of type information for certain classes of programs. This project is a work in progress. For those interested, see the documentation for #:optimized.

  • The web-server/formlets library adds a formlet* form that allows dynamic formlet construction, as opposed to formlet which requires syntactic Xexprs and static formlets. Several new library formlets are added.

  • The syntax/parse library has new support for matching literals at different phases using the #:phase argument for literals and literal sets.

  • RackUnit now includes a GUI test runner as rackunit/gui.

  • The 2htdp/image library now includes flip-vertical and flip-horizontal operations that mirror images (vertically and horizontally).

more →

07 Jun 2010

Racket

posted by Eli Barzilay

PLT is happy to announce the release of Racket, available from http://racket-lang.org/

With Racket, you can script command shells and web servers; you can quickly prototype animations and complex GUIs; regexps and threads are here to serve you. To organize your systems, you can mix and match classes, modules or components. Best of all, you start without writing down types. If you later wish to turn your script into a program, equip your Racket modules with explicit type declarations as you wish. And Racket doesn’t just come as a typed variant; you can also write your modules in a purely functional and lazy dialect.

Racket comes in so many flavors because Racket is much more than a standard scripting language or a plain programming language. Racket supports language extensibility to an unequaled degree. A Racket programmer knows that making up a new language is as easy as writing a new library.

To help you start quickly, Racket includes batteries in all shapes and sizes, most importantly, extensive documentation and all kinds of libraries.

Racket occupies a unique position between research and practice. It inherits many major ideas from language research, among them type safety (when the type system says that x is a number, then at runtime it always is a number) and memory safety (when some memory is reclaimed by the garbage collector it is impossible to still have a reference to it). At the same time, user demand governs rigid adherence to purely theoretical principles.

Racket, formerly PLT Scheme, is a product of over 15 years of development. Although Racket starts with a mature software base and an established user community, its new name reflects our view that this is just the beginning of Racket’s evolution.

more →

02 Apr 2010

PLT Scheme v4.2.5

posted by Eli Barzilay

PLT Scheme version 4.2.5 is now available from http://plt-scheme.org/

  • PLT now supports multi-core parallelism via futures. Futures create tasks that run in parallel, as long as the tasks stay in the “fast path” of the runtime system. For more information, see the guide.
  • Our unit testing framework, schemeunit, is now included in the distribution. A graphical test runner is available via schemeunit/gui.
  • The support languages for the “Programming Languages: Application and Interpretation” textbook by Shriram Krishnamurthi are now part of PLT Scheme. In addition the PLAI GC language comes with a random mutator generator (to help test collectors) and an improved heap visualizer.
  • New Russian and Ukranian translations, thanks to Sergey Semerikov.
  • A number of improvements to Redex’s typesetting facilities.
  • Typed Scheme users can now automatically generate predicates from types with define-predicate. Typed code can be inserted in untyped modules by requiring with-type from typed/scheme.
  • The scheme/class library now provides contract combinators for classes (class/c) and objects (object/c). See the Reference and Guide for details. Also, a backwards-compatible object-contract version of object/c has replaced the old object-contract combinator.
  • Writing new kinds of contracts is now easier with keyword-based constructors (make-contract and make-flat-contract), a simpler set of structure properties (prop:contract and prop:flat-contract), and the introduction of blame objects for tracking contract metadata.
  • The Scheme-implemented bytecode reader fails less often. This is used by “mzc —decompile”. The Scheme-implemented bytecode writer uses the compact bytecode format and fails less often. This may be used in the future for Scheme-implement bytecode processors.
  • The language dialog now suggests using "#lang" more strongly as the default language. DrScheme no longer uses the term `Module language’.
more →

08 Mar 2010

Talk at Flourish

posted by Robby Findler

The image in this post shows a tree where the interior nodes represent directories and the leaf nodes represent files in the PLT source code. The leaves are colored based on the programming language used. (To avoid clutter, if there is more than one file in a given directory written in a particular language, that language only gets a single dot.)

Some highlights: the blues are Scheme-like languages, the reds are langauges we use to write documentation (see Scribble for more about them), the greens are teaching languages, orange is the language we use to bootstrap new languages, and yellow is a language for metadata about nearby files.

Curious about how we managed to write and use so many different languages? I’ll be giving a talk at Flourish 2010 next week (3/19 @11am, UIC in Chicago) explaining how. Come to learn more!

more →

28 Feb 2010

DAGs vs Trees

posted by Robby Findler

As I wondering whether or not there is a better layout algorithm for the module browser window, I looked into tree maps. Of course, the modules in a program form a DAG, not a tree, so I wondered just how big the tree would get if all of the shared structure in the DAG were replicated. Hey, I figured, if a tree map can handle showing me my entire filesystem, maybe that could work.

… yeah, no. Turns out to be hopeless. In the spirit of a geeky take off on a jelly bean counting contest, lets see if you can guess just how big these things get. Consider the module graph from the program #lang scheme (ie, the graph that just contains an empty program). This program loads 170 modules with 917 connections between modules (counting the main file that just contains the #lang scheme).

So, the question: how many nodes are there in the unsharified tree? First one to come within 1 billion of the right answer gets all of the fame and glory that this blog brings to bear (har har). I’ll post the answer in the comments in a few days (and no fair cheating, those of you that know enough to be able to get your hands on the DAG).

more →

29 Jan 2010

Benchmarks

posted by Matthew Flatt

First, the usual disclaimer:

That said, I’ve run the latest version of PLT Scheme on two sets of benchmarks:

  • Benchmarks in the PLT sources – vs. Bigloo, Chicken, Gambit, Guile, Ikarus, Larceny, MIT Scheme, and Scheme48; safe operations and generic arithmetic only

  • Benchmarks in the Gambit sources – vs. Bigloo and Gambit; generic vs. fixnum-/flonum-specific arithmetic, safe vs. unsafe operationsThe second set is why I started running benchmarks. Fixnum-/flonum-specific arithmetic and unsafe operations are new in PLT Scheme 4.2.4. The benchmark results suggest that the new operations in PLT Scheme offer roughly the same performance benefits as in Bigloo and Gambit. There’s room for improvement, but it’s a good first cut.

For the other results: PLT Scheme is rarely the fastest implementation on a given benchmark. For most purposes, though, it’s in the same ballpark – except for programs that spend all their time capturing and invoking continuations.

It’s fun to run benchmarks occasionally. Now, back to working on language design, libraries, documentation, usability…

more →

28 Jan 2010

PLT Scheme v4.2.4

posted by Eli Barzilay

PLT Scheme version 4.2.4 is now available from http://plt-scheme.org/

  • The scheme/flonum and scheme/fixnum libraries provide flonum- and fixnum-specific operations. In the case of flonum-specific operations, the JIT compiler can recognize combinations of operations (including local bindings) and improve performance by “unboxing” intermediate results.
  • The scheme/unsafe/ops library provides arithmetic and other operations that are implemented without dynamic checks. Avoiding checks can sometimes improve performance, but at the expense of safety.
  • 2htdp/universe: We have severed the connection between universe and an image library and made a few other, minor changes. Most programs will now have to change to require the htdp/image library explicitly. For the full details, see the new Porting World Programs section of the documentation.
  • The 2htdp/image library continues to grow. In this release, it is supported by 2htdp/universe, equality changed to be based on how the images are drawn, cropping and curves were added, and support for more kinds of pens were added.
  • htdp/world: The old world teachpack remains deprecated. HtDP/2e exclusively uses the new 2htdp/universe library. For backwards compatibility, the world teachpack will remain in the distribution until the coming summer.
  • The scheme/class library now provides this%, which refers to the class of the current object (i.e. this).
  • scheme/generator has convenient functions for infinite generators, and for converting a generator to a sequence for iteration.
  • PLT Scheme’s add-on directory can be customized by the $PLTADDONDIR environment variable or --addon/-A command-line flags. This controls where downloaded Planet packages and their compiled Scribble documentation are installed.
  • Additional extensions include: saving errno in foreign calls, much improved sort speed, normalized results from procedure-arity, and more.
more →

02 Jan 2010

Scheme Videos (Lectures and Talks)

posted by John Clements

Scheme Videos (Lectures and Talks)(thanks to Geoffrey Knauth and Hari)Following a mailing-list request, it turns out that there are quite a lot of Scheme-related lectures and talks floating around out there in video format. The following list was compiled by Geoffrey Knauth, with contributions from Hari and Michael Sperber, and at least one insertion from me, right at the front.

  • My sequence of introductory videos on YouTube, recorded long after this post was made.

  • There’s the SICP course Abelson & Sussman gave to [HP, I think] in the mid–1980s: http://groups.csail.mit.edu/mac/classes/6.001/abelson-sussman-lectures/

  • MIT OCW / 6.001 using SICP, Spring 2005: http://ocw.mit.edu/OcwWeb/Electrical-Engineering-and-Computer-Science/6–001Spring–2005/VideoLectures/index.htm

  • All the ICFP 2009 videos (man this made my day!!): http://vidiowiki.com/feature/list/fnu/ICFP_2009

  • Daniel P Friedman - A Celebration (this too!): http://www.cs.indiana.edu/dfried_celebration.html

  • DrScheme v4.0 Tour: http://www.youtube.com/watch?v=vgQO_kHl39g&fmt=18

  • Similar, if you understand Russian:

  • http://www.youtube.com/watch?v=wECY7s9k-V0

  • http://www.youtube.com/watch?v=2CVJjqOT6WM

  • Matthias Felleisen - Programming at Northeastern: http://www.savevid.com/video/matthias-felleisen-programming-at-northeastern-university.html

  • Matthew Flatt - Processes without Partitions: http://www.researchchannel.org/prog/displayevent.aspx?rID=3892

  • Shriram Krishnamurthi on WeScheme: http://vidiowiki.com/watch/cydr9yk/

  • Robby Findler - Macros Matter: http://www.mefeedia.com/video/26348171

  • Using PLT Scheme in the Game Industry: http://www.youtube.com/watch?v=2CVJjqOT6WM

  • Stanford Lecture (Kawa): http://www.youtube.com/watch?v=_cV8NWQCxnE

  • Bluetooth communication using PLT Scheme: http://www.youtube.com/watch?v=pmR_dIXm6sY

  • SICP at UCB: http://webcast.berkeley.edu/course_details.php?seriesid=1906978454
  • http://www.aduni.org/courses/sicp/ from ADUni by Holly Yanco. It comes with pretty good lecture notes and problem sets.

  • Michael Sperber’s DMdA lectures (in German, natch): http://timms.uni-tuebingen.de/List/List01.aspx?rpattern=UT_200[89]_____00[12]info1_000

more →

Made with Frog, a static-blog generator written in Racket.
Source code for this blog.