2014-12-10

The Racket package system and Planet

We have recently moved the majority of Racket's code base into packages and repositories separate from the main core repository. This has given the Racket package system another cycle of attention. Whenever this happens, there are often questions and confusion about how to solve various distribution problems with the package system. A natural point of comparison is the older Planet system provided by Racket that appears to solve similar problems. In this blog post, I attempt to explain the purpose of the package system and its relation to Planet.


The package system and Planet do not solve the same problem and don't exist for the same reason.


Planet is:

  1. A file distribution mechanism for source code.


    Via .plt files that are installed into a particular place on your machine and then raco setup'd.

  2. A mechanism for automatically downloading and installing source code just before it is needed by programs.


    Via the (planet ...) require form.

  3. A centralized database of libraries


    Via the Planet website and its server & protocol which were undocumented and proprietary for the majority of Planet's life

  4. A prescriptive model of how programs and libraries should be composed.


    Specifically the system of major/minor versions, tagging packages by author name, and embedding the names of packages in source code.


In contrast, the package system is:

  1. A file distribution mechanism for source code, byte code, and documentation.

    Via the raco pkg command.


In this way, the package system is almost identical to an operating system package system like Debian's dpkg and apt systems. The problem is very finely tailored and becomes more flexible as a result (notice that we can now distribute byte code and documentation.) This design aspires to follow the admonition of holy writ: "Programming languages should be designed not by piling feature on top of feature, but by removing the weaknesses and restrictions that make additional features appear necessary."


Furthermore, it was intended to solve practical problems throughout the Racket ecosystem. In particular, one of the common complaints people had and have about Planet is the very long install times because of long builds. The package system allows this problem to be solved by distributing pre-built code.


Since the package system specifically does not address jobs 2, 3, or 4 of Planet, we have to ask, "Do they need to be solved?" and if so, "How can we solve them on top of the package system, i.e. as a library in honor of the design principle?".


In particular, 2 and 3 are very painful for people wanting to just use the file distribution mechanism of Planet. 2 causes unpredictability, because you don't know if running a program will start a long invocation of "raco setup", require Internet access, and start running un-vetted code. 3 requires you to share your code if you want to use the file distribution mechanism and is a single point of failure for doing installation.


By not mandating how to address 2 and 3 in the package system, we offer flexibility. Here is where the solutions to these jobs are now:


2. There is currently no way to get automatic installs of packages. However, both DrRacket and xrepl offer advice about which packages you might want to install to compile and run the program. It would be natural to extend this advice to be automatic and patches are welcome. Given the experiences of operating systems which merely make suggestions (nethack: command not found, provided by nethack-console), I personally feel like we are at the sweet spot.


3. The file distribution mechanism's flexible package sources combine with a very simple protocol for package catalogs (Take a URL, add/pkg/, add a string, get a read-able hash table) to look up packages you don't yet have. As a service, we run a few catalogs (one for each release, plus pkgs.r-l.o). But we expect that users with special needs (such as sensitive installations that need exactly certain tested and trusted versions, especially with proprietary software) will build their own catalogs on private Web sites.


Clearly, however, job 4 is where Planet and the package system differ the most.


With the package system, we follow the precedent of operating systems. An OS package's job is to get files into the right spot. An OS package contains a binary and instructions to install it as /usr/bin/zsh. It is not typical in OSes to be able to install multiple packages (such as different "versions" of the "same" package) that both provide /usr/bin/zsh. When you're at a Unix prompt, you don't have to write zsh-5.0.5/usr/bin/zsh. It's possible that many consider this is a big problem with OSes and indeed we do observe that it is fairly common to provide packages that provide binaries and libraries with embedded names such as how on my machine I have python2.6, python2.7, and python3.2 all in my $PATH. It is important to realize, however, that the deb format and the apt tool didn't need to change to support this change or future changes in perspective in how to compose code.


I hope this analogy helps understand the Racket package system. In the package system, a package doesn't install "binaries", "man pages", and "init scripts", but installs similar things, such as "module paths", "documentation manuals", and "raco commands". Each of these has a notion of conflict: can't have two zshs or two racket/lists; can't have two zsh.1 pages or two docs named doc; can't have two modules trying to provide raco neo-tokyo-is-about-to-explode. If you find a random .deb on the Internet, can you predict what binaries it will contain from its name? No. The same goes for Racket packages. However, if you are egregiously weird, then people probably won't want to install your packages, just like for random debs.


However, clearly rules are helpful. In the world of operating systems, you know that basically all packages distributed by Debian can be installed at the same time, except for "virtual packages" that do stuff like selecting whether postfix or sendmail should be responsible for the sendmail command. These rules are not enforced through technology, though. Instead, the Debian maintainers have a social process that enforces them, with information being provided by technology (such as regression systems that identify unintended conflicts.) The catalog server that the Racket team provides helps facilitate a similar process with the concentric rings (all ring <=1 packages can be installed at once and ring 1< packages can do anything.)


Non-conflicting sets of packages is the simplest rule to define and enforce. Other rules about backwards compatibility are much more complicated to define and enforce. I do not believe there is much precedent in the world of OSes, although we can see a little bit of what they do through things like libgtk, libgtk2, and libgtk3, where generally code written for one libgtk2 package is compatible with all libgtk2 packages made in the future, but libgtk3 is effectively a totally different package and introduces totally separate binaries like gtk3-config.


The most that the Racket team attempts to do here is to say, "Here are the rules we will follow and we think you should follow them too." Specifically, that we will maintain backwards compatibility or make a new package. We can't and won't enforce this, nor do we always live up to it with our own work (but we feel really bad about it when we do.)


Although my main goal of this section has been to explain my solution to (4), a great thing about the package system is that it is not binding at all. You can decide to follow the same rules as Planet. It is easy to do so:


  • Always name your packages $AUTHOR-$PACKAGE-$MAJOR
  • Always provide modules from only the collection, $AUTHOR-$PACKAGE-$MAJOR
  • Maintain backwards compatibility within releases of $AUTHOR-$PACKAGE-$MAJOR
  • Update the 'version metadata in the package info.rkt to reflect the $MINOR version.

And, boom!, you've recreated the rules of Planet to a T except for two things: (a) you'll still need to put a dependency on $AUTHOR-$PACKAGE-$MAJOR on the outside of code in a package info.rkt file rather than just inside files and (b) you can't use $AUTHOR-$PACKAGE to refer to "whatever the current $MAJOR" is.


The first compromise of adding something to the info.rkt is fairly modest, as it requires O(1) line modifications.


The second compromise is more severe, although actually you could just maintain such a package and deal with the breakage that occurs when you try to upgrade. Such breakage, however, was present in Planet too, as when a package was installed based on $AUTHOR-$PACKAGE only the local machine would cache the version used, so if you took the requiring module to another machine, it would download a new version and, potentially, have a backwards incompatibility problem. Using the package system in the most naive way (i.e. installing the $AUTHOR-$PACKAGE at some point and programming to that) would work exactly the same as Planet, except that the package system was designed to be able to port installations from one machine to another with raco pkg migrate.


I hope this blog post has helped explain the package system and shown that it does not prevent you from doing anything that Planet let you do, it only allows you to do more.

2014-11-23

800!

800
(Racket Tasks On Rosetta Code)

Since (and even before) Asumu Takikawa’s post “200!” at the beginning of March 2013, folk have been beavering away, implementing tasks on Rosetta Code. And on November 15th 2014:

800 tasks have now been Implemented1 in Racket on the Rosetta Code website!

Before I go any further it must be said that, without a doubt… this is awesome! This achievement represents a lot of work, and a lot of code. And everyone who has participated should be thanked and congratulated for getting this far.

So thank you. And well done!

2014-11-04

Racket v6.1.1

Racket version 6.1.1 is now available from http://racket-lang.org/

  • The Mac OS X Yosemite compatibility problems are fixed. We bundled a patched Pango text-drawing library with Racket.
  • The Windows [32-bit] releases fixes the window-update crashes. We bundled a patched Cairo drawing library with Racket.
  • Typed Racket closes two safety holes in the exception system. The revised type system restricts raise to send only instances of the exn structure type and flat data to handlers. It also checks exception handlers properly. Note: Previously well-typed programs may fail to typecheck.
  • Typed Racket's typed regions support casts and predicates.
  • 2htdp/image's notion of equality ignores an image's baseline.
  • The package manager supports a binary library installation mode, which allows users to install packages without source or documentation. Use the --binary-lib option with raco pkg install.
  • The new drracket-tool-lib package factors out parts of DrRacket's IDE so that they can be reused with other editors, such as Emacs.
  • The compiler's use-before-defined analysis has been repaired for certain forms of nested letrec, some let forms, and some uses of set! or with-continuation-mark.
  • The compiler performs additional bytecode optimizations. Thanks to Gustavo Massaccesi.
  • The CML library comes with a new replace-evt event constructor. Thanks to Jan Dvořák.
  • Redex's benchmark suite comes with a description of the benchmark programs.
  • Redex's metafunctions can be typeset using the "large left brace" notation for conditionals.
  • The contract library comes with an improved contract-stronger?. Its error messages note that the contract itself might be wrong.
  • The GUI library is DPI-aware on Windows.
  • The openssl library supports Server Name Indication for servers. Thanks to Jay Kominek.
  • The syntax/parse library allows the definition of new pattern forms via pattern expanders, similar to match expanders. Thanks to Alex Knauth.
  • OpenGL on Linux no longer depends on libgtkgl, and core profiles are supported (see set-legacy?).
  • The teaching languages' unit test framework supports check-satisfied, a construct for checking whether a result satisfies a predicate, e.g.:
    (check-satisfied (sort l) sorted?)
Feedback Welcome

2014-10-07

PLT Redex Summer School, Call for Participation

LOCATION: University of Utah, Salt Lake City
DATES: July 27 - July 31, 2015
http://www.cs.utah.edu/~mflatt/plt-redex/


PLT Redex is a lightweight, embedded DSL for modeling programming
languages, their reduction semantics, and their type systems. It comes with
an IDE and a toolbox for exploring, testing, debugging, and type-setting
language models. The PLT research group has successfully used Redex to
model and analyze a wide spectrum of published models.

The summer school will introduce students to the underlying theory of
reduction semantics, programming in the Redex language, and using its
tool suite effectively.  The course is intended for PhD students and
researchers in programming languages. Enrollment is limited to 25
attendees.

While the workshop itself is free, attendees must pay for travel, room, and
board. We expect room and board to be around $500, assuming an arrival in
the evening of Sunday July 26 and leaving Friday July 31 or August 1.
Partial financial support for PhD students is available.

To register, send email to Matthew Flatt (mflatt@cs.utah.edu). If you
are a PhD student and requesting financial support, CC your advisor
and ask for a one-line confirmation email.

Literature

  Matthias Felleisen, Robert Bruce Findler, Matthew Flatt.
  Semantics Engineering with PLT Redex. MIT Press, 2012.

  Casey Klein, John Clements, Christos Dimoulas, Carl Eastlund,
  Matthias Felleisen, Matthew Flatt, Jay McCarthy, Jon Rafkind, Sam
  Tobin-Hochstadt, Robert Bruce Findler. Run Your Research: On the
  Effectiveness of Lightweight Mechanization. POPL 2012.

2014-08-02

Racket v6.1

PLT Design Inc. announces the release of Racket version 6.1 at

http://racket-lang.org/

The major innovation concerns local recursive variable definitions. Instead of initializing variables with an undefined value, Racket raises an exception when such a variable is used before its definition. (Thanks to Claire Alvis for adapting Dybvig's "Fixing Letrec" work.)

Since programs are rarely intended to produce #<undefined>, raising an exception provides early and improved feedback. Module-level variables have always triggered such an exception when used too early, and this change finally gives local bindings — including class fields — the same meaning.

This change is backwards-incompatible with prior releases of Racket. Aside from exposing a few bugs, the change will mainly affect programs that include

(define undefined (letrec ([x x]) x))

to obtain the #<undefined> value. In its stead, Racket provides the same value via the racket/undefined library (which was introduced in the previous release). Programmers are encouraged to use it in place of the pattern above to obtain the "undefined" value.

The release also includes the following small changes:

  • Plumbers generalize the flush-on-exit capability of primitive output ports to enable arbitrary flushing actions and to give programmers control over the timing of flushes (i.e., a composable atexit). New functions include current-plumber, plumber-add-flush!, and plumber-flush-all.
  • Contracts: the contract system's random testing facility has been strengthened so that it can easily find simple mistakes in contracted data structure implementations (e.g. an accidental reverse of a conditional in a heap invariant check).
  • Redex: the semantics of mis-match patterns (variables followed by _!_) inside ellipses has changed in a backwards-incompatible way. This change simplifies the patterns' semantics and increases the usefulness of these patterns.
  • Teaching languages: check-random is an addition to the preferred unit testing framework in the teaching languages. It enables the testing of students' functions that use random-number generation. (Thanks to David Van Horn (UMaryland) for proposing this idea.)
  • Upgraded and normalized versions of graphics libraries and dependencies (Pango, Cairo, GLib, etc.) that are bundled with Racket on Windows and Mac OS X. For example, FreeType support is consistently enabled.
  • Typed Racket: its standard library includes contracted exports from the Racket standard library, such as the formatting combinators of racket/format. It also supports Racket's asynchronous channels; see the typed/racket/async-channel library.
  • SSL: The openssl library supports forward secrecy via DHE and ECDHE cipher suites (thanks to Edward Lee) and Server Name Indication (thanks to Jay Kominek).
  • The mzlib/class100 library has been removed. Use racket/class instead.

2014-07-26

Scheme Workshop 2014

DEADLINE: 5 September 2014, (23:59 UTC-12)
WEBSITE: http://homes.soic.indiana.edu/jhemann/scheme-14/
LOCATION: Washington, DC (co-located with Clojure/conj)
DATE: 19 November 2014


The 2014 Scheme and Functional Programming Workshop is calling for
submissions.


Submissions related to Scheme and functional programming are welcome
and encouraged. Topics of interest include but are not limited to:


  • Program-development environments, debugging, testing
  • Implementation (interpreters, compilers, tools, benchmarks, etc)
  • Syntax, macros, and hygiene
  • Distributed computing, concurrency, parallelism
  • Interoperability with other languages, FFIs
  • Continuations, modules, object systems, types
  • Theory, formal semantics, correctness
  • History, evolution and standardization of Scheme
  • Applications, experience and industrial uses of Scheme
  • Education
  • Scheme pearls (elegant, instructive uses of Scheme)

We also welcome papers related to dynamic or multiparadigmatic
languages and programming techniques.

Full papers are due 5 September 2014.
Authors will be notified by 10 October 2014.
Camera-ready versions are due 24 Oct 2014.
All deadlines are (23:59 UTC-12), "Anywhere on Earth".


For more information, please see:


http://homes.soic.indiana.edu/jhemann/scheme-14/

See you there!

2014-05-08

Racket v6.0.1

Racket version 6.0.1 is now available from

http://racket-lang.org/

  • A new racket/undefined library exports undefined as the value currently produced by
    (letrec ([x x]) x)
    This library anticipates a future where that expression will raise an exception. The racket/undefined library will continue to offer the undefined value as a bridge between versions and as a last resort.
  • The drawing and GUI libraries provide improved support for high-resolution bitmaps and their use on Retina displays. For example, read-bitmap includes a #:try-@2x? option to trigger substitutions through the usual "@2x" naming convention.
  • Check Syntax cooperates with Typed Racket to show arrows and other Check Syntax highlighting even when there is a type error.
  • Functions provided via contract-out that have first-order contracts perform better.
  • The contract boundary between typed/untyped modules is much less expensive. Typed Racket now avoids generating contracts for places where contracts failures cannot happen.
  • Occurrence typing now works better with when/unless. Example:
    (let ((x (read)))
      (unless (number? x) (error 'bad-input))
      (add1 x))
  • Types in Typed Racket are now pretty-printed.
  • Function types can now be written in prefix style, which is now preferred and is used for printing. Infix function types are still accepted for backwards compatibility.
  • A new ->* type constructor is used for writing types for functions with optional and keyword arguments. The notation is similar to the matching contract combinator.
  • Typed Racket forms do not have a : suffix by default now. For example, the struct form replaces struct:. The suffixed versions are all provided for backwards compatibility.
  • Typed Racket now has preliminary support for classes and objects. However, it is still experimental and the APIs are subject to change.
  • Type aliases in Typed Racket now support recursion and mutual recursion. For example, (define-type (MyList X) (U Null (Pair X (MyList X)))) is now a valid type alias.
  • Plot correctly renders intersecting 3D graphs and non-grid-aligned 3D rectangles.
  • Elements in plots output in PDF/PS format have the same relative scale as in other formats. In particular, it is not necessary to adjust plot-font-size to make PDF plots look the same as PNG.