2015-08-11

Modules, Packages and Collections

Racket, the Racket docs and Racketeers use a number of terms to refer to various units of Racket code. Of those, module, package and collection refer to related but distinct concepts. Their exact  relations and distinctions can be confusing for new users. This is an attempt at explaining those concepts, what they are for, and how they relate to each other.

To begin with the smallest of the three, a file that begins with #lang and the name of a language is a module. There are also other ways to construct modules, but let's not worry about those.

A module is the basic unit of functionality for Racket code.

Once your Racket programs get larger, though, you'll want to split them over multiple modules. This allows you to organize your source better, enables separate compilation, and makes it possible for you to mix and match modules written in different Racket languages (Racket, Typed Racket, Datalog, Scribble, etc.).

That's where packages and collections come in. They help you organize your modules.

A package is an group of modules that you can install together, and that usually provide one piece of functionality. To pick a random example, take the pict3d package from pkgs.racket-lang.org. That package is a collection of modules which together implement a functional 3D engine. You can install it using raco pkg install pict3d, or via the graphical package manager in DrRacket.

So, to sum up, packages are units of code distribution.

A collection is a group of modules whose functionality is related to the same topic, for example data structures (the data collection), or wrapper libraries for use with Typed Racket (the typed collection). Modules are referred to and required using collection paths. For example, when you require racket/class, you're requiring the class module from the racket collection.

Modules within a collection do not necessarily come from the same package, and may not be developed together. For example, some data structures in the data collection are provided as part of the core of Racket, such as the integer sets in data/integer-set. Other data structures are provided by additional packages which you may need to install separately, such as the hash-array-mapped tries in data/hamt, which are provided by the hamt package. Having both of those in the data collection signals that they both provide data structures. If you develop your own data structures, putting them in the data collection is probably the right thing to do.

Many packages, however, provide functionality that does not fall under existing categories, and provide their own, new collection. For example, the pict3d package we discussed above puts its modules in the pict3d collection. For that reason, the distinction between package and collection is sometimes a bit blurred.

So, to sum up, collections are units of code classification.

The term library does not have a technical meaning in Racket. We usually use it to refer to a package, or to a set of packages that are developed together. For example, the Rackunit library is split across multiple packages: rackunit, rackunit-lib, rackunit-gui, rackunit-plugin-lib, rackunit-doc and rackunit-test. This allows packages to only depend on part of Rackunit. For example, a package for a string-processing library probably should not depend on the Racket GUI library (to be deployed on headless servers, for example), and so should depend on the rackunit-lib package for its testing, instead of on the full rackunit package, which brings in GUI support via the rackunit-gui package, and would introduce a dependency to Racket's GUI library.

Hopefully, this clarifies the Racket code organization terminology a bit.

2015-08-10

Racket v6.2.1

Racket v6.2.1 is now available from http://racket-lang.org/

Version 6.2.1 patches the recent v6.2 release in three small ways:

  • For the How to Design Programs teaching languages, DrRacket offers an option to use the old style for printing the constants true, false, and empty instead of #true, #false, and '().
  • The teaching languages come with some additional functions to match the August 2015 stable release of HtDP 2nd edition.
  • A repair to the compiler avoids an infinite loop at compile time for certain expressions that should loop forever at run time.
Feedback Welcome

2015-06-19

Racket v6.2

Racket version 6.2 is now available from http://racket-lang.org/

With this release we are taking a major step forward to get our user community even more involved than in the past. Over the past six months, we have re-organized the Racket code base into a small core code repo and many other package repos, all found on GitHub. If you have time and if you wish to get involved, please take a look at the GitHub repos and find your favorite places to learn, fix, and enhance our world.

The core repo is at https://github.com/plt/racket, and the package repos are listed at https://github.com/racket/.

core repo

  • The package manager supports a direct references to Git repositories via "git://[...]", "http://[...].git", and "https://[...].git" URLs. (Previously, only references to GitHub were supported.)
  • A --clone option for raco pkg install or raco pkg update facilitates Git-based package development. If a package X has a Git repository source, installing and updating the package pulls from the repository in a read-only mode. Using raco pkg update --clone X switches the local installation to a repository checkout that is suitable for modifying the package implementation, issuing pull requests, pushing changes, and so on.
    Using raco pkg update --lookup X switches the package back to the default installation mode.
drracket
  • Its on-line check syntax works with graphical content.
  • Increased availability of DrRacket's blueboxes, including method and constructor information.
  • The "Open Require Path" menu item supports ".." in relative pathnames.
data
  • Added data/enumerate, a library that supports efficient enumeration of data structures
redex
  • Its redex-check facility uses data (in addition to random) enumeration to try to find counter-examples.
  • Its generate-term function accepts additional arguments to return the "i"-th member of a pattern using data/enumerate (meaning it efficiently supports very large values of "i").
  • The examples collection includes Launchbury's 1993 big-step lazy semantics.
htdp
  • 2htdp/image's polygon may be built out of bezier curves instead of just straight lines (see the docs for pulled-point).
  • 2htdp/abstraction is a teachpack for instructors and students who wish to use for/* loops, match, define-type and type-cases in ISL and ISL+.
  • 2htdp/universe programs can be exported using DrRacket's executable creation mechanism and they behave properly when run independently.
typed-racket
  • Typed Racket in DrRacket displays tooltips that show the types of expressions. Tooltips are also displayed for type errors.
  • Typed Racket loads generated contracts only when needed. This reduces memory use and startup time for Typed Racket programs.
  • Typed Racket has improved support for prefab structures, future semaphores, and async channels.
  • Typed Racket understands when two different variables refer to the same thing, and updates types accordingly. This particularly improves the type checking of macros such as match.

Feedback Welcome

2015-05-03

King of the Hill on Rosetta Code

Racket is “King of the Hill” on Rosetta Code

This announcement is a follow up to “800!”.
In it I said we’d "[S]ee you at 1000!"; but you’ll understand why we stopped at this milestone.
Please read that article if you need an introduction to Rosetta Code, and the efforts being made to implement Racket tasks there, and more detail on how you can help. It is more instructive and less braggart than this post.

On Rosetta Code (RC), early in the morning on April 29th, Racket drew level with Tcl in the number of tasks that had been implemented for it. Shortly after that we could announce that:

Racket has the Most Tasks Implemented in Any Language on Rosetta Code!

Before I go into too much detail, it must be said that this is another amazing achievement. I, and I’m sure the rest of the Racket community, want to thank and congratulate everyone who has contributed to this effort.

How Did This Happen?

On the front page of RC’s site, it states its goal as:

.. to present solutions to the same task in as many different languages as possible, to demonstrate how languages are similar and different, and to aid a person with a grounding in one approach to a problem in learning another.

As well as achieving these comparative goals, implementing tasks also provides a useful library of tools, applications and examples for Racket users themselves. Therefore, doing so is a laudable activity in its own right. The persistent effort and progress have been made by Racketeers on RC, both before and since the “800!” tasks post has been (mostly) performed in that spirit. And that should be plenty enough incentive for you to do so, too.

But I admit, there is a competitive element that creeps in (affecting some more than others). After having passed the 800 task mark… after spending so much time in second place… to get past the current leader, Tcl… to stay ahead of Python1… these, too, provide plenty of motivation to implement tasks. And if winning isn’t important, why, then, do we keep score?

And in that spirit, early in the morning on April 29th, I was busily cherry-picking2 tasks on Rosetta Code to help close the gap with Tcl; when I thought I would take a quick check on Tcl’s and Racket’s task counts3. From what I could see, both had a task count of 845! Racket had drawn level with, Tcl as the Joint Most Popular Programming Language on RC.

I got the independent verification of this from the #racket IRC Channel4. It was true! But Racket was only joint first. This point was not lost on the denizens of IRC (zedoary being one); who posted two more tasks in very quick succession, bringing Racket up to 847 — two clear of the previous leader!

How does this Help Racket?

Plenty of Examples

Look back at the intentions of Rosetta Code itself. It is expected that users of other languages can come and compare what they know with what Racket provides. Strictly speaking, of course, in a lot of cases they won’t be able to compare since the other language won’t be represented whereas Racket will.

There is also, now, a large collection of Racket examples, which Racketeers themselves can use to improve their understanding of Racket. Strangely, this is not actually one of the stated objectives of RC; it is a welcome side-effect of the work.

A Tool for Advocacy

Advocates of Racket can use this position on Rosetta Code to show that Racket is as, if not more, capable than any language. Especially for general purpose computing.

“Racket is Number One on Rosetta Code” isn’t a bad place to start with, I guess.

Additionally, I would like to point out that whatever any of the other languages (or tasks) seem to throw at it, there is something in Racket that allows it to take it in its stride. Sometimes the implementations have had high line counts5; but they rarely, if ever, seem contrived.

If you need to provide reasons for tasks not being implemented in Racket, here are a few you can use:

  • Nobody has implemented them “yet”: let it be known that we’ve done the best part of 850 tasks, and there are only so many hours in the day.
  • Someone has written an FFI for Tcl to an obscure library: The task for Tcl has then simply been to load the FFI. The task for Racket is either to a) implement the library, which is much more effort than Tcl put in or b) to produce FFI bindings itself, which after the first time doesn’t bring much to the party. The same holds true for tasks written for languages which are basically DSLs, showing off how they work in domain for which they are specific.
  • The task is written and documented entirely in Russian: This makes translating it an “exercise.”

Is it Time to Rest on our Laurels?

That was a rhetorical question.
Please ignore it.

There are many reasons to continue to work on Rosetta Code.

We Haven’t Finished

Implement Some Outstanding Tasks!

There are 922 tasks on Rosetta Code. 849 are implemented in Racket (more have been added as we speak)! Even excluding the impossible and Russian tasks, that’s still many more tasks to implement.

Improve Existing Tasks!

Some tasks are old, and lack style. Some may even be re-branded Scheme tasks. Anyone can edit these tasks. Add style to them. Tasks can then not only be an example of how to use the syntax and features of Racket, but also exemplars of well-written code.

Propose New Tasks!

There are things that Racket and other Lisps do well that haven’t been illustrated on RC. How about the fancier macro facilities that Racket provides?

I’m sure you can think of something. Might you suggest something involving anaphoric macros?

Oh, and if you do suggest something, maybe you can implement it, too!

They Haven’t Finished

New Tasks are Being Invented!

Tasks are being added to Rosetta Code constantly. Keep an eye out, some of these are really quite interesting.

Tasks are Being Implemented!

Tcl and Python (and maybe others in the future) will want what we have earned here, and they are going to continue to propose and implement tasks. “King of the Hill” is a precarious place. The more clear blue water between us and them… Just do it!… Buy glucose sports drinks…

Maybe I am getting too competitive.

Finally

Once again, many thanks to the people who have contributed to Racket on Rosetta Code. Including those who have answered questions on the mailing list or IRC. Your help has been invaluable even if the questions made you wonder “why on earth does he or she want to do that?

Finally, but certainly not least: Thanks to the folk at Rosetta Code. They’ve provided a site and experience which have been instructive, educational and fun; and without whom none of this would have been possible.


  1. Python, is also doing magnificently well, to be sure. It even had the audacity to draw level with Racket according to the FUPPLR a couple of times. 

  2. A good way to start on Rosetta Code is to find tasks that are easy to implement. In order to find easy tasks you will need to browse the unimplemented tasks (and maybe some implemented ones, too) and decide what you could either implement and/or translate without breaking too much of a sweat. In the process you will also develop a sense of what tasks are out there ready to be implemented. A good example of an easy task would have been Pentagram

  3. There is a Frequently Updated Popular Programming Languages Report, which I refer to but recently it has been miscounting tasks, and needs a bit of a look at. 

  4. The #racket IRC channel is a fantastic community if you need support with your Racket issues 

  5. Remember that Rosetta Code is not a Golf site. If it were, J’s weird 20-character-strings-that-do-anything (if only you could remember what they do 30 seconds after you’ve written them) would win hands down. Keep to the Style Guide as best you can. And since RC is a wiki, if you’re not perfect, others can improve the style of your code. 

2015-04-20

Scheme Workshop 2015

Call For Papers:

Scheme and Functional Programming Workshop 2015
Vancouver, British Columbia, Canada
(Co-located with ICFP 2015)
http://andykeep.com/SchemeWorkshop2015/

Submissions related to Scheme, Racket, Clojure, and functional programming are welcome and encouraged. Topics of interest include but are not limited to:

  • Program-development environments, debugging, testing
  • Implementation (interpreters, compilers, tools, benchmarks, etc.)
  • Syntax, macros, hygiene
  • Distributed computing, concurrency, parallelism
  • Interoperability with other languages, FFIs
  • Continuations, modules, object systems, types
  • Theory, formal semantics, correctness
  • History, evolution and standardization of Scheme
  • Applications, experience and industrial uses of Scheme
  • Education
  • Scheme pearls (elegant, instructive uses of Scheme)
We also welcome submissions related to dynamic or multiparadigmatic languages and programming techniques.


Important Dates:
May 22nd, 2015 - Paper deadline
June 26th, 2015 - Author notification
July 19th, 2015 - Camera-ready deadline
September 4th, 2015 - Workshop

Submissions must be in ACM proceedings format, no smaller than 9-point type (10-point type preferred). Microsoft Word and LaTeX templates for this format are available at: http://www.acm.org/sigs/sigplan/authorInformation.htm

Submissions should be in PDF and printable on US Letter.

To encourage authors to submit their best work, this year we are encouraging shorter papers (around 6 pages, excluding references). This is to allow authors to submit longer, revised versions of their papers to archival conferences or journals. Longer papers (10--12 pages) are also acceptable, if the extra space is needed. There is no maximum length limit on submissions, but good submissions will likely be in the range of 6 to 12 pages.

More information available at: http://andykeep.com/SchemeWorkshop2015/

Organizers:
Andy Keep, Cisco Systems Inc. (General Chair)
Ryan Culpepper, Northeastern University (Program Chair)

(Apologies for duplications from cross-posting.)

2014-12-10

The Racket package system and Planet

We have recently moved the majority of Racket's code base into packages and repositories separate from the main core repository. This has given the Racket package system another cycle of attention. Whenever this happens, there are often questions and confusion about how to solve various distribution problems with the package system. A natural point of comparison is the older Planet system provided by Racket that appears to solve similar problems. In this blog post, I attempt to explain the purpose of the package system and its relation to Planet.


The package system and Planet do not solve the same problem and don't exist for the same reason.


Planet is:

  1. A file distribution mechanism for source code.


    Via .plt files that are installed into a particular place on your machine and then raco setup'd.

  2. A mechanism for automatically downloading and installing source code just before it is needed by programs.


    Via the (planet ...) require form.

  3. A centralized database of libraries


    Via the Planet website and its server & protocol which were undocumented and proprietary for the majority of Planet's life

  4. A prescriptive model of how programs and libraries should be composed.


    Specifically the system of major/minor versions, tagging packages by author name, and embedding the names of packages in source code.


In contrast, the package system is:

  1. A file distribution mechanism for source code, byte code, and documentation.

    Via the raco pkg command.


In this way, the package system is almost identical to an operating system package system like Debian's dpkg and apt systems. The problem is very finely tailored and becomes more flexible as a result (notice that we can now distribute byte code and documentation.) This design aspires to follow the admonition of holy writ: "Programming languages should be designed not by piling feature on top of feature, but by removing the weaknesses and restrictions that make additional features appear necessary."


Furthermore, it was intended to solve practical problems throughout the Racket ecosystem. In particular, one of the common complaints people had and have about Planet is the very long install times because of long builds. The package system allows this problem to be solved by distributing pre-built code.


Since the package system specifically does not address jobs 2, 3, or 4 of Planet, we have to ask, "Do they need to be solved?" and if so, "How can we solve them on top of the package system, i.e. as a library in honor of the design principle?".


In particular, 2 and 3 are very painful for people wanting to just use the file distribution mechanism of Planet. 2 causes unpredictability, because you don't know if running a program will start a long invocation of "raco setup", require Internet access, and start running un-vetted code. 3 requires you to share your code if you want to use the file distribution mechanism and is a single point of failure for doing installation.


By not mandating how to address 2 and 3 in the package system, we offer flexibility. Here is where the solutions to these jobs are now:


2. There is currently no way to get automatic installs of packages. However, both DrRacket and xrepl offer advice about which packages you might want to install to compile and run the program. It would be natural to extend this advice to be automatic and patches are welcome. Given the experiences of operating systems which merely make suggestions (nethack: command not found, provided by nethack-console), I personally feel like we are at the sweet spot.


3. The file distribution mechanism's flexible package sources combine with a very simple protocol for package catalogs (Take a URL, add/pkg/, add a string, get a read-able hash table) to look up packages you don't yet have. As a service, we run a few catalogs (one for each release, plus pkgs.r-l.o). But we expect that users with special needs (such as sensitive installations that need exactly certain tested and trusted versions, especially with proprietary software) will build their own catalogs on private Web sites.


Clearly, however, job 4 is where Planet and the package system differ the most.


With the package system, we follow the precedent of operating systems. An OS package's job is to get files into the right spot. An OS package contains a binary and instructions to install it as /usr/bin/zsh. It is not typical in OSes to be able to install multiple packages (such as different "versions" of the "same" package) that both provide /usr/bin/zsh. When you're at a Unix prompt, you don't have to write zsh-5.0.5/usr/bin/zsh. It's possible that many consider this is a big problem with OSes and indeed we do observe that it is fairly common to provide packages that provide binaries and libraries with embedded names such as how on my machine I have python2.6, python2.7, and python3.2 all in my $PATH. It is important to realize, however, that the deb format and the apt tool didn't need to change to support this change or future changes in perspective in how to compose code.


I hope this analogy helps understand the Racket package system. In the package system, a package doesn't install "binaries", "man pages", and "init scripts", but installs similar things, such as "module paths", "documentation manuals", and "raco commands". Each of these has a notion of conflict: can't have two zshs or two racket/lists; can't have two zsh.1 pages or two docs named doc; can't have two modules trying to provide raco neo-tokyo-is-about-to-explode. If you find a random .deb on the Internet, can you predict what binaries it will contain from its name? No. The same goes for Racket packages. However, if you are egregiously weird, then people probably won't want to install your packages, just like for random debs.


However, clearly rules are helpful. In the world of operating systems, you know that basically all packages distributed by Debian can be installed at the same time, except for "virtual packages" that do stuff like selecting whether postfix or sendmail should be responsible for the sendmail command. These rules are not enforced through technology, though. Instead, the Debian maintainers have a social process that enforces them, with information being provided by technology (such as regression systems that identify unintended conflicts.) The catalog server that the Racket team provides helps facilitate a similar process with the concentric rings (all ring <=1 packages can be installed at once and ring 1< packages can do anything.)


Non-conflicting sets of packages is the simplest rule to define and enforce. Other rules about backwards compatibility are much more complicated to define and enforce. I do not believe there is much precedent in the world of OSes, although we can see a little bit of what they do through things like libgtk, libgtk2, and libgtk3, where generally code written for one libgtk2 package is compatible with all libgtk2 packages made in the future, but libgtk3 is effectively a totally different package and introduces totally separate binaries like gtk3-config.


The most that the Racket team attempts to do here is to say, "Here are the rules we will follow and we think you should follow them too." Specifically, that we will maintain backwards compatibility or make a new package. We can't and won't enforce this, nor do we always live up to it with our own work (but we feel really bad about it when we do.)


Although my main goal of this section has been to explain my solution to (4), a great thing about the package system is that it is not binding at all. You can decide to follow the same rules as Planet. It is easy to do so:


  • Always name your packages $AUTHOR-$PACKAGE-$MAJOR
  • Always provide modules from only the collection, $AUTHOR-$PACKAGE-$MAJOR
  • Maintain backwards compatibility within releases of $AUTHOR-$PACKAGE-$MAJOR
  • Update the 'version metadata in the package info.rkt to reflect the $MINOR version.

And, boom!, you've recreated the rules of Planet to a T except for two things: (a) you'll still need to put a dependency on $AUTHOR-$PACKAGE-$MAJOR on the outside of code in a package info.rkt file rather than just inside files and (b) you can't use $AUTHOR-$PACKAGE to refer to "whatever the current $MAJOR" is.


The first compromise of adding something to the info.rkt is fairly modest, as it requires O(1) line modifications.


The second compromise is more severe, although actually you could just maintain such a package and deal with the breakage that occurs when you try to upgrade. Such breakage, however, was present in Planet too, as when a package was installed based on $AUTHOR-$PACKAGE only the local machine would cache the version used, so if you took the requiring module to another machine, it would download a new version and, potentially, have a backwards incompatibility problem. Using the package system in the most naive way (i.e. installing the $AUTHOR-$PACKAGE at some point and programming to that) would work exactly the same as Planet, except that the package system was designed to be able to port installations from one machine to another with raco pkg migrate.


I hope this blog post has helped explain the package system and shown that it does not prevent you from doing anything that Planet let you do, it only allows you to do more.

2014-11-23

800!

800
(Racket Tasks On Rosetta Code)

Since (and even before) Asumu Takikawa’s post “200!” at the beginning of March 2013, folk have been beavering away, implementing tasks on Rosetta Code. And on November 15th 2014:

800 tasks have now been Implemented1 in Racket on the Rosetta Code website!

Before I go any further it must be said that, without a doubt… this is awesome! This achievement represents a lot of work, and a lot of code. And everyone who has participated should be thanked and congratulated for getting this far.

So thank you. And well done!