Teacher's Guide: Callbacks

Overview

This lesson draws students' attention to the fact that, in functional programming languages like JavaScript, functions are a type of object and therefore can be manipulated much like any other data value. This has several implications for coding with functions, one of which is that it enables functions to be passed as arguments to other functions. A callback function is a function that gets passed to a second function, for that second, so-called higher-order function to execute it at some later point. Callbacks are prevalent in JavaScript libraries and are key to the use of many of the more advanced features available in Pencil Code. Callbacks are introduced in this lesson; various applications of callbacks will be explored in subsequent lessons.

More about the lesson

Functions differ from all other JavaScript data types in that they are callable—that is, we can instruct the program to execute the statements contained in a function. Otherwise, functions are treated essentially just like any other data type. As a result, in JavaScript, functions have the following features:

1. a function can be assigned as a value to a variable or stored in a data structure, i.e., in an array or object
2. a function can be passed as an argument to other functions
3. a function can be the value returned by another function

The fact that functions can be passed around programs like any other data type has profound implications for how JavaScript can be used to accomplish different tasks. Not all programming languages offer this feature; those that do are said to support first class functions.

In this lesson, students will make use of all three of these first-class-function features, though the lesson's emphasis is on passing functions as arguments to other functions. The function that gets passed to other functions has a special name, a callback. The function to which it the callback is passed is referred to as a higher-order function.

The use of callbacks is fairly straightforward. However, there is one common stumbling block. One of the most important points to emphasize to students is that when the callback is passed to the higher-order function, it should not be called. In practical terms, this means that the function name cannot be followed by parentheses. For example, in lesson's coding snippet, the higher-order spiro function is invoked with this statement:

spiro(drawSquare, 10)

rather than with the following, incorrect code:

spiro(drawSquare(), 10) # Function crashes with a TypeError

Given the drawSquare function as defined above, this may not seem like a particularly troublesome or confusing issue, and in any event it is a problem that is easy to remedy (i.e., just remove the offending parentheses). A more relevant example is an effort to pass, as a callback, a function that is defined to be called with one or more arguments. As an example, consider what happens when the following alternative definition of drawSquare is passed to the spiro function:

drawSquare = (sideLength)-> 
  for [1..4]
    fd sideLength
    rt 90  

spiro(drawSquare(100), 10) # Function crashes with a TypeError

This call to spiro fails because drawSquare(100) is executed before the execution of spiro even begins. The value passed to the higher-order function is therefore not the callback itself, but instead the return value of the executed callback (which in this case is an array of references to the turtle sprite). Thus, the result is that the program draws a single square, and then crashes. The proximate cause of the failure is a TypeError exception, which JavaScript throws when spiro attempts to execute the non-function value returned by drawSquare.

The upshot is that callbacks cannot be passed with arguments, at least not directly. Although this limits the functions that can be used as callbacks, there are workarounds. See the Beyond the Lesson notes, below, for more details

The lesson's coding snippet provides a simple example not only of coding and using a callback, but also of creating the function that expects the callback as an argument, i.e., the higher-order function. Higher-order functions contain the logic for when the callback function gets executed. The lesson simultaneously explores the higher-order function and the associated callback(s) because doing so provides a more complete view of callback use.

Pedagogical goals aside, however, student use of callback functions will most commonly involve defining callbacks to use with preexisting higher-order functions, a point that certainly should be shared with students. Admittedly, in this lesson students will only get a small taste of defining callbacks for use with built-in higher-order functions, such as when they define callbacks to pass to the Array class's sort function. The additional activities provide some additional examples, but these are limited.

The need to define and use callbacks will become much more prevalent in subequent lessons. Students will soon see that callbacks are required for loading and running external data and scripts, setting up event handlers (required for making programs more interactive), and to make use of the forever function, which opens up the possibility of writing programs utilizing frame-based animation rather than the queue-based animation which students have heretofore employed. The bottom line is that callbacks are central to coding in JavaScript. The only reason students haven't encountered them previously was because this curriculum has been carefully structured to avoid using them until students had built up sufficient experience with other language features to use them effectively.

Notes to activities

The Starburst activity is designed to help students develop an understanding of the links between higher-order functions and their associated callbacks. Admittedly, the activity is not entirely "authentic" (i.e., one doesn't really need callbacks to get the job done), but its simplicity helps provide transparency and insight into the inner workings of callbacks.

The MathGraphs activity provides a more challenging and arguably more authentic example, one that tends to be popular with students as well. The greatest challenges involved in this activity have little to do with callbacks, but rather with the need to translate the input and output values of the underlying function to a set of coordinates that look good on screen. These computational challenges have been explored previously in the additional exercises of both the Notes to the Custom Functions! lesson and Notes to the Return! lesson.

The ArraySort activity illustrates the role callbacks can play in working with built-in functions. In this case, it is much easier to use the built-in sort function than it is to derive and code an efficient and reliable sorting algorithm of our own!

Additional activities

• ArrayExplorations: The Array class includes a number of additional built-in functions that require the use of callbacks. These include filter, map, and reduce. Like sort, these functions process all the data in the original array. However, an important difference is that these functions leave the original array unchanged. They either return a new array (in the case of filter and map) or a value (reduce). The sort function, in contrast, mutates (i.e., changes) the original array.

  filter returns a new array containing the values of the original array that meet certain criteria. The function is written to be very flexible: the end user provides the filtering criteria in the form of a callback function that accepts a single value as an argument and returns a boolean value (i.e., true or false) indicating whether that value meets the criteria. The following example illustrates defining a callback that provides a rule directing filter to return values of the original array that are even:

  

  map is used to create a new array of equal length to the original, but which can have modified values based on a function applied to each element of the original array. The following example illustrates making use of a callback that squares each value in an array.

  

  reduce also processes an array, but it does calculations based on each successive pair of elements, and it returns a single value. For example, the following example illustrates a slick way to compute the sum of all the elements of an array:

  

  As its name is meant to suggest, the goal of this activity is for you to explore the possibilities of using array functions to manipulate arrays in interesting ways. Come up with your own examples, with at least one for each of the three functions described above.

• DeepCopy: Create an array of randomly colored Turtles, and then attempt to copy that array using the Array methods filter, map, and reduce, as described in the previous activity. Then make the sprites in each array move in opposite directions. (Do that now, before reading the rest of these instructions. Come back when you get frustrated!)

  Chances are, your attempt at this seemingly straightforward task didn't work right. But don't feel bad—copying an array can be a surprisingly difficult challenge. In this case, you likely created a new collection, but the elements in the collection are the same turtles as in the original array, rather than copies of the original turtles.

  The source of the difficulty here relates to how values are stored and referenced in JavaScript (a topic discussed in the Notes to the Arrays! lesson). In short, when copying an array, we need to be sure to copy of each elements in the original array, rather than the references to each of those elements.

  Copying arrays is trivial when the array contains primitive elements, such as numbers or strings, because assignments made using references to each element will result in actual copies of each original, primitive value. As in the previous activity, the new array will be independent of the old, containing its own values.

  However, the copying process is more complicated when the elements in the original array are themselves objects. Assignments using references to an object (including arrays, a specialized form of object) will result in a copy of the reference, rather than of the underlying object. To successfully copy objects, we have to create a new object and individually copy the values of each of its properties. Moreover, if properties of the object are themselves objects, then we have to repeat this process, recursively, until we reach the level of primitive data types. This process is known as a deep copy. In contrast, simply copying each element in an object (and likewise, of an array), by making assignments based on references to the elements in that array, results in what is termed a shallow copy.

  Fortunately, we can create a copy of an array of turtle objects using a shallow copy because the Turtle class includes a copy method, which does the difficult work required of a deep copy for us. Use this fact to write a program that uses the map method to create (effectively) a deep copy of the original array of turtles.

• PlannedCSS: In earlier lessons involving labels, we ran into the problem that calls to the html and css methods were not included in the animation queue. At that point, we had no alternative other than to rely on calls to await done defer() to get the timing of these function calls right. This activity explores a much more elegant solution. The plan function allows you to insert code to be run at a specific point in the animation. The code to be inserted in the queue is passed to plan as a callback.

  Code a program that creates a label using an id so that you can subsequently reference that label using the jQuery id selector (this was the topic of the Label Recycling! lesson). Make changes to the format of your label every few seconds (e.g., either after calls to pause or after other animations). Ensure that the changes take place at the appropriate point of the animation by embedding calls to html and css in callback functions, which you should pass to plan at the appropriate point of your program. For example, if you have a label referenced by the variable name words, then you might define your callback as:

  changeToRed = () ->
    words.css({backgroundColor:red})

  And subsequently execute this code using plan(changeToRed).

• ProgramTimer: The last activity in the Date Objects! lesson instructed you to create a program timer. With your newfound knowledge of functions and callbacks, you are now in a position to code a much more sophisticated and elegant solution.
  

  For this activity, code a timer function that can measure how long it takes to execute a block of code. This block of code should be referenced via a function, such as drawStarburst, and should be passed to timer as a callback.

  As in the Date Objects! activity, you should measure time elapsed by computing the difference between two calls to Date.now(). Recall that Date objects represent time in milliseconds since Jan. 1, 1970. The now function of the Date class provides this value directly, without generating a new Date object. As in the original Timer activity in the Date Objects! lesson, the main challenge in this program is that calls to Date.now, like calls to getxy or direction, are not placed in the animation queue. Previously, we solved that problem using calls to await done defer(). This time around, however, you should rely instead on the plan function. As described in the previous activity, plan allows you to ensure that functions get executed at the appropriate point of the animation queue.

• PenPlus: The Notes to the Return! lesson included an additional activity that described how to access the css hook turtlePenDown and turtlePenDown properties. Extend that work, by creating two pen functions, togglePen and quietPen. togglePen should provide an alternative to pu and pd, similar to how the built-in toggle function provides an alternative to hide and show. However, by modifying css properties using the css function, rather than making calls to pu or pd you can avoid the Pencil Code pen animation. Similarly, define a quietPen program, that should works like pen, but likewise without the (sometimes annoying, and always time-consuming) animation. Both functions should take a turtle as an argument; quietPen should also take a pen color and width.

  When using css hooks, it can be challenging to get the timing right in the animation queue. Make this function always operate at the right point of the animation queue, by making a "plan sandwich": the code that calls the css function should be embedded in a function that is passed as a callback to plan. That call to plan should be embedded in the togglePen or quietPen functions.

Beyond the lesson

Callbacks and arguments (and closures)

When defining custom functions, we regularly use arguments to pass information to the body of that function. It is therefore natural to want to do this with callbacks as well. However, as noted above, we cannot pass the argument to the callback directly, at the same point in the code as when the higher-order function is called. Doing so would cause the function to be executed immediately, rather than invoked later on by the higher-order function as a callback. This presents a frustrating and, for those new to callbacks, confusing limitation, but as with all coding challenges, there are workarounds.

The simplest workaround, conceptually at least, is to define the higher-order function so that it takes additional arguments, and pass that information to the callback when calling it. The second, arguably superior approach is to employ a coding idiom—a useful recurring construct in one or more programming languages—called a closure.

For context, consider the following coding snippet, which contains a higher-order function (spiro) and three functions intended to be used as callbacks. spiro takes two arguments, a callback which provides instructions for a shape and a number which specifies the number of times to repeat the shape. The functions drawSquare and drawCircle are both suitable for use as callbacks with spiro. However, drawPolygon is incompatible. Given how spiro is written, there is no way to submit the sides argument and get the desired result.

When the user of a higher-order function is also its author, a straightforward workaround is redefine the higher-order function's definition to accommodate one or more extra arguments. For example, we might modify spiro as defined above to accept a third argument, sides. The value of this argument can then be passed to drawPolygon function when it is subsequently executed. This strategy is executed below. The call to spiro creates a spirograph consisting of 10 pentagons by executing successive calls of drawPolygon(5).

spiro = (drawShapeCallback, n, sides) ->
  for [1..n]
    drawShapeCallback(sides)
  
spiro(drawPolygon,10,5) 

Though this modification accommodates the drawPolygon function, it is not an ideal solution. One issue is that it is not a practical solution if we are facing a wide range of possible shapes. For example, a bullseye shape might depend on two arguments rather than one (for example, radius and number of rings); some other shape might depend on an array of coordinates. True, there is a way to specify spiro that would accommodate all of these, but it also means that every time we want to add a new shape, we would potentially have to update the higher-level function to accept the associated callback. That is, it might not be feasible to create a single higher-order function that could accommodate the full range of possible callbacks and their related argument needs. This suggests that we might be better off pursuing a different approach.

Moreover, the foregoing strategy is only an option if the author of the callback is also the author of the higher-order function. This typically is not the case. The higher-level functions that coders rely on most tend to be built-in JavaScript functions (such as methods of the Array class, the focus of this activity) or other third-party tools accessed from an external library, such as jQuery. In such situations, making changes to the higher-order function to suit our individual needs is not an option.

A more general solution to the callbacks-argument challenge is write a function to generate the desired callback function with the desired customizations imbedded in it. The following snippet provides an example of this approach. The call to makePolyCallback(drawPolygon,5) returns a callback function that contains instructions for drawing a pentagon; it is this callback that gets passed to the higher-level function, spiro, to generate the spirograph.

makePolyCallback is an example of a closure, a useful coding idiom that is widely used in programming languages that support first-class functions, such as JavaScript. Admittedly, the closure example provided is rather forced—we could have simply coded a drawPentagon function and saved ourselves a lot of complexity and trouble. More relevant examples are provided in the Closures! lesson.

Use of arguments with callbacks

The most common source of errors when using callbacks is executing the callback when it is being passed to the higher-order function. This mistake is particularly likely to be made when attempting to use a function that accepts arguments as a callback, as illustrated in the following example:

spiro=(drawShapeCallback, n)->
  for [1..n]
    drawShapeCallback()  
    rt 360/n

drawPolygon = (sides) ->
  for [1..sides]
    fd 100
    rt 360/sides

spiro(drawPolygon(5), 10) # Function crashes with TypeError 

This pitfall, as well as remedies, are discussed in depth in the Beyond the Lesson section of these notes.

Callbacks and Variable Scoping issues

As explained in the notes to the Custom Functions lesson, the rules of variable scope make it possible to access variables from the containing scope from within a function. Those notes recommend discouraging this practice in most cases (enouraging instead that data be explicitly passed to the function as arguments), it is common to see such variable usage in practice.

Given the flexibility of JavaScript's variable scoping rules, students may opt to directly reference to a function in the enclosing scope from inside a higher-order function, rather than pass the reference to that function to that higher-order function as a callback. This is not an error, per se, but it does defeat the purpose of the activities in this section.

As an example, consider a simple callback and higher-order function combo that might correctly be coded as follows:

However, a student might code a working solution that looks like this:

Note that this ability use custom functions to circumvent the use of callbacks only arises when the coder is responsible for defining both the higher-order function and the callback, such as for the lesson's Starbursts and MathGraphs activities. This approach is not possible when making use of built-in higher-order functions, such as when using the array class's sort method in the lesson's ArraySort activity.

Technicalities

Asynchronous vs. Synchronous callbacks

A callback is any function passed to another function for the purpose of being invoked at some later point. In common parlance, however, the term callback is used more narrowly, to describe callbacks used with asynchronous (rather than synchronous) processes. The purpose of this section is to explore this distinction, as it is conceptually important for subsequent lessons.

Consider a script that needs to import data from an external file to complete a particular task. The script needs to wait for the data to load before processing it. However, loading data can take some time, especially if the source file is large and/or stored on a remote device.

Conceptually, we have two ways to proceed: (1) wait until the loading process completes, blocking subsequent lines of code from being processed, or (2) allow subsequent lines of code to continue processing, and alert the system when the load process is complete.

The former approach will cause the program to hang, meaning that it will appear to freeze, not allowing any continued external end-user input. Additionally, the program can't continue to work on other tasks unrelated to the data being loaded. This is an inefficient use of system resources and provides a suboptimal user experience. The latter, non-blocking approach passes the task (here, loading data) from JavaScript to the browser, freeing up JavaScript to continue processing other items. When the browser is finished the task, it alerts JavaScript that the file load has successfully completed. This latter approach is generally preferable, though it necessitates some more advanced coding techniques.

These two options describe tackling the task of data input synchronously or asynchronously. Synchronous processes execute sequentially; each task must be finished before the next begins. Thus, a synchronous operation blocks other program activity until the operation completes. This approach works well for tasks that are carried out quickly, but becomes problematic for tasks that take longer to complete. Asynchronous processes, in contrast, initiate a task (such as loading data), but then allow program control to pass to subsequent lines in the script or other system tasks while the data is loading. Asyncronous operations are therefore non-blocking. Synchronous processes do not depend (directly) on each other's outcome, and can therefore occur seemingly simultaneously.

The Pencil Code load function faciliates the task of loading data from an external file. It has the following usage, which makes use of a callback. The callback specifies the code to be processed once the system has compelted reading in the data.

load(url, cb) Loads data from the url and passes it to cb.

The call to load initiates the data import process, but then passes the task to the Web API. Program control then passes to subsequent lines of the script. When the load process completes, the Web API alerts JavaScript, which then executes of the callback.

This script provides an example of the load function, and illustrates it's asynchronous execution. The basic structure of the script is:

    fileURL = "https://some_path/some_filename.txt"

    callback = (data)->
        words = data.split \n"
        wordleWords = []
        
        # ...more lines of code to populate wordleWords...

        cs()
        label "Wordle start word for today: " + random(wordleWords)

    load fileURL, callback

    label "Working on your start word for today..."

Execution of load initiates the data import process, which it then hands off to the web browser. Program execution moves on to the next statement, producing the following output:

When the load process is complete, the Web API alerts JavaScript, which then executes the callback, which processes the loaded data, clears the screen, and outputs the following to the screen:

In summary, the term "callback" is frequently used to refer specifically in situations in which the callback function will be called in the future but at an unknown time, upon completion of an asynchronous process or task. Students will encounter many examples of asynchronous callbacks in subsequent lessons, beginning with functions that facilitate writing code to respond to system events, such as mouse clicks or keystrokes.

However, the broader definition of callbacks is in nonetheless valid, i.e., a function passed to a higher-level function for execution at a specific later point, rather than at some time contingent on the completion of an asynchronous task, as also a callback. All the examples of callbacks used in this lesson and these notes, including the built-in filter and map methods of the Array object, are synchronous callbacks.

plan

plan is one of two higher-order Pencil Code functions that can be used to manipulate the animation queue (the other, done, is discussed below). plan offers an alternative (and less drastic) means to resolve to animation queue difficulties that, until now, students tackled through calls the statement await done defer(). Code passed to plan (as a callback) is inserted into the animation queue based on where the call to plan appears in your script. Thus, plan provides a means by which we can insert, into the animation queue, functions that otherwise would be executed immediately, such as distance, getxy, remove and css.

The use of plan is explored in this lesson's Additional Activities PenPlus and PlannedCSS. Note that plan is sprite-specific. Insert code into the animation queue for a specific sprite using dot notation.

done and await done defer()

done is another higher-order function that can be used to manipulate the animation queue. done works similarly to plan (discussed above), except that code passed (as a callback) to done is placed at the end of the animation queue. For example, in the following snippet, the callback passed to done is executed after all other animations have completed, i.e., the text appears last:

whatToDoWhenDone = ()->
  label "(I'm done!)", 25
  
done whatToDoWhenDone 

for i in [1..360]
  dot hsl(i, 1,0.5), 365-i

In the Pencil Code environment, we typically make use of done as part of the statement await done defer(). The workings of this powerful statement was initially described in the Notes to Array Destructuring lesson. Writing the statement with explicit use of parentheses,

await(done(defer()))

When program execution reaches the statement await done defer(), the following occurs:

1. defer is invoked. The call defer() does two things. It creates a deferral, a flag stored as a global variable that will be referenced by the await function when it is invoked. The deferral is initially set to communicate to await that it should not allow subsequent lines of the script to be read. defer() also creates and returns a callback, which when executed will update the deferral, communicating to await that it can allow normal program execution to proceed. However, this callback is not run immediately, but rather passed to done.
2. done is invoked. done places the callback created and returned by defer() at the end of the animation queue.
3. await is invoked. The await function blocks all subsequent lines of code in the program from being read or executed. Importantly, it does this without causing the browser to freeze up or "hang". The script therefore continues to process animations that had been scheduled prior to the call to await done defer().
4. The last code in the animation queue is the callback created by defer(). The callback is now invoked. It updates the deferral, communicating to await that it can stop blocking.
5. Normal program flow of subsequent lines of code resumes.

Iced CoffeeScript

The done function is a Pencil Code feature, typically used as part of the statement await done defer(). The functions await and defer, however, are features of a CoffeeScript superset, Iced CoffeeScript, which is used in Pencil Code source code (i.e., as part of jQuery-turtle.js).

done was written to be used in conjunction with await and defer. However, await and defer have broader use. For example, they can be used with the Pencil Code read, readnum, and listen functions, as illustrated in coding snippets provided in the block-coding Palette under Text and Sound. A brief exploration of read, used both asynchronously (with await) and synchronously (directly passing the callback to it) is provided here. More generally, await and defer can be used with any asynchronous process, as described in the Iced CoffeeScript documentation and on this this gitghub preview doc.