Merge branch 'development'

V 0.3

NovelIO.sln

@@ -25,6 +25,7 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "tools", "tools", "{83F16175
 EndProject
 Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "content", "content", "{8E6D5255-776D-4B61-85F9-73C37AA1FB9A}"
 	ProjectSection(SolutionItems) = preProject
+		docs\content\channels.fsx = docs\content\channels.fsx
 		docs\content\files.fsx = docs\content\files.fsx
 		docs\content\index.fsx = docs\content\index.fsx
 		docs\content\motivation.fsx = docs\content\motivation.fsx

RELEASE_NOTES.md

@@ -1,3 +1,10 @@
+#### 0.3.0-alpha - 12/07/2016
+Added support for recursive PUs
+Renamed Handles to Channels - part of wider API improvements
+Added modules containing operations on each Channel type
+Made a seperate operators module so that it can be opened independently
+Misc performance improvements
+
 #### 0.2.0-alpha - 12/06/2016
 Added char PUs for fixed length string encodings
 Redesigned the naming structure of big endian / little endian PUs to make them easier to work with

docs/content/channels.fsx

@@ -0,0 +1,54 @@
+(*** hide ***)
+// This block of code is omitted in the generated HTML documentation. Use 
+// it to define helpers that you do not want to show in the documentation.
+#I "../../bin"
+#r "NovelIO/NovelIO.dll"
+open NovelFS.NovelIO
+
+(**
+Channels
+======================
+
+Channels represent a mechanism of one or two way communication with some kind of resource, such as a file or remote server (e.g. via TCP).
+
+Channels come in two flavours, a `TChannel` which supports text data and a `BChannel` which supports binary data, functions are provided in the `TextChannel` and `BinaryChannel` modules respectively.
+
+## Controlling lifetime
+
+The typical method of controlling method of explicitly controlling lifetime in .NET is to use `IDispoable`, however this approach fundamentally relies on side-effects.  NovelIO therefore uses a different approach: the bracket pattern.
+
+`bracket` is a function supplied in the `IO` module: `val bracket : IO<'a> -> ('a -> IO<'b>) -> ('a -> IO<'c>) -> IO<'c>`.
+
+The first argument is an action of type `IO<'a>` which creates a resource.
+
+The second argument is a function which takes the resource created by the first action and cleans it up.
+
+The third argument is a function which takes the resource created by the first action and returns a new action, this new action is then returned by the `bracket` function.
+
+Put succinctly, there is a way of creating a resource, a way of cleaning it up and a function to happen in between.
+
+### Using the bracket pattern
+
+In general, you don't need to worry about using 'bracket' explictly.  Functions are created for different resources to avoid you having to fill out all of `bracket`'s arguments manually.
+
+An example is the `File.withTextChannel` function:
+
+*)
+
+io {
+    let withChannelOR = File.withTextChannel FileMode.Open FileAccess.Read 
+    return withChannelOR (File.assumeValidFilename "test.txt") (fun channel -> 
+            io {
+                let! l1 = TextChannel.getLine channel
+                let! l2 = TextChannel.getLine channel
+                return l1, l2
+            })
+}
+
+(**
+
+By using the `withTextChannel` function, we can supply an argument of the form `TChannel -> IO<'a>` which determines what to do with the channel during its entire lifetime.  This is equivalent to just the final argument of `bracket` where the two preceeding arguments are filled in for us.
+
+`with_` functions are provided throughout the library for other types of channels and resources but they follow the same pattern as described here.
+
+*)

docs/content/files.fsx

@@ -39,4 +39,26 @@ The `File` modules contains functions very similar to `System.IO.File` defined i
 io {
    let! lines = File.readLines fName
    return lines
-}
+}
+
+(**
+## File Channels
+
+If you need more fine-grained control over File IO, the way to achieve this is using Channels.  Text and Binary Channels (`TChannels` and `BChannels`) support explicit reading and writing of their corresponding datatype.
+
+*)
+
+let readLines file = 
+    io {
+        let withChannelOR = File.withTextChannel FileMode.Open FileAccess.Read 
+        return! withChannelOR file (fun channel -> 
+                IO.Loops.untilM (TextChannel.isEOS channel) (TextChannel.getLine channel))
+    }
+
+(**
+
+It is recommended that you use that the `withChannel` functions provided so that the channel will be automatically cleaned up after its use rather than explicitly opening and closing channels manually.
+
+You can find more about channels on the [channels page](channels.html). 
+
+*)

docs/content/index.fsx

@@ -86,8 +86,8 @@ In order to execute items in parallel, we can simply build a list of the IO acti
 
 io {
     let fName = File.assumeValidFilename "file.txt"
-    let! handle = File.openFileHandle FileMode.Open FileAccess.Read fName
-    return IO.Parallel.par [Console.readLine; IO.hGetLine handle]
+    let! channel = File.openTextChannel FileMode.Open FileAccess.Read fName
+    return IO.Parallel.par [Console.readLine; TextChannel.getLine channel]
 } |> IO.run
 
 (**

docs/content/motivation.fsx

@@ -48,51 +48,6 @@ Once again, we can freely replace `yPure` and `Random.nextIO` wherever they appe
 
 As mentioned in the introduction, `IO.run` is the only non-referentially transparent function exposed by this library and, as such, should be used sparingly!
 
-## Lazy evaluation and exceptions
-
-This example is more or less taken from Erik Meijer's Curse of the excluded middle (https://queue.acm.org/detail.cfm?ref=rss&id=2611829)
-
-Consider the following code where we try to combine lazy evaluation with File IO:
-
-*)
-
-let floatLines = 
-    try
-        System.IO.File.ReadLines("testfile.txt")
-        |> Seq.map (float) // parse each line as a float
-    with
-        | _ -> Seq.empty
-
-Seq.iter (printfn "%f") floatLines // print each float to the console
-
-(**
-
-This code appears to be relatively safe - we have a comforting `try`/`with` block around a function that may fail at runtime.  This code, however, does not function in the way it immediately appears to.
-
-In reality, the map is not actually evaluated until we enumerate the sequence with `Seq.iter`, this means that any exception, if triggered, will actually be thrown outside the `try`/`with` block causing the program to crash.
-
-Consider an alternative using NovelIO's expression of IO:
-
-*)
-
-let fName = File.assumeValidFilename "testfile.txt"
-
-let fileIO = io {
-    let! lines = File.readLines fName // sequence of io actions which each read a line from a file
-    let! floatLines = IO.mapM (IO.map float) lines // parse each line, collecting the results
-    do! IO.iterM (IO.putStrLn << sprintf "%f") floatLines // print each float to the console
-}
-
-try
-    IO.run fileIO // side effects occur *only* on this line
-with 
-    |_ -> () // error case
-
-
-(**
-
-This code describes exactly the same problem but we know that side-effects can occur in exactly one place `IO.run`.  That means that success or failure need be handled in only that one location.  We can therefore design complicated programs where IO is described using pure, referentially transparent functions and potentially error-prone behaviour is made very explicit and side-effects are restricted to very specific and obvious locations.
-
 ## Side-effects and lazy evaluation
 
 In general, writing code that combines side effects and lazy evaluation can be complex and error prone, the developer can often be left with little idea when effects will actually be triggered.
@@ -118,7 +73,7 @@ At first glance, this program might appear to record key strokes until the user
 
 In reality, this program counts key strokes until the user presses 'Enter' and prints this length, then it records key strokes again until the user presses 'Enter' and prints the result.
 
-If we express this program using this library, the side effects are clearly apparent:
+If we express this program using this library, the effects are clearly apparent:
 
 *)
 
@@ -166,45 +121,108 @@ Consider this code:
 
 *)
 
-let randomSeq = Seq.init 20 (fun _ -> rnd.Next())
+let randomSeq = Seq.init 4 (fun _ -> rnd.Next())
 let sortedSeq = Seq.sort randomSeq
 
 printfn "Sorted: %A" sortedSeq
 printfn "Random: %A" randomSeq
 
 (**
 
-Indeed, the numbers shown in the 'Sorted' and 'Random' lists could be totally different.  Each time we enumerate **randomSeq**, the side effect of getting the next random number is produced again!
+Let's at the results of an example run of this program:
 
-Here is the same program written using NovelIO.  Notice that we have to explicitly ask for a second sequence.
+> Sorted: seq [42595606; 980900814; 1328311795; 1497661916]
+> Random: seq [308839936; 1514073672; 36105878; 741971034]
+
+While this program appears to generate one sequence, sort it, then print the sorted and unsorted result - that isn't what it actually does.  What it actually does is effectively define two random sequence generators, one of which is sorted and the other is not.
+
+Each time we enumerate `randomSeq` or `sortedSeq`, the original side effect of getting random numbers is produced again and again!
+
+Here is the original program we desired to write using NovelIO.
 
 *)
 
 io {
-    let randomSeqIO = IO.replicateM (Random.nextIO) 20
+    let randomSeqIO = IO.replicateM (Random.nextIO) 4
     let! randomSeq = randomSeqIO
-    let! randomSeq2 = randomSeqIO
-    let sortedSeq = Seq.sort randomSeq2
+    let sortedSeq = Seq.sort randomSeq
     do! IO.putStrLn <| sprintf "Sorted: %A" sortedSeq
     do! IO.putStrLn <| sprintf "Random: %A" randomSeq
     } |> IO.run
 
 (**
 
-If we do not ask for the second sequence, we get what was the original desired behaviour of the program:
+> Sorted: seq [75121301; 124930198; 609009994; 824551074]
+> Random: [824551074; 609009994; 75121301; 124930198]
+
+Notice that now, both sequences contain the same values.  The generation of actual random numbers is triggered by the line `let! randomSeq = randomSeqIO` which makes the effect completely explicit.
+
+In order to get our program to behave like the original one that uses a sequence with side effects, we have to explicitly ask for a second set of evaluated effects.
 
 *)
 
 io {
-    let randomSeqIO = IO.replicateM (Random.nextIO) 20
+    let randomSeqIO = IO.replicateM (Random.nextIO) 4
     let! randomSeq = randomSeqIO
-    let sortedSeq = Seq.sort randomSeq
+    let sortedSeq = Seq.sort randomSeq // sort the first set
+    let! randomSeq2 = randomSeqIO // evaluate the effects of randomSeqIO again
     do! IO.putStrLn <| sprintf "Sorted: %A" sortedSeq
-    do! IO.putStrLn <| sprintf "Random: %A" randomSeq
+    do! IO.putStrLn <| sprintf "Random: %A" randomSeq2
     } |> IO.run
 
 (**
 
-Hopefully this demonstrates how being explicit about when side-effects occur can massively improve the ability of developers to understand and reason about their code.
+> Sorted: seq [79034179; 1625119183; 1651455963; 1775638512]
+> Random: [1801985798; 963004958; 1819358047; 292397249]
+
+So, using this approach we can easily describe either behaviour while still keeping the intent clear and explicit.
+
+## Lazy evaluation and exceptions
+
+This example is more or less taken from Erik Meijer's Curse of the excluded middle (https://queue.acm.org/detail.cfm?ref=rss&id=2611829)
+
+Consider the following code where we try to combine lazy evaluation with File IO:
+
+*)
+
+let floatLines = 
+    try
+        System.IO.File.ReadLines("testfile.txt")
+        |> Seq.map (float) // parse each line as a float
+    with
+        | _ -> Seq.empty
+
+Seq.iter (printfn "%f") floatLines // print each float to the console
+
+(**
+
+This code appears to be relatively safe - we have a comforting `try`/`with` block around a function that may fail at runtime.  This code, however, does not function in the way it immediately appears to.
+
+In reality, the map is not actually evaluated until we enumerate the sequence with `Seq.iter`, this means that any exception, if triggered, will actually be thrown outside the `try`/`with` block causing the program to crash.
+
+Consider an alternative using NovelIO's expression of IO:
+
+*)
+
+let fName = File.assumeValidFilename "testfile.txt"
+
+let fileIO = io {
+    let! lines = File.readLines fName // sequence of io actions which each read a line from a file
+    let! floatLines = IO.mapM (IO.map float) lines // parse each line, collecting the results
+    do! IO.iterM (IO.putStrLn << sprintf "%f") floatLines // print each float to the console
+}
+
+try
+    IO.run fileIO // side effects occur *only* on this line
+with 
+    |_ -> () // error case
+
+
+(**
+
+This code describes exactly the same problem but we know that side-effects can occur in exactly one place `IO.run`.  That means that success or failure need be handled in only that one location.  We can therefore design complicated programs where IO is described using pure, referentially transparent functions and potentially error-prone behaviour is made very explicit and side-effects are restricted to very specific and obvious locations.
+
+Hopefully this demonstrates how being explicit about when effects occur can massively improve the ability of developers to understand and reason about their code.
 
 *)
+

docs/content/oopintro.fsx

@@ -60,7 +60,7 @@ open NovelFS.NovelIO.BinaryPickler
 let readIntFromFile file =
     io {
         let! bytes = File.readAllBytes file
-        return BinaryPickler.unpickle (BinaryPickler.pickleInt32) bytes
+        return BinaryPickler.unpickle (BinaryPickler.intPU) bytes
     }
 
 (**

docs/content/pickler.fsx

@@ -154,21 +154,79 @@ let utf8PU = BinaryPickler.utf8PU
 
 (**
 
+## Encoding Discriminated Unions (the `alt` combinator)
+
+Consider a simple data type:
+
+*)
+
+type Shape =
+    |Circle of float
+    |Rectangle of float * float
+
+let shapePU =
+    // create a pickler for the circle and recangle case, wrap takes a method of constructing and deconstructing each case
+    let circlePU = BinaryPickler.wrap (Circle, function Circle r -> r) BinaryPickler.floatPU
+    let rectanglePU = BinaryPickler.wrap (Rectangle, function Rectangle (w, h) -> w, h) (BinaryPickler.tuple2 BinaryPickler.floatPU BinaryPickler.floatPU)
+    // a tag map : 0 -> circle, 1 -> rectangle defining which PU to use for which tag
+    let altMap = Map.ofList [(0, circlePU); (1, rectanglePU)]
+    // use the alt combinator and the deconstruction of Shape to the tags defined above
+    BinaryPickler.alt (function | Circle _ -> 0 | Rectangle _ -> 1) altMap
+
+(**
+
+The `alt` combinator is the key to this process.  It accepts a function that deconstructs a data type into a simple numeric tag and a `Map` which defines the PU to use internally for each of the cases.
+
+## Encoding Recursive Values
+
+Since F# is an eagerly evaluated language, we cannot define recursive values as they would never resolve.  To avoid this problem, a `RecursivePU` constructor is provided to allow the recursive definition of the PU to be deferred until required.
+
+A good example of a suitable data type is provided in the paper:
+
+*)
+
+type Bookmark =
+    |URL of string
+    |Folder of string * Bookmark list
+
+(**
+
+We can define a PU for this type by using a mutally recusive value and a function in combination with the `RecursivePU` constructor.
+
+*)
+
+
+let rec bookmarkPU = RecursivePU bookmarkPURec
+and private bookmarkPURec() =
+    // define a PU for the URL case, this is just a UTF-8 PU with a way of constructing and deconstructing a Bookmark
+    let urlPU = BinaryPickler.wrap (URL, function URL x -> x) BinaryPickler.utf8PU
+    // a pickler for the folder case is a tuple2 PU with a UTF-8 PU for the name and a list pickler of bookmarkPU's and a way of constructing
+    // and deconstructing the Bookmark
+    let folderPU = BinaryPickler.wrap (Folder, function Folder (st, bms) -> st, bms) (BinaryPickler.tuple2 BinaryPickler.utf8PU (BinaryPickler.list bookmarkPU))
+    // define that tag 0 means urlPU and tag 1 means folderPU
+    let m = Map.ofList [(0, urlPU);(1, folderPU)]
+    // define that URL should mean use tag 0 and Folder should mean use tag 1
+    m |> BinaryPickler.alt (function | URL _ -> 0 | Folder _ -> 1)
+
+(**
+
+This approach permits the pickling/unpickling of potentially very complex data types with very little development work required.
+
 ## Incremental Pickling
 
 In many cases, especially when dealing with large binary files, it could be desirable to not have to convert back and forth between extremely large byte arrays, indeed this approach might not be viable due to available memory.
 
 In this case, we can use incremental pickling to read/write as part of the pickling process.  Unlike the simple conversion process shown above, this action is effectful so is encapsulated within `IO`.
 
-This process is quite simple, instead of using the `pickle` and `unpickle` functions, we use the `pickleIncr` and `unpickleIncr` functions.  These simply take the additional argument of a `BinaryHandle` upon which they will act.
+This process is quite simple, instead of using the `pickle` and `unpickle` functions, we use the `pickleIncr` and `unpickleIncr` functions.  These simply take the additional argument of a `BChannel` upon which they will act.
 
 Example of incremental unpickling:
 
 *)
 
 io {
-    let! handle = File.openBinaryHandle FileMode.Open FileAccess.Read (File.assumeValidFilename "test.txt")
-    return! BinaryPickler.unpickleIncr complexPickler handle
+    let! channel = File.openBinaryChannel FileMode.Open FileAccess.Read (File.assumeValidFilename "test.txt")
+    return! BinaryPickler.unpickleIncr complexPickler channel
 }
 
 (**
@@ -178,7 +236,7 @@ Example of incremental pickling:
 *)
 
 io {
-    let! handle = File.openBinaryHandle FileMode.Create FileAccess.Write (File.assumeValidFilename "test.txt")
+    let! channel = File.openBinaryChannel FileMode.Create FileAccess.Write (File.assumeValidFilename "test.txt")
     let data = [("A", 7.5, 16); ("B", 7.5, 1701)]
-    return! BinaryPickler.pickleIncr complexPickler handle data
+    return! BinaryPickler.pickleIncr complexPickler channel data
 }