I suggest we ...

add a memory-mapped array type

Add a memory mapped array similar to what is available in the Julia and Python languages. This would allow to process arrays backed by a binary file that do not fit in RAM (on a single host) transparently as if they did. This would be very useful for scientific programming / machine learning on big data on a single host. The syntax/interface of the array should be the same as a standard array/ distributed array (see my proposal concerning distributed arrays) so that one can transparently switch the implementation from standard array -> mmapped array -> distributed array to scale up from small data/single host -> large data/single host -> large data/multiple hosts to transparently scale in data size & computing power size.

5 votes
Vote
Sign in
(thinking…)
Sign in with: facebook google
Signed in as (Sign out)
You have left! (?) (thinking…)
Steven Sagaert shared this idea  ·   ·  Flag idea as inappropriate…  ·  Admin →

Support for memory mapped files already exists in .NET 4.5 and can be used already.

It’s possible that nicer support can be added for an F#-defined type too. However that should be done in an upstack library like MathNet.Numerics or FSharp.Data or Deedle or elsewhere, rather then in the language and FSharp.Core library.

If you’re interested in contributing some functionality here, http://fslab.org may be a place to do so?

Don Syme, F# Lang

2 comments

Sign in
(thinking…)
Sign in with: facebook google
Signed in as (Sign out)
Submitting...

F# Language

Feedback and Knowledge Base