Add Support for Fibers (via JS Generators) #2207

This sounds fun, I'll give it a shot.

If you need any assistance, let me know.

I have done some thinking about this, so have a little brain dump about possible pain points

1. Ruby uses Fiber.yield to transfer control, JS uses the yield keyword. This may mean that we'll have to special-case Fiber.yield in the parser, which is less-than-ideal, as I could see somebody doing something weird like calling yield on a variable which may sometimes be the constant Fiber
2. JS generators don't have an equivalent of Fiber.transfer that I can see.
3. Exceptions should work relatively transparently, since Ruby and JS seem to handle them in the same way (bubbling them up to the calling context of the fiber/generator)
4. The way arguments may be passed to fibers seems fairly transparent as well. In ruby you use fiber.resume(arg), and in JS you use fiber.next(arg)
5. It may be possible to implement await within the Promise library via Fibers. If this turns out to be true, it would make using Opal a lot easier, since you could pretend that code is synchronous. Using the library's example, this:

HTTP.get("user_details").then do |response|
  HTTP.get("user_contacts?id=#{response.json[:user_id]}")
end.then do |response|
  HTTP.get("user_details?id=#{response.json[:contact_id]}")
end.then do
  puts "All finished"
end

May be translatable into something like this:

response = HTTP.get("user_details").await
contacts = HTTP.get("user_contacts?id=#{response.json[:user_id]}").await
details = HTTP.get("user_details?id=#{response.json[:contact_id]}").await
puts "All finished"

Which seems much cleaner.
6. Once Fibers are working, I think it would be beneficial to implement Enumerable using them, which is how I believe the Ruby standard library does it. This is a patch I'm fairly certain I actually would be able to contribute.

Ok first pass of experimentation and reading.

Issues:

• The yield keyword must be present within the body of the generator function, this means calling Fiber.yield within another block won't work, at all.
• This means Fiber.new and Fiber.yield cannot be implemented like other things and will require compiler support to convert Fiber.new into a function* declaration with any call to Fiber.yield compiled as a barebone yield.
• This means I'm not entirely sure how useful this would be in practice since looping is done with block calls.

Marking this for discussion for further ideas, implementing this in the compiler shouldn't be too hard, it's just a matter of special casing Fiber.new and Fiber.yield and avoid creating intermediary functions or restricting the semantics.

Main issue is how useful this would actually be since you wouldn't be able to call Fiber.yield within another block.

@AnthonySuper damn, you beat me by few seconds 🐼

The main issue I see is the fact you can't call yield at any other call depth than the root of the generator function, any ideas to work around that?

Well, after screwing around with it for a while, I don't have any easy answers, unfortunately. A common use case of fibers is something like:

fiber = Fiber.new do
  some_array.each do |elem|
    Fiber.yield elem
  end
end

You can sort of fake this by doing something like this:

var Fiber = {
  yieldValues: [],
  yield: function(val) {
    if(val) {
      this.yieldValues.push(val);
    }
  },
  [Symbol.iterator]: function*() {
    yield* this.yieldValues[Symbol.iterator]();
  }
}

some_array = [1, 2, 3];

function *gen(){
  some_array.forEach(function(elem) {
    Fiber.yield(elem);
  });
  yield* Fiber;
}

var f = gen();
var q = f.next();
while(! q.finished) {
  console.log(q);
  q = f.next();
}

However, this ruins the entire point of Fibers, which is user-controlled scheduling, so it's pretty much useless. Implementing this might not actually be possible, which seems pretty unfortunate.

Yeah, it seems to be a dead end for now, I'll leave the issue open in case someone comes up with a good idea.

Almost implemented it locally but then realized that it's not that easy 😄

I've started with the following code, for some reason I thought that it may cover all cases.

I've added two AST rewriters that convert:

• Fiber.new {} to a separate node :fiber_new (required for function *())
• block passed to Fiber.yield to a separate node :fiber_iter (required for yield)

So, opal --sexp -e "Fiber.new { Fiber.yield 123 }" became:

s(:send,
  s(:const, nil, :Fiber), :new,
  s(:fiber_iter,
    s(:args),
    s(:fiber_yield,
      s(:int, 123))))

Both nodes that compile these new nodes to JS are quite simple like this:

module Opal
  module Nodes
    class FiberYieldNode < Base
      handle :fiber_yield
      chlidren :value

      def compile
        push "yield ", expr(value), ";"
      end
    end
  end
end

I've made it working with simple cases like Fiber.new { Fiber.yield 123 }.resume. But then I've tried:

f = Fiber.new do
  i = 1
  loop { Fiber.yield(i); i += 1 }
end

Generated code doesn't work.

• Fiber.new takes a function *
• The body of this function invokes Kernel#loop which takes a function (a block that is passed to loop)
• This function invokes yield inside.
• This is wrong

From what I see this code can't work:

a = function *() {
  (function() { yield 123 })()
}

b = a()
b.next()

Any ideas how to implement Fiber.yield inside a block?

I think that Fiber.yield may not be something we can implement easily inside a block, unfortunately, because Javascript generators are heavily limited and sort of awkward. Like JS in general.

The truth is that JS doesn't support coroutines, and simulate Fiber using either async await or generator will have runtime overhead given that all the functions nested inside needs to be marked as async .

A possible approach might be compiling two versions of methods (generator and normal function) and decide which to run either at runtime or compile time.