Final 1.0 Roadmap

[jdantonio]

It's been suggested (#246) that we're ready for a 1.0 release. People are using this gem in production and the Rails team is considering making concurrent-ruby a dependency of Rails (#251). I believe we've reached a level of maturity, stability, and performance that warrants a 1.0 release. Our community deserves it. So with that in mind I would like to set a final 1.0 roadmap. Because of the major refactoring of the global configuration object (#255) being undertaken--and the number of method calls this will deprecate--I want to have one more major release before 1.0. The goals for these releases will be:

• 0.9 Stabilize the API and fix known bugs
• 1.0 Performance, stability, test cleanup, and documentation
• Post 1.0 Major functionality and API updates

v0.9 (codename Ultron)

Target Date: 18 May 2015

This goal of this release is to stabilize the API. The core update in this release is the refactor of the global configuration object and the renaming of the global thread pools (#255). Other features are merged/in-progress updates (monotonic clock) and bug fixes.

• Monotonic clock updates (Replace Time.now with new monotonic clock #256 -- @jdantonio)
• Gem configuration and global thread pool refactor (Thread Pool Updates #255, Concurrent::RejectedExecutionError when running with minitest/autorun via JRuby #248, Document or improve global-pool's auto termination #227, Remove configuration of global pools #149 -- @jdantonio)
• Depend on ref gem version 2.0 (@lucasallan)
• Read-write lock (Provide a read-write lock #146 -- @jdantonio)
• Bug fixes
  • unlocks outside of ensure block (unlocks outside of ensure block #244 -- @jdantonio)
  • Atomic get_and_set faulty in C (Atomic get_and_set faulty in C #233 -- @jdantonio)
• Actor updates (@pitr-ch)
• Update all dev/test gem dependencies to latest versions
• Promise becomes subclass of IVar (@jdantonio, @pitr-ch)

v1.0.pre1 (codename Fury Road)

Target Date: 29 May 2015

The goal of this pre-release is to clean up everything that is being removed and create the first build of the edge gem. Pre-release and release candidate builds of the edge gem will occur simultaneously with pre-release and release candidate builds of the core gem. Only the version number will be different.

• Remove all deprecated code
• Move Actor and Channel to the edge gem
• Merge thread_safe gem (Merging the thread_safe gem #230 -- @jdantonio)
• Release concurrent-ruby-edge v0.1.0.pre1 (PR Edge Gem #268)

v1.0 (codename Ant-Man)

Target Date: 17 July 2015

This release will have no major API updates (with the possible exception of Actor). It will focus on stability, performance, and documentation.

• Remove all deprecated code (@jdantonio)
• Actor updates (@pitr-ch)
• TimerSet: cancel/reset task (TimerSet: cancel/reset task #76 -- @jdantonio)
• Stress tests (@chrisseaton)
• Test suite improvements
  • Improve test coverage
  • Remove brutal thread killing in tests (Remove brutal thread killing in tests #226 -- @jdantonio)
  • Intermittently Failing Tests (Intermittently Failing Tests #263 -- @jdantonio)
  • Replace sleep based testing (Replace sleep based testing #225 -- @jdantonio)
  • LazyRegister
• Documentation updates
  • Improve inch score
  • Merge markdown files into source files when appropriate
  • Channel (Channel documentation #234)
  • Memory model (Document what "memory model" guarantees each Atomic method provides #232)
  • Global configuration yardoc
  • Actor yardoc
  • Exchanger yardoc
  • LazyRegister yardoc
  • ScheduledTask yardoc (@jdantonio)
  • ThreadLocalVar yardoc

Post 1.0

These items are not a priority right now. We will revisit them once 1.0 has been released. However, should any of these features get implemented before the 1.0 release we will consider merging them, so long as they are stable, well documented/tested, and we are comfortable with the API.

• Common synchronization layer (Migrate form using mutex to synchronization layer #242, Synchronization layer #241)
• Parallel Enumerable (Parallel Enumerable #222)
• Actors un/become or state machine (Actors un/become or state machine #148)
• Remote actors (Remote Actors #147)
• I/O connector for Actors (IO example #144)
• Consolidate IVar childen into one object (Consolidate IVar childen into one object #139)
• Safe final immutable objects Safe final immutable objects #120
• Fail subsequent posts to Agent after failure (Fail subsequent posts to Agent after failure #70)
• Fork/join

Target Dates

The target dates listed above are soft targets. Quality and correctness are tantamount. Everything else is secondary. We will adjust these target dates as necessary to ensure quality and correctness.

[chrisseaton]

Have you thought about setting a timescale on this? Do we want 1.0 in a month? Three months? A year?

I think the most important part of 1.0 will be correctness. We need to set our reputation from our 1.0 release. I'm going to do what I've been saying I'll do for a while and set up some stress tests (and maybe some performance tests). I have some compute resources I can dedicate to this full time to help us catch the one-in-a-billion synchronisation errors I'm worried about.

[jdantonio]

No specific timeline yet, but setting one is a good idea. My main goal here was to provide focus and clarity on what will be included. Having said that:

• The 0.9 release will be as soon as I can reasonably finish those items. I'd like to give our users plenty of time to remove any deprecated method calls. No more than a month or two.
• I'd like to have 1.0 release before @tenderlove releases a version of Rails that uses concurrent-ruby. When that happens we'll see 100,000 downloads virtually overnight. I'll reach out to him regarding his plans.
• If I had to set a release date for 1.0 irrespective of Rails I'd say 3 to 6 months after the release of 0.9, unless we discover major stability problems. In that case I'd push the release back as far as necessary. As you suggested, the 1.0 release needs to be rock-solid if we want to establish a good reputation.

[pitr-ch]

@tenderlove when is the next Rails release which would possibly use concurrent-ruby ?

I agree I would also aim for correctness and quality. To achieve that I am toying with and idea to split concurrent-ruby into two parts. Rock solid core which we release as 1.0, moving rest (like actors since it still evolves, and agents since they are incomplete, parts still lacking documentation if left, etc.) to another gem concurrent-ruby-incubator. Incubator description:

• same repository
• semantic versioning but rapid, breaking changes would be probably coming often
• goal is to stabilize features and move to core, not to do something like contrib
• moving to core would mostly just mean minor++ since adding is not breaking
• no C extensions for incubator
• probably all low-level stuff would go to core
• incubator would share same namespace, so when features are moved to core users don't need to do anything.
• it should shrink the amount of work needed for 1.0 allowing us to do it maybe sooner and with more quality.

What do you think? What hidden problems are lurking here?

I would like to go over every code going into 1.0 to see what problems I may find. After that I would like to bring up again the ack-ing process of PRs to keep core very stable. More eyes more problems caught. I am already reviewing all the PRs but sometimes they are merged before I get to them and that sometimes leads to more PRs being created which makes it harder to track. I think longer opened PRs are fine.

I'll have some time in April. Could we postpone 0.9 to the end of April? I would like to use the time finish few undone things and to fix bugs we have reported in the issues. I am also thinking that I am postpone some of the actor updates (erlang supervisor) in favor of the Future/Promise #139 I would very much like to see that in 1.0 which means it must be in 0.9 too.

@chrisseaton great you are starting with the performance testing. Are you also planning to compare Ruby vs C/Java extensions? I saw once some weird numbers around Ruby vs Java pools, I've forgot about that since then :/

I've filled few new issues for 0.9 and 1.0. I am excited that we will go 1.0 👍

[tenderlove]

@tenderlove when is the next Rails release which would possibly use concurrent-ruby ?

Unknown. I haven't broached the subject with the rest of the team yet, but we need thread safe hashes (for caches), latch classes (for doing streaming stuff), and a LIFO queue would be really nice too (though I haven't checked if there is one in here). I think it will be an easy sell. I don't think any of us are interested in maintaining that stuff.

Anyway, the soonest would be with Rails 5 which is sometime this year. IMO 1.0 isn't that important to us since I think at first we'd just use the gem to rm our Latch implementation.

[jdantonio]

@pitr-ch Thoughts, in no particular order:

• I'm comfortable creating a second gem. That is an idea I proposed in Issue v0.7.0, GitHub Organization, and Merging Atomic #83 back in May. It's just a question of what would go into the second gem.
• If you don't feel that Actor is ready for 1.0 then moving that to a second gem makes sense.
• I think Agent can stay in core. It hasn't changed in a very long time. We can still add featured post-1.0.
• Anything that lacks documentation now should be a priority for 1.0, rather than leaving it out. I've started writing more documentation.
• I'm comfortable with creating some conventions around PRs. Since we've been so informal it's difficult to know who wants to provide feedback and how long we should wait.
• April 5th was just a proposed release date. We can move it back so long as we have a plan. I only chose that date because it's my birthday. :-)
• I agree that the second gem, whatever we call it, should use the same namespace and not use C extensions. I'd go a step further and say no Java extensions, either (but JRuby interop is OK). That gem should be used for working out the APIs. performance optimizations can occur after code is moved to core.
• I disagree with Consolidate IVar childen into one object #139 and First draft IVar-Future-Promise-Probe unification #176 being in 0.9 and 1.0. The APIs for those objects have been stable for several releases. People are using this gem in production and no one from the community has suggested a need to change them. Although I like many of your suggestions, it feels very experimental to me. I think we serve the community better by creating a stable, performant 1.0 release with the current APIs then growing the user base. We can then let production use cases drive our evolution.

[jdantonio]

@pitr-ch How about concurrent-ruby-edge for the new gem? The word "incubator" is grammatically correct but for some reason (which I cannot fully explain) feels not right. The word "edge" used used in similar software situations, such as Edge Rails. The connotation of "cutting edge" feels better. I don't feel strongly about this, it's just a thought.

I've also begun thinking about moving Channel to that gem as well. That code was written a long time ago and I've never used it. It isn't documented and--to be perfectly honest--I've never looked closely enough at the code to know how it works.

[mighe]

@jdantonio I agree to move Channel to the edge gem.
I'm too busy at the moment to document it (I'm developing a system using our gem and it will be deployed next month, I'll probably come back after it), but the real problem is that, AFAIK, nobody is using Channel at the moment, so it is still not battle tested.
Not only: Channel can be implemented in an external gem that uses concurrent-ruby, there is probably no need to keep it inside the main gem.

[jdantonio]

@pitr-ch I've pushed back the target release dates. Again, these aren't written in stone. We'll push back as often as necessary to make sure we are confident in the code. The current release dates correspond to this summer's Marvel superhero movie release dates. :-)

[pitr-ch]

I am not native English so I may get it wrong but edge implies for me 'same stuff but newer and maybe unstable' where incubator implies 'making/trying out new stuff for core' the second seems more accurate to me.

Thanks for pushing it back.

About integration of #139 and #176. We already felt the pain of the scattered implementations with @iNecas when porting dynflow to concurrent-ruby. Things like: we needed to use join or chaining but that's not available on IVar. There was no zip on promise at that time (dataflow is blocking threads so we did not want to use it). So it's not just me experimenting it comes from what I saw as problem for users. Admittedly, I took it quite far breaking quite a lot of the old APIs. That's one of the reasons I would like to get it to 0.9. Other is that it is quite powerful with the delay and schedule integration so it can be used to implement and simplify some of the classes already present in concurrent-ruby. Third reason is that if it is not going to 1.0 it'll be delayed to 2.0 (for its nature) which is quite far. Please, Let's not close this discussion just yet. Let me get this in more stable/documented/with-examples state in April and let's revisit it again then.