[VOTE] Heron to enter Apache Incubator
Locked
12
12
[VOTE] Heron to enter Apache Incubator
 
|
Hi,
Based on the discussion on the incubator mailing list[1] I would like to call a vote to add Heron to the Apache Incubator. The full proposal is available below, and is also available on the Apache Incubator wiki at: https://wiki.apache.org/incubator/HeronProposal Please vote: [ ] +1, bring Heron into Incubator [ ] -1, do not bring Heron into Incubator, because... The vote will open for 7 days until Friday June 23 at 14:00 PT. Thank you 1 - https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E = Heron Proposal = = Abstract = Heron is a real-time, distributed, fault-tolerant stream processing engine initially developed by Twitter. = Proposal = Heron is a real-time stream processing engine built for high performance, ease of manageability, performance predictability and developer productivity[1]. We wish to develop a community around Heron to increase contributions and see Heron thrive in an open forum. = Background = Heron provides the ability for developers to compose directed acyclic graphs (DAGs) of real-time query execution logic (i.e. a topology) and submit the topology to execute on a pluggable job scheduling system (e.g., Apache Aurora, YARN, Marathon, etc). Users can employ either the native Heron API or the Apache Storm API to develop the topology. Heron supports the Storm API for ease of migration, but beyond that Heron’s architecture differs considerably from Storm’s. Users submit a topology to the scheduler using the Heron client, which uses the Heron binary libraries to deploy all daemons required to run and manage the topology. The topology therefore has no reliance on centrally managed Heron services, only on a generic job scheduling system, which lends itself well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among others). The scheduler runs each topology as a job consisting of multiple containers. One of the containers runs the topology master, responsible for managing the topology. The remaining containers each runs a stream manager responsible for data routing, a metrics manager that collects and reports various metrics and a number of processes called Heron instances which run the user-defined logic on the stream of tuples. Parallelism is achieved via process-based isolation of Heron instances, which provides predictable performance while simplifying debugging. The containers are allocated and managed by the scheduler framework based on resource availability of nodes in the cluster. The metadata for the topology, such as the physical plan and execution details, are stored in the pluggable Heron State Manager (e.g. Apache ZooKeeper). = Rationale = Heron is a general-purpose, modular and extensible platform that can be leveraged to support common, real-time analytics use cases. There is an increasing demand for open-source, scalable real-time analytics systems. We believe that Heron can be leveraged by other organizations to build streaming applications that can benefit from its robustness, high performance, adaptability to cloud environments and ease of use. Moreover, we hope that open-sourcing Heron will help to further evolve the technology as the project attracts contributors with diverse backgrounds and areas of expertise. We believe the Apache foundation is a great fit as the long-term home for Heron, as it provides an established process for community-driven development and decision making by consensus. This is exactly the model we want for future Heron development. = Initial Goals = * Move the existing codebase, website, documentation, and mailing lists to Apache-hosted infrastructure. * Integrate with the Apache development process. * Ensure all dependencies are compliant with Apache License version 2.0. * Incrementally develop and release per Apache guidelines. = Current Status = Heron is a stable project used in production at Twitter since 2014 and open sourced under the ASL v2 license in 2016. The Heron source code is currently hosted at github.com (https://github.com/twitter/heron), which will seed the Apache git repository. = Meritocracy = By submitting this incubator proposal, we’re expressing our intent to build a diverse developer community around Heron that will conduct itself according to The Apache Way and use a meritocratic means of building it's committer base. Several companies and universities have already expressed interest in and contributed to Heron. Our goal is to grow the Heron community by encouraging open communication, contribution and participation of all types, and ensuring that contributors are recognized appropriately. = Community = Heron is currently being used by Twitter, Google, Machine Zone and ndustrial.io and has received significant contributions by Microsoft and Streamlio. By bringing Heron into the Apache ecosystem, we believe we can attract even more developers who are interested in creating real-time systems to build the project's contributor base. == Core Developers == Current core developers are engineers from Twitter, Google, Microsoft and Streamlio. == Alignment == Heron utilizes a number of Apache technologies. Heron leverages Apache ZooKeeper for coordination and has scheduler implementations to integrate with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF) as well as spout implementations to integrate with Apache Kafka and metrics implementations to integrate with Scribe. Heron also implements the Apache Storm user-level API, which allows topologies written against Storm to run in Heron. We believe that having Heron at Apache will help further the growth of the streaming compute community, as well as encourage cooperation and developer cross pollination with other Apache projects. = Known Risks = == Orphaned Products == The risk of the Heron project being abandoned is minimal. It is used in production at Twitter and Google and other companies are evaluating or adopting it for production use. == Inexperience with Open Source == All of the core contributors to the project have considerable experience with open source software development. Bill Graham[2], Ashvin Agrawal[3] and Supun Kamburugamuve[4], committers on the project, are PMCs on other Apache projects and Bill and Ashvin have gone through the Apache incubator process. Twitter has already donated numerous projects to the ASF (e.g., Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored by experienced ASF members that can help with any roadblocks. == Homogenous Developers == Initial committers come from 5 separate organizations. Our intention is increase the diversity of contributing developers and their affiliations. To date github contributions have come from approximately 50 contributors from outside the Twitter team. == Reliance on Salaried Developers == It is expected that Heron development will occur on both salaried time and on volunteer time. The majority of initial committers are paid by their employers to contribute to this project. We are committed to recruiting additional committers from other organizations as well as non-salaried committers to join project. == Relationships with Other Apache Products == As mentioned in the Alignment section, Heron implements the Apache Storm API and integrates with multiple Apache schedulers (Apache Mesos, Apache Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache Thrift. == An Excessive Fascination with the Apache Brand == Heron's popularity is growing in the streaming compute space and we are long time supporters of the Apache brand. This proposal is not for the purpose of generating publicity through. Rather, the primary benefits to joining Apache are those of community building and open decision making outlined in the Rationale section. == Documentation == This proposal exists online as http://wiki.apache.org/incubator/HeronProposal. Extensive documentation can be found on github at https://twitter.github.io/heron and the source code is well documented. == Source and Intellectual Property Submission Plan == The Heron codebase is currently hosted on Github: https://github.com/twitter/heron. During incubation, the codebase will be migrated to Apache infrastructure. The source code is already ASF 2.0 licensed. == External Dependencies == All external libraries have ASF 2.0 compatible licenses except for pylint. The pylint library is GPL licensed, but is only used for pre-build Python style checks and is neither bundled with, nor relied upon by, the Heron source or binary release artifacts. == Cryptography == Heron does not use any cryptography libraries. = Required Resources = == Mailing lists == * [hidden email] (with moderated subscriptions) * [hidden email] * [hidden email] * [hidden email] == Subversion Directory == Git is the preferred source control system: git://git.apache.org/heron == Issue Tracking == JIRA: Heron (HERON) == Initial Committers == * Andrew Jorgensen (andrew at andrewjorgensen dot com) * Ashvin Agrawal (ashvin at apache dot org)* * Avrilia Floratou (avrilia dot floratou at gmail dot com) * Bill Graham (billgraham at apache dot org)* * Brian Hatfield (bmhatfield at gmail dot com) * Chris Kellogg (cckellogg at gmail dot com) * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) * Karthik Ramasamy (karthik at gmail dot com) * Maosong Fu (maosongfu at gmail dot com) * Neng Lu(freeneng at gmail dot com) * Runhang Li (obj dot runhang at gmail dot com) * Sanjeev Kulkarni (sanjeevrk at gmail dot com) * Supun Kamburugamuve (supun at apache dot org)* * Thomas Sun (tom dot ssf at gmail dot com) * Yaliang Wang (yaliang dot w dot wang at ieee dot org) == Affiliations == * Andrew Jorgensen (Google) * Ashvin Agrawal (Microsoft) * Avrilia Floratou (Microsoft) * Bill Graham (Twitter) * Brian Hatfield (Google) * Chris Kellogg (Twitter) * Huijun Wu (Twitter) * Karthik Ramasamy (Streamlio) * Maosong Fu (Twitter) * Neng Lu (Twitter) * Runhang Li (Twitter) * Sanjeev Kulkarni (Streamlio) * Supun Kamburugamuve (Indiana University) * Thomas Sun (Twitter) * Yaliang Wang (Twitter) = Sponsors = == Champion == * Julien Le Dem (julien at apache dot org) == Nominated Mentors == * Jake Farrell (jfarrell at apache dot org) * Jacques Nadeau (jacques at apache dot org) * Julien Le Dem (julien at apache dot org) * P. Taylor Goetz (ptgoetz at apache dot org) == Sponsoring Entity == The Apache Incubator == Footnotes == * 1 - Papers detailing Heron are available at http://dl.acm.org/citation.cfm?id=2742788 and http://sites.computer.org/debull/A15dec/p15.pdf. * 2 - http://home.apache.org/phonebook.html?uid=billgraham * 3 - http://home.apache.org/phonebook.html?uid=ashvin * 4 - http://home.apache.org/phonebook.html?uid=supun |
Re: [VOTE] Heron to enter Apache Incubator
|
+1
On 6/16/17, 1:41 PM, "Bill Graham" <[hidden email]> wrote: Hi, Based on the discussion on the incubator mailing list[1] I would like to call a vote to add Heron to the Apache Incubator. The full proposal is available below, and is also available on the Apache Incubator wiki at: https://wiki.apache.org/incubator/HeronProposal Please vote: [ ] +1, bring Heron into Incubator [ ] -1, do not bring Heron into Incubator, because... The vote will open for 7 days until Friday June 23 at 14:00 PT. Thank you 1 - https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E = Heron Proposal = = Abstract = Heron is a real-time, distributed, fault-tolerant stream processing engine initially developed by Twitter. = Proposal = Heron is a real-time stream processing engine built for high performance, ease of manageability, performance predictability and developer productivity[1]. We wish to develop a community around Heron to increase contributions and see Heron thrive in an open forum. = Background = Heron provides the ability for developers to compose directed acyclic graphs (DAGs) of real-time query execution logic (i.e. a topology) and submit the topology to execute on a pluggable job scheduling system (e.g., Apache Aurora, YARN, Marathon, etc). Users can employ either the native Heron API or the Apache Storm API to develop the topology. Heron supports the Storm API for ease of migration, but beyond that Heron’s architecture differs considerably from Storm’s. Users submit a topology to the scheduler using the Heron client, which uses the Heron binary libraries to deploy all daemons required to run and manage the topology. The topology therefore has no reliance on centrally managed Heron services, only on a generic job scheduling system, which lends itself well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among others). The scheduler runs each topology as a job consisting of multiple containers. One of the containers runs the topology master, responsible for managing the topology. The remaining containers each runs a stream manager responsible for data routing, a metrics manager that collects and reports various metrics and a number of processes called Heron instances which run the user-defined logic on the stream of tuples. Parallelism is achieved via process-based isolation of Heron instances, which provides predictable performance while simplifying debugging. The containers are allocated and managed by the scheduler framework based on resource availability of nodes in the cluster. The metadata for the topology, such as the physical plan and execution details, are stored in the pluggable Heron State Manager (e.g. Apache ZooKeeper). = Rationale = Heron is a general-purpose, modular and extensible platform that can be leveraged to support common, real-time analytics use cases. There is an increasing demand for open-source, scalable real-time analytics systems. We believe that Heron can be leveraged by other organizations to build streaming applications that can benefit from its robustness, high performance, adaptability to cloud environments and ease of use. Moreover, we hope that open-sourcing Heron will help to further evolve the technology as the project attracts contributors with diverse backgrounds and areas of expertise. We believe the Apache foundation is a great fit as the long-term home for Heron, as it provides an established process for community-driven development and decision making by consensus. This is exactly the model we want for future Heron development. = Initial Goals = * Move the existing codebase, website, documentation, and mailing lists to Apache-hosted infrastructure. * Integrate with the Apache development process. * Ensure all dependencies are compliant with Apache License version 2.0. * Incrementally develop and release per Apache guidelines. = Current Status = Heron is a stable project used in production at Twitter since 2014 and open sourced under the ASL v2 license in 2016. The Heron source code is currently hosted at github.com (https://github.com/twitter/heron), which will seed the Apache git repository. = Meritocracy = By submitting this incubator proposal, we’re expressing our intent to build a diverse developer community around Heron that will conduct itself according to The Apache Way and use a meritocratic means of building it's committer base. Several companies and universities have already expressed interest in and contributed to Heron. Our goal is to grow the Heron community by encouraging open communication, contribution and participation of all types, and ensuring that contributors are recognized appropriately. = Community = Heron is currently being used by Twitter, Google, Machine Zone and ndustrial.io and has received significant contributions by Microsoft and Streamlio. By bringing Heron into the Apache ecosystem, we believe we can attract even more developers who are interested in creating real-time systems to build the project's contributor base. == Core Developers == Current core developers are engineers from Twitter, Google, Microsoft and Streamlio. == Alignment == Heron utilizes a number of Apache technologies. Heron leverages Apache ZooKeeper for coordination and has scheduler implementations to integrate with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF) as well as spout implementations to integrate with Apache Kafka and metrics implementations to integrate with Scribe. Heron also implements the Apache Storm user-level API, which allows topologies written against Storm to run in Heron. We believe that having Heron at Apache will help further the growth of the streaming compute community, as well as encourage cooperation and developer cross pollination with other Apache projects. = Known Risks = == Orphaned Products == The risk of the Heron project being abandoned is minimal. It is used in production at Twitter and Google and other companies are evaluating or adopting it for production use. == Inexperience with Open Source == All of the core contributors to the project have considerable experience with open source software development. Bill Graham[2], Ashvin Agrawal[3] and Supun Kamburugamuve[4], committers on the project, are PMCs on other Apache projects and Bill and Ashvin have gone through the Apache incubator process. Twitter has already donated numerous projects to the ASF (e.g., Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored by experienced ASF members that can help with any roadblocks. == Homogenous Developers == Initial committers come from 5 separate organizations. Our intention is increase the diversity of contributing developers and their affiliations. To date github contributions have come from approximately 50 contributors from outside the Twitter team. == Reliance on Salaried Developers == It is expected that Heron development will occur on both salaried time and on volunteer time. The majority of initial committers are paid by their employers to contribute to this project. We are committed to recruiting additional committers from other organizations as well as non-salaried committers to join project. == Relationships with Other Apache Products == As mentioned in the Alignment section, Heron implements the Apache Storm API and integrates with multiple Apache schedulers (Apache Mesos, Apache Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache Thrift. == An Excessive Fascination with the Apache Brand == Heron's popularity is growing in the streaming compute space and we are long time supporters of the Apache brand. This proposal is not for the purpose of generating publicity through. Rather, the primary benefits to joining Apache are those of community building and open decision making outlined in the Rationale section. == Documentation == This proposal exists online as http://wiki.apache.org/incubator/HeronProposal. Extensive documentation can be found on github at https://twitter.github.io/heron and the source code is well documented. == Source and Intellectual Property Submission Plan == The Heron codebase is currently hosted on Github: https://github.com/twitter/heron. During incubation, the codebase will be migrated to Apache infrastructure. The source code is already ASF 2.0 licensed. == External Dependencies == All external libraries have ASF 2.0 compatible licenses except for pylint. The pylint library is GPL licensed, but is only used for pre-build Python style checks and is neither bundled with, nor relied upon by, the Heron source or binary release artifacts. == Cryptography == Heron does not use any cryptography libraries. = Required Resources = == Mailing lists == * [hidden email] (with moderated subscriptions) * [hidden email] * [hidden email] * [hidden email] == Subversion Directory == Git is the preferred source control system: git://git.apache.org/heron == Issue Tracking == JIRA: Heron (HERON) == Initial Committers == * Andrew Jorgensen (andrew at andrewjorgensen dot com) * Ashvin Agrawal (ashvin at apache dot org)* * Avrilia Floratou (avrilia dot floratou at gmail dot com) * Bill Graham (billgraham at apache dot org)* * Brian Hatfield (bmhatfield at gmail dot com) * Chris Kellogg (cckellogg at gmail dot com) * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) * Karthik Ramasamy (karthik at gmail dot com) * Maosong Fu (maosongfu at gmail dot com) * Neng Lu(freeneng at gmail dot com) * Runhang Li (obj dot runhang at gmail dot com) * Sanjeev Kulkarni (sanjeevrk at gmail dot com) * Supun Kamburugamuve (supun at apache dot org)* * Thomas Sun (tom dot ssf at gmail dot com) * Yaliang Wang (yaliang dot w dot wang at ieee dot org) == Affiliations == * Andrew Jorgensen (Google) * Ashvin Agrawal (Microsoft) * Avrilia Floratou (Microsoft) * Bill Graham (Twitter) * Brian Hatfield (Google) * Chris Kellogg (Twitter) * Huijun Wu (Twitter) * Karthik Ramasamy (Streamlio) * Maosong Fu (Twitter) * Neng Lu (Twitter) * Runhang Li (Twitter) * Sanjeev Kulkarni (Streamlio) * Supun Kamburugamuve (Indiana University) * Thomas Sun (Twitter) * Yaliang Wang (Twitter) = Sponsors = == Champion == * Julien Le Dem (julien at apache dot org) == Nominated Mentors == * Jake Farrell (jfarrell at apache dot org) * Jacques Nadeau (jacques at apache dot org) * Julien Le Dem (julien at apache dot org) * P. Taylor Goetz (ptgoetz at apache dot org) == Sponsoring Entity == The Apache Incubator == Footnotes == * 1 - Papers detailing Heron are available at http://dl.acm.org/citation.cfm?id=2742788 and http://sites.computer.org/debull/A15dec/p15.pdf. * 2 - http://home.apache.org/phonebook.html?uid=billgraham * 3 - http://home.apache.org/phonebook.html?uid=ashvin * 4 - http://home.apache.org/phonebook.html?uid=supun --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
Re: [VOTE] Heron to enter Apache Incubator
|
+1
On Fri, Jun 16, 2017 at 3:45 PM, Debo Dutta (dedutta) <[hidden email]> wrote: > +1 > > On 6/16/17, 1:41 PM, "Bill Graham" <[hidden email]> wrote: > > Hi, > > Based on the discussion on the incubator mailing list[1] I would like > to > call a vote to add Heron to the Apache Incubator. > > The full proposal is available below, and is also available on the > Apache > Incubator wiki at: > https://wiki.apache.org/incubator/HeronProposal > > Please vote: > [ ] +1, bring Heron into Incubator > [ ] -1, do not bring Heron into Incubator, because... > > The vote will open for 7 days until Friday June 23 at 14:00 PT. > > Thank you > > 1 - > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7 > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E > > > > = Heron Proposal = > > = Abstract = > Heron is a real-time, distributed, fault-tolerant stream processing > engine > initially developed by Twitter. > > = Proposal = > > Heron is a real-time stream processing engine built for high > performance, > ease of manageability, performance predictability and developer > productivity[1]. We wish to develop a community around Heron to > increase > contributions and see Heron thrive in an open forum. > > = Background = > > Heron provides the ability for developers to compose directed acyclic > graphs (DAGs) of real-time query execution logic (i.e. a topology) and > submit the topology to execute on a pluggable job scheduling system > (e.g., > Apache Aurora, YARN, Marathon, etc). Users can employ either the native > Heron API or the Apache Storm API to develop the topology. Heron > supports > the Storm API for ease of migration, but beyond that Heron’s > architecture > differs considerably from Storm’s. > > Users submit a topology to the scheduler using the Heron client, which > uses > the Heron binary libraries to deploy all daemons required to run and > manage > the topology. The topology therefore has no reliance on centrally > managed > Heron services, only on a generic job scheduling system, which lends > itself > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN > (among > others). > > The scheduler runs each topology as a job consisting of multiple > containers. One of the containers runs the topology master, > responsible for > managing the topology. The remaining containers each runs a stream > manager > responsible for data routing, a metrics manager that collects and > reports > various metrics and a number of processes called Heron instances which > run > the user-defined logic on the stream of tuples. Parallelism is > achieved via > process-based isolation of Heron instances, which provides predictable > performance while simplifying debugging. The containers are allocated > and > managed by the scheduler framework based on resource availability of > nodes > in the cluster. The metadata for the topology, such as the physical > plan > and execution details, are stored in the pluggable Heron State Manager > (e.g. Apache ZooKeeper). > > = Rationale = > > Heron is a general-purpose, modular and extensible platform that can be > leveraged to support common, real-time analytics use cases. There is an > increasing demand for open-source, scalable real-time analytics > systems. We > believe that Heron can be leveraged by other organizations to build > streaming applications that can benefit from its robustness, high > performance, adaptability to cloud environments and ease of use. > Moreover, > we hope that open-sourcing Heron will help to further evolve the > technology > as the project attracts contributors with diverse backgrounds and > areas of > expertise. > > We believe the Apache foundation is a great fit as the long-term home > for > Heron, as it provides an established process for community-driven > development and decision making by consensus. This is exactly the > model we > want for future Heron development. > > = Initial Goals = > > * Move the existing codebase, website, documentation, and mailing > lists to > Apache-hosted infrastructure. > * Integrate with the Apache development process. > * Ensure all dependencies are compliant with Apache License version > 2.0. > * Incrementally develop and release per Apache guidelines. > > = Current Status = > > Heron is a stable project used in production at Twitter since 2014 and > open > sourced under the ASL v2 license in 2016. The Heron source code is > currently hosted at github.com (https://github.com/twitter/heron), > which > will seed the Apache git repository. > > = Meritocracy = > > By submitting this incubator proposal, we’re expressing our intent to > build > a diverse developer community around Heron that will conduct itself > according to The Apache Way and use a meritocratic means of building > it's > committer base. Several companies and universities have already > expressed > interest in and contributed to Heron. Our goal is to grow the Heron > community by encouraging open communication, contribution and > participation > of all types, and ensuring that contributors are recognized > appropriately. > > = Community = > > Heron is currently being used by Twitter, Google, Machine Zone and > ndustrial.io and has received significant contributions by Microsoft > and > Streamlio. By bringing Heron into the Apache ecosystem, we believe we > can > attract even more developers who are interested in creating real-time > systems to build the project's contributor base. > > == Core Developers == > > Current core developers are engineers from Twitter, Google, Microsoft > and > Streamlio. > > == Alignment == > > Heron utilizes a number of Apache technologies. Heron leverages Apache > ZooKeeper for coordination and has scheduler implementations to > integrate > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache > REEF) > as well as spout implementations to integrate with Apache Kafka and > metrics > implementations to integrate with Scribe. Heron also implements the > Apache > Storm user-level API, which allows topologies written against Storm to > run > in Heron. We believe that having Heron at Apache will help further the > growth of the streaming compute community, as well as encourage > cooperation > and developer cross pollination with other Apache projects. > > = Known Risks = > > == Orphaned Products == > > The risk of the Heron project being abandoned is minimal. It is used in > production at Twitter and Google and other companies are evaluating or > adopting it for production use. > > == Inexperience with Open Source == > > All of the core contributors to the project have considerable > experience > with open source software development. Bill Graham[2], Ashvin > Agrawal[3] > and Supun Kamburugamuve[4], committers on the project, are PMCs on > other > Apache projects and Bill and Ashvin have gone through the Apache > incubator > process. Twitter has already donated numerous projects to the ASF > (e.g., > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be > mentored > by experienced ASF members that can help with any roadblocks. > > == Homogenous Developers == > > Initial committers come from 5 separate organizations. Our intention is > increase the diversity of contributing developers and their > affiliations. > To date github contributions have come from approximately 50 > contributors > from outside the Twitter team. > > == Reliance on Salaried Developers == > > It is expected that Heron development will occur on both salaried time > and > on volunteer time. The majority of initial committers are paid by their > employers to contribute to this project. We are committed to recruiting > additional committers from other organizations as well as non-salaried > committers to join project. > > == Relationships with Other Apache Products == > > As mentioned in the Alignment section, Heron implements the Apache > Storm > API and integrates with multiple Apache schedulers (Apache Mesos, > Apache > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache > Thrift. > > == An Excessive Fascination with the Apache Brand == > > Heron's popularity is growing in the streaming compute space and we are > long time supporters of the Apache brand. This proposal is not for the > purpose of generating publicity through. Rather, the primary benefits > to > joining Apache are those of community building and open decision making > outlined in the Rationale section. > > == Documentation == > > This proposal exists online as > http://wiki.apache.org/incubator/HeronProposal. Extensive > documentation can > be found on github at https://twitter.github.io/heron and the source > code > is well documented. > > == Source and Intellectual Property Submission Plan == > > The Heron codebase is currently hosted on Github: > https://github.com/twitter/heron. During incubation, the codebase > will be > migrated to Apache infrastructure. The source code is already ASF 2.0 > licensed. > > == External Dependencies == > > All external libraries have ASF 2.0 compatible licenses except for > pylint. > The pylint library is GPL licensed, but is only used for pre-build > Python > style checks and is neither bundled with, nor relied upon by, the Heron > source or binary release artifacts. > > == Cryptography == > > Heron does not use any cryptography libraries. > > = Required Resources = > > == Mailing lists == > > * [hidden email] (with moderated subscriptions) > * [hidden email] > * [hidden email] > * [hidden email] > > == Subversion Directory == > > Git is the preferred source control system: git://git.apache.org/heron > > == Issue Tracking == > > JIRA: Heron (HERON) > > == Initial Committers == > > * Andrew Jorgensen (andrew at andrewjorgensen dot com) > * Ashvin Agrawal (ashvin at apache dot org)* > * Avrilia Floratou (avrilia dot floratou at gmail dot com) > * Bill Graham (billgraham at apache dot org)* > * Brian Hatfield (bmhatfield at gmail dot com) > * Chris Kellogg (cckellogg at gmail dot com) > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) > * Karthik Ramasamy (karthik at gmail dot com) > * Maosong Fu (maosongfu at gmail dot com) > * Neng Lu(freeneng at gmail dot com) > * Runhang Li (obj dot runhang at gmail dot com) > * Sanjeev Kulkarni (sanjeevrk at gmail dot com) > * Supun Kamburugamuve (supun at apache dot org)* > * Thomas Sun (tom dot ssf at gmail dot com) > * Yaliang Wang (yaliang dot w dot wang at ieee dot org) > > == Affiliations == > > * Andrew Jorgensen (Google) > * Ashvin Agrawal (Microsoft) > * Avrilia Floratou (Microsoft) > * Bill Graham (Twitter) > * Brian Hatfield (Google) > * Chris Kellogg (Twitter) > * Huijun Wu (Twitter) > * Karthik Ramasamy (Streamlio) > * Maosong Fu (Twitter) > * Neng Lu (Twitter) > * Runhang Li (Twitter) > * Sanjeev Kulkarni (Streamlio) > * Supun Kamburugamuve (Indiana University) > * Thomas Sun (Twitter) > * Yaliang Wang (Twitter) > > = Sponsors = > > == Champion == > > * Julien Le Dem (julien at apache dot org) > > == Nominated Mentors == > > * Jake Farrell (jfarrell at apache dot org) > * Jacques Nadeau (jacques at apache dot org) > * Julien Le Dem (julien at apache dot org) > * P. Taylor Goetz (ptgoetz at apache dot org) > > == Sponsoring Entity == > > The Apache Incubator > > == Footnotes == > > * 1 - Papers detailing Heron are available at > http://dl.acm.org/citation.cfm?id=2742788 and > http://sites.computer.org/debull/A15dec/p15.pdf. > * 2 - http://home.apache.org/phonebook.html?uid=billgraham > * 3 - http://home.apache.org/phonebook.html?uid=ashvin > * 4 - http://home.apache.org/phonebook.html?uid=supun > > > |
Re: [VOTE] Heron to enter Apache Incubator
|
+1
On Fri, Jun 16, 2017 at 1:46 PM, William Markito Oliveira < [hidden email]> wrote: > +1 > > On Fri, Jun 16, 2017 at 3:45 PM, Debo Dutta (dedutta) <[hidden email]> > wrote: > > > +1 > > > > On 6/16/17, 1:41 PM, "Bill Graham" <[hidden email]> wrote: > > > > Hi, > > > > Based on the discussion on the incubator mailing list[1] I would like > > to > > call a vote to add Heron to the Apache Incubator. > > > > The full proposal is available below, and is also available on the > > Apache > > Incubator wiki at: > > https://wiki.apache.org/incubator/HeronProposal > > > > Please vote: > > [ ] +1, bring Heron into Incubator > > [ ] -1, do not bring Heron into Incubator, because... > > > > The vote will open for 7 days until Friday June 23 at 14:00 PT. > > > > Thank you > > > > 1 - > > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7 > > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E > > > > > > > > = Heron Proposal = > > > > = Abstract = > > Heron is a real-time, distributed, fault-tolerant stream processing > > engine > > initially developed by Twitter. > > > > = Proposal = > > > > Heron is a real-time stream processing engine built for high > > performance, > > ease of manageability, performance predictability and developer > > productivity[1]. We wish to develop a community around Heron to > > increase > > contributions and see Heron thrive in an open forum. > > > > = Background = > > > > Heron provides the ability for developers to compose directed acyclic > > graphs (DAGs) of real-time query execution logic (i.e. a topology) > and > > submit the topology to execute on a pluggable job scheduling system > > (e.g., > > Apache Aurora, YARN, Marathon, etc). Users can employ either the > native > > Heron API or the Apache Storm API to develop the topology. Heron > > supports > > the Storm API for ease of migration, but beyond that Heron’s > > architecture > > differs considerably from Storm’s. > > > > Users submit a topology to the scheduler using the Heron client, > which > > uses > > the Heron binary libraries to deploy all daemons required to run and > > manage > > the topology. The topology therefore has no reliance on centrally > > managed > > Heron services, only on a generic job scheduling system, which lends > > itself > > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN > > (among > > others). > > > > The scheduler runs each topology as a job consisting of multiple > > containers. One of the containers runs the topology master, > > responsible for > > managing the topology. The remaining containers each runs a stream > > manager > > responsible for data routing, a metrics manager that collects and > > reports > > various metrics and a number of processes called Heron instances > which > > run > > the user-defined logic on the stream of tuples. Parallelism is > > achieved via > > process-based isolation of Heron instances, which provides > predictable > > performance while simplifying debugging. The containers are allocated > > and > > managed by the scheduler framework based on resource availability of > > nodes > > in the cluster. The metadata for the topology, such as the physical > > plan > > and execution details, are stored in the pluggable Heron State > Manager > > (e.g. Apache ZooKeeper). > > > > = Rationale = > > > > Heron is a general-purpose, modular and extensible platform that can > be > > leveraged to support common, real-time analytics use cases. There is > an > > increasing demand for open-source, scalable real-time analytics > > systems. We > > believe that Heron can be leveraged by other organizations to build > > streaming applications that can benefit from its robustness, high > > performance, adaptability to cloud environments and ease of use. > > Moreover, > > we hope that open-sourcing Heron will help to further evolve the > > technology > > as the project attracts contributors with diverse backgrounds and > > areas of > > expertise. > > > > We believe the Apache foundation is a great fit as the long-term home > > for > > Heron, as it provides an established process for community-driven > > development and decision making by consensus. This is exactly the > > model we > > want for future Heron development. > > > > = Initial Goals = > > > > * Move the existing codebase, website, documentation, and mailing > > lists to > > Apache-hosted infrastructure. > > * Integrate with the Apache development process. > > * Ensure all dependencies are compliant with Apache License version > > 2.0. > > * Incrementally develop and release per Apache guidelines. > > > > = Current Status = > > > > Heron is a stable project used in production at Twitter since 2014 > and > > open > > sourced under the ASL v2 license in 2016. The Heron source code is > > currently hosted at github.com (https://github.com/twitter/heron), > > which > > will seed the Apache git repository. > > > > = Meritocracy = > > > > By submitting this incubator proposal, we’re expressing our intent to > > build > > a diverse developer community around Heron that will conduct itself > > according to The Apache Way and use a meritocratic means of building > > it's > > committer base. Several companies and universities have already > > expressed > > interest in and contributed to Heron. Our goal is to grow the Heron > > community by encouraging open communication, contribution and > > participation > > of all types, and ensuring that contributors are recognized > > appropriately. > > > > = Community = > > > > Heron is currently being used by Twitter, Google, Machine Zone and > > ndustrial.io and has received significant contributions by Microsoft > > and > > Streamlio. By bringing Heron into the Apache ecosystem, we believe we > > can > > attract even more developers who are interested in creating real-time > > systems to build the project's contributor base. > > > > == Core Developers == > > > > Current core developers are engineers from Twitter, Google, Microsoft > > and > > Streamlio. > > > > == Alignment == > > > > Heron utilizes a number of Apache technologies. Heron leverages > Apache > > ZooKeeper for coordination and has scheduler implementations to > > integrate > > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache > > REEF) > > as well as spout implementations to integrate with Apache Kafka and > > metrics > > implementations to integrate with Scribe. Heron also implements the > > Apache > > Storm user-level API, which allows topologies written against Storm > to > > run > > in Heron. We believe that having Heron at Apache will help further > the > > growth of the streaming compute community, as well as encourage > > cooperation > > and developer cross pollination with other Apache projects. > > > > = Known Risks = > > > > == Orphaned Products == > > > > The risk of the Heron project being abandoned is minimal. It is used > in > > production at Twitter and Google and other companies are evaluating > or > > adopting it for production use. > > > > == Inexperience with Open Source == > > > > All of the core contributors to the project have considerable > > experience > > with open source software development. Bill Graham[2], Ashvin > > Agrawal[3] > > and Supun Kamburugamuve[4], committers on the project, are PMCs on > > other > > Apache projects and Bill and Ashvin have gone through the Apache > > incubator > > process. Twitter has already donated numerous projects to the ASF > > (e.g., > > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be > > mentored > > by experienced ASF members that can help with any roadblocks. > > > > == Homogenous Developers == > > > > Initial committers come from 5 separate organizations. Our intention > is > > increase the diversity of contributing developers and their > > affiliations. > > To date github contributions have come from approximately 50 > > contributors > > from outside the Twitter team. > > > > == Reliance on Salaried Developers == > > > > It is expected that Heron development will occur on both salaried > time > > and > > on volunteer time. The majority of initial committers are paid by > their > > employers to contribute to this project. We are committed to > recruiting > > additional committers from other organizations as well as > non-salaried > > committers to join project. > > > > == Relationships with Other Apache Products == > > > > As mentioned in the Alignment section, Heron implements the Apache > > Storm > > API and integrates with multiple Apache schedulers (Apache Mesos, > > Apache > > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and > Apache > > Thrift. > > > > == An Excessive Fascination with the Apache Brand == > > > > Heron's popularity is growing in the streaming compute space and we > are > > long time supporters of the Apache brand. This proposal is not for > the > > purpose of generating publicity through. Rather, the primary benefits > > to > > joining Apache are those of community building and open decision > making > > outlined in the Rationale section. > > > > == Documentation == > > > > This proposal exists online as > > http://wiki.apache.org/incubator/HeronProposal. Extensive > > documentation can > > be found on github at https://twitter.github.io/heron and the source > > code > > is well documented. > > > > == Source and Intellectual Property Submission Plan == > > > > The Heron codebase is currently hosted on Github: > > https://github.com/twitter/heron. During incubation, the codebase > > will be > > migrated to Apache infrastructure. The source code is already ASF 2.0 > > licensed. > > > > == External Dependencies == > > > > All external libraries have ASF 2.0 compatible licenses except for > > pylint. > > The pylint library is GPL licensed, but is only used for pre-build > > Python > > style checks and is neither bundled with, nor relied upon by, the > Heron > > source or binary release artifacts. > > > > == Cryptography == > > > > Heron does not use any cryptography libraries. > > > > = Required Resources = > > > > == Mailing lists == > > > > * [hidden email] (with moderated subscriptions) > > * [hidden email] > > * [hidden email] > > * [hidden email] > > > > == Subversion Directory == > > > > Git is the preferred source control system: git:// > git.apache.org/heron > > > > == Issue Tracking == > > > > JIRA: Heron (HERON) > > > > == Initial Committers == > > > > * Andrew Jorgensen (andrew at andrewjorgensen dot com) > > * Ashvin Agrawal (ashvin at apache dot org)* > > * Avrilia Floratou (avrilia dot floratou at gmail dot com) > > * Bill Graham (billgraham at apache dot org)* > > * Brian Hatfield (bmhatfield at gmail dot com) > > * Chris Kellogg (cckellogg at gmail dot com) > > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) > > * Karthik Ramasamy (karthik at gmail dot com) > > * Maosong Fu (maosongfu at gmail dot com) > > * Neng Lu(freeneng at gmail dot com) > > * Runhang Li (obj dot runhang at gmail dot com) > > * Sanjeev Kulkarni (sanjeevrk at gmail dot com) > > * Supun Kamburugamuve (supun at apache dot org)* > > * Thomas Sun (tom dot ssf at gmail dot com) > > * Yaliang Wang (yaliang dot w dot wang at ieee dot org) > > > > == Affiliations == > > > > * Andrew Jorgensen (Google) > > * Ashvin Agrawal (Microsoft) > > * Avrilia Floratou (Microsoft) > > * Bill Graham (Twitter) > > * Brian Hatfield (Google) > > * Chris Kellogg (Twitter) > > * Huijun Wu (Twitter) > > * Karthik Ramasamy (Streamlio) > > * Maosong Fu (Twitter) > > * Neng Lu (Twitter) > > * Runhang Li (Twitter) > > * Sanjeev Kulkarni (Streamlio) > > * Supun Kamburugamuve (Indiana University) > > * Thomas Sun (Twitter) > > * Yaliang Wang (Twitter) > > > > = Sponsors = > > > > == Champion == > > > > * Julien Le Dem (julien at apache dot org) > > > > == Nominated Mentors == > > > > * Jake Farrell (jfarrell at apache dot org) > > * Jacques Nadeau (jacques at apache dot org) > > * Julien Le Dem (julien at apache dot org) > > * P. Taylor Goetz (ptgoetz at apache dot org) > > > > == Sponsoring Entity == > > > > The Apache Incubator > > > > == Footnotes == > > > > * 1 - Papers detailing Heron are available at > > http://dl.acm.org/citation.cfm?id=2742788 and > > http://sites.computer.org/debull/A15dec/p15.pdf. > > * 2 - http://home.apache.org/phonebook.html?uid=billgraham > > * 3 - http://home.apache.org/phonebook.html?uid=ashvin > > * 4 - http://home.apache.org/phonebook.html?uid=supun > > > > > > > |
Re: [VOTE] Heron to enter Apache Incubator
|
+1
Op vr 16 jun. 2017 om 22:49 schreef Julien Le Dem <[hidden email]> > +1 > > On Fri, Jun 16, 2017 at 1:46 PM, William Markito Oliveira < > [hidden email]> wrote: > > > +1 > > > > On Fri, Jun 16, 2017 at 3:45 PM, Debo Dutta (dedutta) <[hidden email] > > > > wrote: > > > > > +1 > > > > > > On 6/16/17, 1:41 PM, "Bill Graham" <[hidden email]> wrote: > > > > > > Hi, > > > > > > Based on the discussion on the incubator mailing list[1] I would > like > > > to > > > call a vote to add Heron to the Apache Incubator. > > > > > > The full proposal is available below, and is also available on the > > > Apache > > > Incubator wiki at: > > > https://wiki.apache.org/incubator/HeronProposal > > > > > > Please vote: > > > [ ] +1, bring Heron into Incubator > > > [ ] -1, do not bring Heron into Incubator, because... > > > > > > The vote will open for 7 days until Friday June 23 at 14:00 PT. > > > > > > Thank you > > > > > > 1 - > > > > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7 > > > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E > > > > > > > > > > > > = Heron Proposal = > > > > > > = Abstract = > > > Heron is a real-time, distributed, fault-tolerant stream processing > > > engine > > > initially developed by Twitter. > > > > > > = Proposal = > > > > > > Heron is a real-time stream processing engine built for high > > > performance, > > > ease of manageability, performance predictability and developer > > > productivity[1]. We wish to develop a community around Heron to > > > increase > > > contributions and see Heron thrive in an open forum. > > > > > > = Background = > > > > > > Heron provides the ability for developers to compose directed > acyclic > > > graphs (DAGs) of real-time query execution logic (i.e. a topology) > > and > > > submit the topology to execute on a pluggable job scheduling system > > > (e.g., > > > Apache Aurora, YARN, Marathon, etc). Users can employ either the > > native > > > Heron API or the Apache Storm API to develop the topology. Heron > > > supports > > > the Storm API for ease of migration, but beyond that Heron’s > > > architecture > > > differs considerably from Storm’s. > > > > > > Users submit a topology to the scheduler using the Heron client, > > which > > > uses > > > the Heron binary libraries to deploy all daemons required to run > and > > > manage > > > the topology. The topology therefore has no reliance on centrally > > > managed > > > Heron services, only on a generic job scheduling system, which > lends > > > itself > > > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN > > > (among > > > others). > > > > > > The scheduler runs each topology as a job consisting of multiple > > > containers. One of the containers runs the topology master, > > > responsible for > > > managing the topology. The remaining containers each runs a stream > > > manager > > > responsible for data routing, a metrics manager that collects and > > > reports > > > various metrics and a number of processes called Heron instances > > which > > > run > > > the user-defined logic on the stream of tuples. Parallelism is > > > achieved via > > > process-based isolation of Heron instances, which provides > > predictable > > > performance while simplifying debugging. The containers are > allocated > > > and > > > managed by the scheduler framework based on resource availability > of > > > nodes > > > in the cluster. The metadata for the topology, such as the physical > > > plan > > > and execution details, are stored in the pluggable Heron State > > Manager > > > (e.g. Apache ZooKeeper). > > > > > > = Rationale = > > > > > > Heron is a general-purpose, modular and extensible platform that > can > > be > > > leveraged to support common, real-time analytics use cases. There > is > > an > > > increasing demand for open-source, scalable real-time analytics > > > systems. We > > > believe that Heron can be leveraged by other organizations to build > > > streaming applications that can benefit from its robustness, high > > > performance, adaptability to cloud environments and ease of use. > > > Moreover, > > > we hope that open-sourcing Heron will help to further evolve the > > > technology > > > as the project attracts contributors with diverse backgrounds and > > > areas of > > > expertise. > > > > > > We believe the Apache foundation is a great fit as the long-term > home > > > for > > > Heron, as it provides an established process for community-driven > > > development and decision making by consensus. This is exactly the > > > model we > > > want for future Heron development. > > > > > > = Initial Goals = > > > > > > * Move the existing codebase, website, documentation, and mailing > > > lists to > > > Apache-hosted infrastructure. > > > * Integrate with the Apache development process. > > > * Ensure all dependencies are compliant with Apache License > version > > > 2.0. > > > * Incrementally develop and release per Apache guidelines. > > > > > > = Current Status = > > > > > > Heron is a stable project used in production at Twitter since 2014 > > and > > > open > > > sourced under the ASL v2 license in 2016. The Heron source code is > > > currently hosted at github.com (https://github.com/twitter/heron), > > > which > > > will seed the Apache git repository. > > > > > > = Meritocracy = > > > > > > By submitting this incubator proposal, we’re expressing our intent > to > > > build > > > a diverse developer community around Heron that will conduct itself > > > according to The Apache Way and use a meritocratic means of > building > > > it's > > > committer base. Several companies and universities have already > > > expressed > > > interest in and contributed to Heron. Our goal is to grow the Heron > > > community by encouraging open communication, contribution and > > > participation > > > of all types, and ensuring that contributors are recognized > > > appropriately. > > > > > > = Community = > > > > > > Heron is currently being used by Twitter, Google, Machine Zone and > > > ndustrial.io and has received significant contributions by > Microsoft > > > and > > > Streamlio. By bringing Heron into the Apache ecosystem, we believe > we > > > can > > > attract even more developers who are interested in creating > real-time > > > systems to build the project's contributor base. > > > > > > == Core Developers == > > > > > > Current core developers are engineers from Twitter, Google, > Microsoft > > > and > > > Streamlio. > > > > > > == Alignment == > > > > > > Heron utilizes a number of Apache technologies. Heron leverages > > Apache > > > ZooKeeper for coordination and has scheduler implementations to > > > integrate > > > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via > Apache > > > REEF) > > > as well as spout implementations to integrate with Apache Kafka and > > > metrics > > > implementations to integrate with Scribe. Heron also implements the > > > Apache > > > Storm user-level API, which allows topologies written against Storm > > to > > > run > > > in Heron. We believe that having Heron at Apache will help further > > the > > > growth of the streaming compute community, as well as encourage > > > cooperation > > > and developer cross pollination with other Apache projects. > > > > > > = Known Risks = > > > > > > == Orphaned Products == > > > > > > The risk of the Heron project being abandoned is minimal. It is > used > > in > > > production at Twitter and Google and other companies are evaluating > > or > > > adopting it for production use. > > > > > > == Inexperience with Open Source == > > > > > > All of the core contributors to the project have considerable > > > experience > > > with open source software development. Bill Graham[2], Ashvin > > > Agrawal[3] > > > and Supun Kamburugamuve[4], committers on the project, are PMCs on > > > other > > > Apache projects and Bill and Ashvin have gone through the Apache > > > incubator > > > process. Twitter has already donated numerous projects to the ASF > > > (e.g., > > > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be > > > mentored > > > by experienced ASF members that can help with any roadblocks. > > > > > > == Homogenous Developers == > > > > > > Initial committers come from 5 separate organizations. Our > intention > > is > > > increase the diversity of contributing developers and their > > > affiliations. > > > To date github contributions have come from approximately 50 > > > contributors > > > from outside the Twitter team. > > > > > > == Reliance on Salaried Developers == > > > > > > It is expected that Heron development will occur on both salaried > > time > > > and > > > on volunteer time. The majority of initial committers are paid by > > their > > > employers to contribute to this project. We are committed to > > recruiting > > > additional committers from other organizations as well as > > non-salaried > > > committers to join project. > > > > > > == Relationships with Other Apache Products == > > > > > > As mentioned in the Alignment section, Heron implements the Apache > > > Storm > > > API and integrates with multiple Apache schedulers (Apache Mesos, > > > Apache > > > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and > > Apache > > > Thrift. > > > > > > == An Excessive Fascination with the Apache Brand == > > > > > > Heron's popularity is growing in the streaming compute space and we > > are > > > long time supporters of the Apache brand. This proposal is not for > > the > > > purpose of generating publicity through. Rather, the primary > benefits > > > to > > > joining Apache are those of community building and open decision > > making > > > outlined in the Rationale section. > > > > > > == Documentation == > > > > > > This proposal exists online as > > > http://wiki.apache.org/incubator/HeronProposal. Extensive > > > documentation can > > > be found on github at https://twitter.github.io/heron and the > source > > > code > > > is well documented. > > > > > > == Source and Intellectual Property Submission Plan == > > > > > > The Heron codebase is currently hosted on Github: > > > https://github.com/twitter/heron. During incubation, the codebase > > > will be > > > migrated to Apache infrastructure. The source code is already ASF > 2.0 > > > licensed. > > > > > > == External Dependencies == > > > > > > All external libraries have ASF 2.0 compatible licenses except for > > > pylint. > > > The pylint library is GPL licensed, but is only used for pre-build > > > Python > > > style checks and is neither bundled with, nor relied upon by, the > > Heron > > > source or binary release artifacts. > > > > > > == Cryptography == > > > > > > Heron does not use any cryptography libraries. > > > > > > = Required Resources = > > > > > > == Mailing lists == > > > > > > * [hidden email] (with moderated > subscriptions) > > > * [hidden email] > > > * [hidden email] > > > * [hidden email] > > > > > > == Subversion Directory == > > > > > > Git is the preferred source control system: git:// > > git.apache.org/heron > > > > > > == Issue Tracking == > > > > > > JIRA: Heron (HERON) > > > > > > == Initial Committers == > > > > > > * Andrew Jorgensen (andrew at andrewjorgensen dot com) > > > * Ashvin Agrawal (ashvin at apache dot org)* > > > * Avrilia Floratou (avrilia dot floratou at gmail dot com) > > > * Bill Graham (billgraham at apache dot org)* > > > * Brian Hatfield (bmhatfield at gmail dot com) > > > * Chris Kellogg (cckellogg at gmail dot com) > > > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) > > > * Karthik Ramasamy (karthik at gmail dot com) > > > * Maosong Fu (maosongfu at gmail dot com) > > > * Neng Lu(freeneng at gmail dot com) > > > * Runhang Li (obj dot runhang at gmail dot com) > > > * Sanjeev Kulkarni (sanjeevrk at gmail dot com) > > > * Supun Kamburugamuve (supun at apache dot org)* > > > * Thomas Sun (tom dot ssf at gmail dot com) > > > * Yaliang Wang (yaliang dot w dot wang at ieee dot org) > > > > > > == Affiliations == > > > > > > * Andrew Jorgensen (Google) > > > * Ashvin Agrawal (Microsoft) > > > * Avrilia Floratou (Microsoft) > > > * Bill Graham (Twitter) > > > * Brian Hatfield (Google) > > > * Chris Kellogg (Twitter) > > > * Huijun Wu (Twitter) > > > * Karthik Ramasamy (Streamlio) > > > * Maosong Fu (Twitter) > > > * Neng Lu (Twitter) > > > * Runhang Li (Twitter) > > > * Sanjeev Kulkarni (Streamlio) > > > * Supun Kamburugamuve (Indiana University) > > > * Thomas Sun (Twitter) > > > * Yaliang Wang (Twitter) > > > > > > = Sponsors = > > > > > > == Champion == > > > > > > * Julien Le Dem (julien at apache dot org) > > > > > > == Nominated Mentors == > > > > > > * Jake Farrell (jfarrell at apache dot org) > > > * Jacques Nadeau (jacques at apache dot org) > > > * Julien Le Dem (julien at apache dot org) > > > * P. Taylor Goetz (ptgoetz at apache dot org) > > > > > > == Sponsoring Entity == > > > > > > The Apache Incubator > > > > > > == Footnotes == > > > > > > * 1 - Papers detailing Heron are available at > > > http://dl.acm.org/citation.cfm?id=2742788 and > > > http://sites.computer.org/debull/A15dec/p15.pdf. > > > * 2 - http://home.apache.org/phonebook.html?uid=billgraham > > > * 3 - http://home.apache.org/phonebook.html?uid=ashvin > > > * 4 - http://home.apache.org/phonebook.html?uid=supun > > > > > > > > > > > > |
Re: [VOTE] Heron to enter Apache Incubator
|
In reply to this post by Bill Graham-4
+1 (non-binding)
- Sijie On Fri, Jun 16, 2017 at 1:41 PM, Bill Graham <[hidden email]> wrote: > Hi, > > Based on the discussion on the incubator mailing list[1] I would like to > call a vote to add Heron to the Apache Incubator. > > The full proposal is available below, and is also available on the Apache > Incubator wiki at: > https://wiki.apache.org/incubator/HeronProposal > > Please vote: > [ ] +1, bring Heron into Incubator > [ ] -1, do not bring Heron into Incubator, because... > > The vote will open for 7 days until Friday June 23 at 14:00 PT. > > Thank you > > 1 - > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7 > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E > > > > = Heron Proposal = > > = Abstract = > Heron is a real-time, distributed, fault-tolerant stream processing engine > initially developed by Twitter. > > = Proposal = > > Heron is a real-time stream processing engine built for high performance, > ease of manageability, performance predictability and developer > productivity[1]. We wish to develop a community around Heron to increase > contributions and see Heron thrive in an open forum. > > = Background = > > Heron provides the ability for developers to compose directed acyclic > graphs (DAGs) of real-time query execution logic (i.e. a topology) and > submit the topology to execute on a pluggable job scheduling system (e.g., > Apache Aurora, YARN, Marathon, etc). Users can employ either the native > Heron API or the Apache Storm API to develop the topology. Heron supports > the Storm API for ease of migration, but beyond that Heron’s architecture > differs considerably from Storm’s. > > Users submit a topology to the scheduler using the Heron client, which uses > the Heron binary libraries to deploy all daemons required to run and manage > the topology. The topology therefore has no reliance on centrally managed > Heron services, only on a generic job scheduling system, which lends itself > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among > others). > > The scheduler runs each topology as a job consisting of multiple > containers. One of the containers runs the topology master, responsible for > managing the topology. The remaining containers each runs a stream manager > responsible for data routing, a metrics manager that collects and reports > various metrics and a number of processes called Heron instances which run > the user-defined logic on the stream of tuples. Parallelism is achieved via > process-based isolation of Heron instances, which provides predictable > performance while simplifying debugging. The containers are allocated and > managed by the scheduler framework based on resource availability of nodes > in the cluster. The metadata for the topology, such as the physical plan > and execution details, are stored in the pluggable Heron State Manager > (e.g. Apache ZooKeeper). > > = Rationale = > > Heron is a general-purpose, modular and extensible platform that can be > leveraged to support common, real-time analytics use cases. There is an > increasing demand for open-source, scalable real-time analytics systems. We > believe that Heron can be leveraged by other organizations to build > streaming applications that can benefit from its robustness, high > performance, adaptability to cloud environments and ease of use. Moreover, > we hope that open-sourcing Heron will help to further evolve the technology > as the project attracts contributors with diverse backgrounds and areas of > expertise. > > We believe the Apache foundation is a great fit as the long-term home for > Heron, as it provides an established process for community-driven > development and decision making by consensus. This is exactly the model we > want for future Heron development. > > = Initial Goals = > > * Move the existing codebase, website, documentation, and mailing lists to > Apache-hosted infrastructure. > * Integrate with the Apache development process. > * Ensure all dependencies are compliant with Apache License version 2.0. > * Incrementally develop and release per Apache guidelines. > > = Current Status = > > Heron is a stable project used in production at Twitter since 2014 and open > sourced under the ASL v2 license in 2016. The Heron source code is > currently hosted at github.com (https://github.com/twitter/heron), which > will seed the Apache git repository. > > = Meritocracy = > > By submitting this incubator proposal, we’re expressing our intent to build > a diverse developer community around Heron that will conduct itself > according to The Apache Way and use a meritocratic means of building it's > committer base. Several companies and universities have already expressed > interest in and contributed to Heron. Our goal is to grow the Heron > community by encouraging open communication, contribution and participation > of all types, and ensuring that contributors are recognized appropriately. > > = Community = > > Heron is currently being used by Twitter, Google, Machine Zone and > ndustrial.io and has received significant contributions by Microsoft and > Streamlio. By bringing Heron into the Apache ecosystem, we believe we can > attract even more developers who are interested in creating real-time > systems to build the project's contributor base. > > == Core Developers == > > Current core developers are engineers from Twitter, Google, Microsoft and > Streamlio. > > == Alignment == > > Heron utilizes a number of Apache technologies. Heron leverages Apache > ZooKeeper for coordination and has scheduler implementations to integrate > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF) > as well as spout implementations to integrate with Apache Kafka and metrics > implementations to integrate with Scribe. Heron also implements the Apache > Storm user-level API, which allows topologies written against Storm to run > in Heron. We believe that having Heron at Apache will help further the > growth of the streaming compute community, as well as encourage cooperation > and developer cross pollination with other Apache projects. > > = Known Risks = > > == Orphaned Products == > > The risk of the Heron project being abandoned is minimal. It is used in > production at Twitter and Google and other companies are evaluating or > adopting it for production use. > > == Inexperience with Open Source == > > All of the core contributors to the project have considerable experience > with open source software development. Bill Graham[2], Ashvin Agrawal[3] > and Supun Kamburugamuve[4], committers on the project, are PMCs on other > Apache projects and Bill and Ashvin have gone through the Apache incubator > process. Twitter has already donated numerous projects to the ASF (e.g., > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored > by experienced ASF members that can help with any roadblocks. > > == Homogenous Developers == > > Initial committers come from 5 separate organizations. Our intention is > increase the diversity of contributing developers and their affiliations. > To date github contributions have come from approximately 50 contributors > from outside the Twitter team. > > == Reliance on Salaried Developers == > > It is expected that Heron development will occur on both salaried time and > on volunteer time. The majority of initial committers are paid by their > employers to contribute to this project. We are committed to recruiting > additional committers from other organizations as well as non-salaried > committers to join project. > > == Relationships with Other Apache Products == > > As mentioned in the Alignment section, Heron implements the Apache Storm > API and integrates with multiple Apache schedulers (Apache Mesos, Apache > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache > Thrift. > > == An Excessive Fascination with the Apache Brand == > > Heron's popularity is growing in the streaming compute space and we are > long time supporters of the Apache brand. This proposal is not for the > purpose of generating publicity through. Rather, the primary benefits to > joining Apache are those of community building and open decision making > outlined in the Rationale section. > > == Documentation == > > This proposal exists online as > http://wiki.apache.org/incubator/HeronProposal. Extensive documentation > can > be found on github at https://twitter.github.io/heron and the source code > is well documented. > > == Source and Intellectual Property Submission Plan == > > The Heron codebase is currently hosted on Github: > https://github.com/twitter/heron. During incubation, the codebase will be > migrated to Apache infrastructure. The source code is already ASF 2.0 > licensed. > > == External Dependencies == > > All external libraries have ASF 2.0 compatible licenses except for pylint. > The pylint library is GPL licensed, but is only used for pre-build Python > style checks and is neither bundled with, nor relied upon by, the Heron > source or binary release artifacts. > > == Cryptography == > > Heron does not use any cryptography libraries. > > = Required Resources = > > == Mailing lists == > > * [hidden email] (with moderated subscriptions) > * [hidden email] > * [hidden email] > * [hidden email] > > == Subversion Directory == > > Git is the preferred source control system: git://git.apache.org/heron > > == Issue Tracking == > > JIRA: Heron (HERON) > > == Initial Committers == > > * Andrew Jorgensen (andrew at andrewjorgensen dot com) > * Ashvin Agrawal (ashvin at apache dot org)* > * Avrilia Floratou (avrilia dot floratou at gmail dot com) > * Bill Graham (billgraham at apache dot org)* > * Brian Hatfield (bmhatfield at gmail dot com) > * Chris Kellogg (cckellogg at gmail dot com) > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) > * Karthik Ramasamy (karthik at gmail dot com) > * Maosong Fu (maosongfu at gmail dot com) > * Neng Lu(freeneng at gmail dot com) > * Runhang Li (obj dot runhang at gmail dot com) > * Sanjeev Kulkarni (sanjeevrk at gmail dot com) > * Supun Kamburugamuve (supun at apache dot org)* > * Thomas Sun (tom dot ssf at gmail dot com) > * Yaliang Wang (yaliang dot w dot wang at ieee dot org) > > == Affiliations == > > * Andrew Jorgensen (Google) > * Ashvin Agrawal (Microsoft) > * Avrilia Floratou (Microsoft) > * Bill Graham (Twitter) > * Brian Hatfield (Google) > * Chris Kellogg (Twitter) > * Huijun Wu (Twitter) > * Karthik Ramasamy (Streamlio) > * Maosong Fu (Twitter) > * Neng Lu (Twitter) > * Runhang Li (Twitter) > * Sanjeev Kulkarni (Streamlio) > * Supun Kamburugamuve (Indiana University) > * Thomas Sun (Twitter) > * Yaliang Wang (Twitter) > > = Sponsors = > > == Champion == > > * Julien Le Dem (julien at apache dot org) > > == Nominated Mentors == > > * Jake Farrell (jfarrell at apache dot org) > * Jacques Nadeau (jacques at apache dot org) > * Julien Le Dem (julien at apache dot org) > * P. Taylor Goetz (ptgoetz at apache dot org) > > == Sponsoring Entity == > > The Apache Incubator > > == Footnotes == > > * 1 - Papers detailing Heron are available at > http://dl.acm.org/citation.cfm?id=2742788 and > http://sites.computer.org/debull/A15dec/p15.pdf. > * 2 - http://home.apache.org/phonebook.html?uid=billgraham > * 3 - http://home.apache.org/phonebook.html?uid=ashvin > * 4 - http://home.apache.org/phonebook.html?uid=supun > |
Re: [VOTE] Heron to enter Apache Incubator
|
In reply to this post by Bill Graham-4
+1
On Fri, Jun 16, 2017 at 1:41 PM, Bill Graham <[hidden email]> wrote: > Hi, > > Based on the discussion on the incubator mailing list[1] I would like to > call a vote to add Heron to the Apache Incubator. > > The full proposal is available below, and is also available on the Apache > Incubator wiki at: > https://wiki.apache.org/incubator/HeronProposal > > Please vote: > [ ] +1, bring Heron into Incubator > [ ] -1, do not bring Heron into Incubator, because... > > The vote will open for 7 days until Friday June 23 at 14:00 PT. > > Thank you > > 1 - > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7 > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E > > > > = Heron Proposal = > > = Abstract = > Heron is a real-time, distributed, fault-tolerant stream processing engine > initially developed by Twitter. > > = Proposal = > > Heron is a real-time stream processing engine built for high performance, > ease of manageability, performance predictability and developer > productivity[1]. We wish to develop a community around Heron to increase > contributions and see Heron thrive in an open forum. > > = Background = > > Heron provides the ability for developers to compose directed acyclic > graphs (DAGs) of real-time query execution logic (i.e. a topology) and > submit the topology to execute on a pluggable job scheduling system (e.g., > Apache Aurora, YARN, Marathon, etc). Users can employ either the native > Heron API or the Apache Storm API to develop the topology. Heron supports > the Storm API for ease of migration, but beyond that Heron’s architecture > differs considerably from Storm’s. > > Users submit a topology to the scheduler using the Heron client, which uses > the Heron binary libraries to deploy all daemons required to run and manage > the topology. The topology therefore has no reliance on centrally managed > Heron services, only on a generic job scheduling system, which lends itself > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among > others). > > The scheduler runs each topology as a job consisting of multiple > containers. One of the containers runs the topology master, responsible for > managing the topology. The remaining containers each runs a stream manager > responsible for data routing, a metrics manager that collects and reports > various metrics and a number of processes called Heron instances which run > the user-defined logic on the stream of tuples. Parallelism is achieved via > process-based isolation of Heron instances, which provides predictable > performance while simplifying debugging. The containers are allocated and > managed by the scheduler framework based on resource availability of nodes > in the cluster. The metadata for the topology, such as the physical plan > and execution details, are stored in the pluggable Heron State Manager > (e.g. Apache ZooKeeper). > > = Rationale = > > Heron is a general-purpose, modular and extensible platform that can be > leveraged to support common, real-time analytics use cases. There is an > increasing demand for open-source, scalable real-time analytics systems. We > believe that Heron can be leveraged by other organizations to build > streaming applications that can benefit from its robustness, high > performance, adaptability to cloud environments and ease of use. Moreover, > we hope that open-sourcing Heron will help to further evolve the technology > as the project attracts contributors with diverse backgrounds and areas of > expertise. > > We believe the Apache foundation is a great fit as the long-term home for > Heron, as it provides an established process for community-driven > development and decision making by consensus. This is exactly the model we > want for future Heron development. > > = Initial Goals = > > * Move the existing codebase, website, documentation, and mailing lists to > Apache-hosted infrastructure. > * Integrate with the Apache development process. > * Ensure all dependencies are compliant with Apache License version 2.0. > * Incrementally develop and release per Apache guidelines. > > = Current Status = > > Heron is a stable project used in production at Twitter since 2014 and open > sourced under the ASL v2 license in 2016. The Heron source code is > currently hosted at github.com (https://github.com/twitter/heron), which > will seed the Apache git repository. > > = Meritocracy = > > By submitting this incubator proposal, we’re expressing our intent to build > a diverse developer community around Heron that will conduct itself > according to The Apache Way and use a meritocratic means of building it's > committer base. Several companies and universities have already expressed > interest in and contributed to Heron. Our goal is to grow the Heron > community by encouraging open communication, contribution and participation > of all types, and ensuring that contributors are recognized appropriately. > > = Community = > > Heron is currently being used by Twitter, Google, Machine Zone and > ndustrial.io and has received significant contributions by Microsoft and > Streamlio. By bringing Heron into the Apache ecosystem, we believe we can > attract even more developers who are interested in creating real-time > systems to build the project's contributor base. > > == Core Developers == > > Current core developers are engineers from Twitter, Google, Microsoft and > Streamlio. > > == Alignment == > > Heron utilizes a number of Apache technologies. Heron leverages Apache > ZooKeeper for coordination and has scheduler implementations to integrate > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF) > as well as spout implementations to integrate with Apache Kafka and metrics > implementations to integrate with Scribe. Heron also implements the Apache > Storm user-level API, which allows topologies written against Storm to run > in Heron. We believe that having Heron at Apache will help further the > growth of the streaming compute community, as well as encourage cooperation > and developer cross pollination with other Apache projects. > > = Known Risks = > > == Orphaned Products == > > The risk of the Heron project being abandoned is minimal. It is used in > production at Twitter and Google and other companies are evaluating or > adopting it for production use. > > == Inexperience with Open Source == > > All of the core contributors to the project have considerable experience > with open source software development. Bill Graham[2], Ashvin Agrawal[3] > and Supun Kamburugamuve[4], committers on the project, are PMCs on other > Apache projects and Bill and Ashvin have gone through the Apache incubator > process. Twitter has already donated numerous projects to the ASF (e.g., > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored > by experienced ASF members that can help with any roadblocks. > > == Homogenous Developers == > > Initial committers come from 5 separate organizations. Our intention is > increase the diversity of contributing developers and their affiliations. > To date github contributions have come from approximately 50 contributors > from outside the Twitter team. > > == Reliance on Salaried Developers == > > It is expected that Heron development will occur on both salaried time and > on volunteer time. The majority of initial committers are paid by their > employers to contribute to this project. We are committed to recruiting > additional committers from other organizations as well as non-salaried > committers to join project. > > == Relationships with Other Apache Products == > > As mentioned in the Alignment section, Heron implements the Apache Storm > API and integrates with multiple Apache schedulers (Apache Mesos, Apache > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache > Thrift. > > == An Excessive Fascination with the Apache Brand == > > Heron's popularity is growing in the streaming compute space and we are > long time supporters of the Apache brand. This proposal is not for the > purpose of generating publicity through. Rather, the primary benefits to > joining Apache are those of community building and open decision making > outlined in the Rationale section. > > == Documentation == > > This proposal exists online as > http://wiki.apache.org/incubator/HeronProposal. Extensive documentation > can > be found on github at https://twitter.github.io/heron and the source code > is well documented. > > == Source and Intellectual Property Submission Plan == > > The Heron codebase is currently hosted on Github: > https://github.com/twitter/heron. During incubation, the codebase will be > migrated to Apache infrastructure. The source code is already ASF 2.0 > licensed. > > == External Dependencies == > > All external libraries have ASF 2.0 compatible licenses except for pylint. > The pylint library is GPL licensed, but is only used for pre-build Python > style checks and is neither bundled with, nor relied upon by, the Heron > source or binary release artifacts. > > == Cryptography == > > Heron does not use any cryptography libraries. > > = Required Resources = > > == Mailing lists == > > * [hidden email] (with moderated subscriptions) > * [hidden email] > * [hidden email] > * [hidden email] > > == Subversion Directory == > > Git is the preferred source control system: git://git.apache.org/heron > > == Issue Tracking == > > JIRA: Heron (HERON) > > == Initial Committers == > > * Andrew Jorgensen (andrew at andrewjorgensen dot com) > * Ashvin Agrawal (ashvin at apache dot org)* > * Avrilia Floratou (avrilia dot floratou at gmail dot com) > * Bill Graham (billgraham at apache dot org)* > * Brian Hatfield (bmhatfield at gmail dot com) > * Chris Kellogg (cckellogg at gmail dot com) > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) > * Karthik Ramasamy (karthik at gmail dot com) > * Maosong Fu (maosongfu at gmail dot com) > * Neng Lu(freeneng at gmail dot com) > * Runhang Li (obj dot runhang at gmail dot com) > * Sanjeev Kulkarni (sanjeevrk at gmail dot com) > * Supun Kamburugamuve (supun at apache dot org)* > * Thomas Sun (tom dot ssf at gmail dot com) > * Yaliang Wang (yaliang dot w dot wang at ieee dot org) > > == Affiliations == > > * Andrew Jorgensen (Google) > * Ashvin Agrawal (Microsoft) > * Avrilia Floratou (Microsoft) > * Bill Graham (Twitter) > * Brian Hatfield (Google) > * Chris Kellogg (Twitter) > * Huijun Wu (Twitter) > * Karthik Ramasamy (Streamlio) > * Maosong Fu (Twitter) > * Neng Lu (Twitter) > * Runhang Li (Twitter) > * Sanjeev Kulkarni (Streamlio) > * Supun Kamburugamuve (Indiana University) > * Thomas Sun (Twitter) > * Yaliang Wang (Twitter) > > = Sponsors = > > == Champion == > > * Julien Le Dem (julien at apache dot org) > > == Nominated Mentors == > > * Jake Farrell (jfarrell at apache dot org) > * Jacques Nadeau (jacques at apache dot org) > * Julien Le Dem (julien at apache dot org) > * P. Taylor Goetz (ptgoetz at apache dot org) > > == Sponsoring Entity == > > The Apache Incubator > > == Footnotes == > > * 1 - Papers detailing Heron are available at > http://dl.acm.org/citation.cfm?id=2742788 and > http://sites.computer.org/debull/A15dec/p15.pdf. > * 2 - http://home.apache.org/phonebook.html?uid=billgraham > * 3 - http://home.apache.org/phonebook.html?uid=ashvin > * 4 - http://home.apache.org/phonebook.html?uid=supun > |
Re: [VOTE] Heron to enter Apache Incubator
|
In reply to this post by Roy Lenferink
+1
On 2017-06-16 13:50 (-0700), Roy Lenferink <[hidden email]> wrote: > +1 > > Op vr 16 jun. 2017 om 22:49 schreef Julien Le Dem <[hidden email]> > > > +1 > > > > On Fri, Jun 16, 2017 at 1:46 PM, William Markito Oliveira < > > [hidden email]> wrote: > > > > > +1 > > > > > > On Fri, Jun 16, 2017 at 3:45 PM, Debo Dutta (dedutta) <[hidden email] > > > > > > wrote: > > > > > > > +1 > > > > > > > > On 6/16/17, 1:41 PM, "Bill Graham" <[hidden email]> wrote: > > > > > > > > Hi, > > > > > > > > Based on the discussion on the incubator mailing list[1] I would > > like > > > > to > > > > call a vote to add Heron to the Apache Incubator. > > > > > > > > The full proposal is available below, and is also available on the > > > > Apache > > > > Incubator wiki at: > > > > https://wiki.apache.org/incubator/HeronProposal > > > > > > > > Please vote: > > > > [ ] +1, bring Heron into Incubator > > > > [ ] -1, do not bring Heron into Incubator, because... > > > > > > > > The vote will open for 7 days until Friday June 23 at 14:00 PT. > > > > > > > > Thank you > > > > > > > > 1 - > > > > > > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7 > > > > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E > > > > > > > > > > > > > > > > = Heron Proposal = > > > > > > > > = Abstract = > > > > Heron is a real-time, distributed, fault-tolerant stream processing > > > > engine > > > > initially developed by Twitter. > > > > > > > > = Proposal = > > > > > > > > Heron is a real-time stream processing engine built for high > > > > performance, > > > > ease of manageability, performance predictability and developer > > > > productivity[1]. We wish to develop a community around Heron to > > > > increase > > > > contributions and see Heron thrive in an open forum. > > > > > > > > = Background = > > > > > > > > Heron provides the ability for developers to compose directed > > acyclic > > > > graphs (DAGs) of real-time query execution logic (i.e. a topology) > > > and > > > > submit the topology to execute on a pluggable job scheduling system > > > > (e.g., > > > > Apache Aurora, YARN, Marathon, etc). Users can employ either the > > > native > > > > Heron API or the Apache Storm API to develop the topology. Heron > > > > supports > > > > the Storm API for ease of migration, but beyond that Heronâs > > > > architecture > > > > differs considerably from Stormâs. > > > > > > > > Users submit a topology to the scheduler using the Heron client, > > > which > > > > uses > > > > the Heron binary libraries to deploy all daemons required to run > > and > > > > manage > > > > the topology. The topology therefore has no reliance on centrally > > > > managed > > > > Heron services, only on a generic job scheduling system, which > > lends > > > > itself > > > > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN > > > > (among > > > > others). > > > > > > > > The scheduler runs each topology as a job consisting of multiple > > > > containers. One of the containers runs the topology master, > > > > responsible for > > > > managing the topology. The remaining containers each runs a stream > > > > manager > > > > responsible for data routing, a metrics manager that collects and > > > > reports > > > > various metrics and a number of processes called Heron instances > > > which > > > > run > > > > the user-defined logic on the stream of tuples. Parallelism is > > > > achieved via > > > > process-based isolation of Heron instances, which provides > > > predictable > > > > performance while simplifying debugging. The containers are > > allocated > > > > and > > > > managed by the scheduler framework based on resource availability > > of > > > > nodes > > > > in the cluster. The metadata for the topology, such as the physical > > > > plan > > > > and execution details, are stored in the pluggable Heron State > > > Manager > > > > (e.g. Apache ZooKeeper). > > > > > > > > = Rationale = > > > > > > > > Heron is a general-purpose, modular and extensible platform that > > can > > > be > > > > leveraged to support common, real-time analytics use cases. There > > is > > > an > > > > increasing demand for open-source, scalable real-time analytics > > > > systems. We > > > > believe that Heron can be leveraged by other organizations to build > > > > streaming applications that can benefit from its robustness, high > > > > performance, adaptability to cloud environments and ease of use. > > > > Moreover, > > > > we hope that open-sourcing Heron will help to further evolve the > > > > technology > > > > as the project attracts contributors with diverse backgrounds and > > > > areas of > > > > expertise. > > > > > > > > We believe the Apache foundation is a great fit as the long-term > > home > > > > for > > > > Heron, as it provides an established process for community-driven > > > > development and decision making by consensus. This is exactly the > > > > model we > > > > want for future Heron development. > > > > > > > > = Initial Goals = > > > > > > > > * Move the existing codebase, website, documentation, and mailing > > > > lists to > > > > Apache-hosted infrastructure. > > > > * Integrate with the Apache development process. > > > > * Ensure all dependencies are compliant with Apache License > > version > > > > 2.0. > > > > * Incrementally develop and release per Apache guidelines. > > > > > > > > = Current Status = > > > > > > > > Heron is a stable project used in production at Twitter since 2014 > > > and > > > > open > > > > sourced under the ASL v2 license in 2016. The Heron source code is > > > > currently hosted at github.com (https://github.com/twitter/heron), > > > > which > > > > will seed the Apache git repository. > > > > > > > > = Meritocracy = > > > > > > > > By submitting this incubator proposal, weâre expressing our intent > > to > > > > build > > > > a diverse developer community around Heron that will conduct itself > > > > according to The Apache Way and use a meritocratic means of > > building > > > > it's > > > > committer base. Several companies and universities have already > > > > expressed > > > > interest in and contributed to Heron. Our goal is to grow the Heron > > > > community by encouraging open communication, contribution and > > > > participation > > > > of all types, and ensuring that contributors are recognized > > > > appropriately. > > > > > > > > = Community = > > > > > > > > Heron is currently being used by Twitter, Google, Machine Zone and > > > > ndustrial.io and has received significant contributions by > > Microsoft > > > > and > > > > Streamlio. By bringing Heron into the Apache ecosystem, we believe > > we > > > > can > > > > attract even more developers who are interested in creating > > real-time > > > > systems to build the project's contributor base. > > > > > > > > == Core Developers == > > > > > > > > Current core developers are engineers from Twitter, Google, > > Microsoft > > > > and > > > > Streamlio. > > > > > > > > == Alignment == > > > > > > > > Heron utilizes a number of Apache technologies. Heron leverages > > > Apache > > > > ZooKeeper for coordination and has scheduler implementations to > > > > integrate > > > > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via > > Apache > > > > REEF) > > > > as well as spout implementations to integrate with Apache Kafka and > > > > metrics > > > > implementations to integrate with Scribe. Heron also implements the > > > > Apache > > > > Storm user-level API, which allows topologies written against Storm > > > to > > > > run > > > > in Heron. We believe that having Heron at Apache will help further > > > the > > > > growth of the streaming compute community, as well as encourage > > > > cooperation > > > > and developer cross pollination with other Apache projects. > > > > > > > > = Known Risks = > > > > > > > > == Orphaned Products == > > > > > > > > The risk of the Heron project being abandoned is minimal. It is > > used > > > in > > > > production at Twitter and Google and other companies are evaluating > > > or > > > > adopting it for production use. > > > > > > > > == Inexperience with Open Source == > > > > > > > > All of the core contributors to the project have considerable > > > > experience > > > > with open source software development. Bill Graham[2], Ashvin > > > > Agrawal[3] > > > > and Supun Kamburugamuve[4], committers on the project, are PMCs on > > > > other > > > > Apache projects and Bill and Ashvin have gone through the Apache > > > > incubator > > > > process. Twitter has already donated numerous projects to the ASF > > > > (e.g., > > > > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be > > > > mentored > > > > by experienced ASF members that can help with any roadblocks. > > > > > > > > == Homogenous Developers == > > > > > > > > Initial committers come from 5 separate organizations. Our > > intention > > > is > > > > increase the diversity of contributing developers and their > > > > affiliations. > > > > To date github contributions have come from approximately 50 > > > > contributors > > > > from outside the Twitter team. > > > > > > > > == Reliance on Salaried Developers == > > > > > > > > It is expected that Heron development will occur on both salaried > > > time > > > > and > > > > on volunteer time. The majority of initial committers are paid by > > > their > > > > employers to contribute to this project. We are committed to > > > recruiting > > > > additional committers from other organizations as well as > > > non-salaried > > > > committers to join project. > > > > > > > > == Relationships with Other Apache Products == > > > > > > > > As mentioned in the Alignment section, Heron implements the Apache > > > > Storm > > > > API and integrates with multiple Apache schedulers (Apache Mesos, > > > > Apache > > > > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and > > > Apache > > > > Thrift. > > > > > > > > == An Excessive Fascination with the Apache Brand == > > > > > > > > Heron's popularity is growing in the streaming compute space and we > > > are > > > > long time supporters of the Apache brand. This proposal is not for > > > the > > > > purpose of generating publicity through. Rather, the primary > > benefits > > > > to > > > > joining Apache are those of community building and open decision > > > making > > > > outlined in the Rationale section. > > > > > > > > == Documentation == > > > > > > > > This proposal exists online as > > > > http://wiki.apache.org/incubator/HeronProposal. Extensive > > > > documentation can > > > > be found on github at https://twitter.github.io/heron and the > > source > > > > code > > > > is well documented. > > > > > > > > == Source and Intellectual Property Submission Plan == > > > > > > > > The Heron codebase is currently hosted on Github: > > > > https://github.com/twitter/heron. During incubation, the codebase > > > > will be > > > > migrated to Apache infrastructure. The source code is already ASF > > 2.0 > > > > licensed. > > > > > > > > == External Dependencies == > > > > > > > > All external libraries have ASF 2.0 compatible licenses except for > > > > pylint. > > > > The pylint library is GPL licensed, but is only used for pre-build > > > > Python > > > > style checks and is neither bundled with, nor relied upon by, the > > > Heron > > > > source or binary release artifacts. > > > > > > > > == Cryptography == > > > > > > > > Heron does not use any cryptography libraries. > > > > > > > > = Required Resources = > > > > > > > > == Mailing lists == > > > > > > > > * [hidden email] (with moderated > > subscriptions) > > > > * [hidden email] > > > > * [hidden email] > > > > * [hidden email] > > > > > > > > == Subversion Directory == > > > > > > > > Git is the preferred source control system: git:// > > > git.apache.org/heron > > > > > > > > == Issue Tracking == > > > > > > > > JIRA: Heron (HERON) > > > > > > > > == Initial Committers == > > > > > > > > * Andrew Jorgensen (andrew at andrewjorgensen dot com) > > > > * Ashvin Agrawal (ashvin at apache dot org)* > > > > * Avrilia Floratou (avrilia dot floratou at gmail dot com) > > > > * Bill Graham (billgraham at apache dot org)* > > > > * Brian Hatfield (bmhatfield at gmail dot com) > > > > * Chris Kellogg (cckellogg at gmail dot com) > > > > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) > > > > * Karthik Ramasamy (karthik at gmail dot com) > > > > * Maosong Fu (maosongfu at gmail dot com) > > > > * Neng Lu(freeneng at gmail dot com) > > > > * Runhang Li (obj dot runhang at gmail dot com) > > > > * Sanjeev Kulkarni (sanjeevrk at gmail dot com) > > > > * Supun Kamburugamuve (supun at apache dot org)* > > > > * Thomas Sun (tom dot ssf at gmail dot com) > > > > * Yaliang Wang (yaliang dot w dot wang at ieee dot org) > > > > > > > > == Affiliations == > > > > > > > > * Andrew Jorgensen (Google) > > > > * Ashvin Agrawal (Microsoft) > > > > * Avrilia Floratou (Microsoft) > > > > * Bill Graham (Twitter) > > > > * Brian Hatfield (Google) > > > > * Chris Kellogg (Twitter) > > > > * Huijun Wu (Twitter) > > > > * Karthik Ramasamy (Streamlio) > > > > * Maosong Fu (Twitter) > > > > * Neng Lu (Twitter) > > > > * Runhang Li (Twitter) > > > > * Sanjeev Kulkarni (Streamlio) > > > > * Supun Kamburugamuve (Indiana University) > > > > * Thomas Sun (Twitter) > > > > * Yaliang Wang (Twitter) > > > > > > > > = Sponsors = > > > > > > > > == Champion == > > > > > > > > * Julien Le Dem (julien at apache dot org) > > > > > > > > == Nominated Mentors == > > > > > > > > * Jake Farrell (jfarrell at apache dot org) > > > > * Jacques Nadeau (jacques at apache dot org) > > > > * Julien Le Dem (julien at apache dot org) > > > > * P. Taylor Goetz (ptgoetz at apache dot org) > > > > > > > > == Sponsoring Entity == > > > > > > > > The Apache Incubator > > > > > > > > == Footnotes == > > > > > > > > * 1 - Papers detailing Heron are available at > > > > http://dl.acm.org/citation.cfm?id=2742788 and > > > > http://sites.computer.org/debull/A15dec/p15.pdf. > > > > * 2 - http://home.apache.org/phonebook.html?uid=billgraham > > > > * 3 - http://home.apache.org/phonebook.html?uid=ashvin > > > > * 4 - http://home.apache.org/phonebook.html?uid=supun > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
Re: [VOTE] Heron to enter Apache Incubator
|
In reply to this post by Bill Graham-4
+1 (non-binding)
On Fri, Jun 16, 2017 at 3:41 PM, Bill Graham <[hidden email]> wrote: > Hi, > > Based on the discussion on the incubator mailing list[1] I would like to > call a vote to add Heron to the Apache Incubator. > > The full proposal is available below, and is also available on the Apache > Incubator wiki at: > https://wiki.apache.org/incubator/HeronProposal > > Please vote: > [ ] +1, bring Heron into Incubator > [ ] -1, do not bring Heron into Incubator, because... > > The vote will open for 7 days until Friday June 23 at 14:00 PT. > > Thank you > > 1 - > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7 > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E > > > > = Heron Proposal = > > = Abstract = > Heron is a real-time, distributed, fault-tolerant stream processing engine > initially developed by Twitter. > > = Proposal = > > Heron is a real-time stream processing engine built for high performance, > ease of manageability, performance predictability and developer > productivity[1]. We wish to develop a community around Heron to increase > contributions and see Heron thrive in an open forum. > > = Background = > > Heron provides the ability for developers to compose directed acyclic > graphs (DAGs) of real-time query execution logic (i.e. a topology) and > submit the topology to execute on a pluggable job scheduling system (e.g., > Apache Aurora, YARN, Marathon, etc). Users can employ either the native > Heron API or the Apache Storm API to develop the topology. Heron supports > the Storm API for ease of migration, but beyond that Heron’s architecture > differs considerably from Storm’s. > > Users submit a topology to the scheduler using the Heron client, which uses > the Heron binary libraries to deploy all daemons required to run and manage > the topology. The topology therefore has no reliance on centrally managed > Heron services, only on a generic job scheduling system, which lends itself > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among > others). > > The scheduler runs each topology as a job consisting of multiple > containers. One of the containers runs the topology master, responsible for > managing the topology. The remaining containers each runs a stream manager > responsible for data routing, a metrics manager that collects and reports > various metrics and a number of processes called Heron instances which run > the user-defined logic on the stream of tuples. Parallelism is achieved via > process-based isolation of Heron instances, which provides predictable > performance while simplifying debugging. The containers are allocated and > managed by the scheduler framework based on resource availability of nodes > in the cluster. The metadata for the topology, such as the physical plan > and execution details, are stored in the pluggable Heron State Manager > (e.g. Apache ZooKeeper). > > = Rationale = > > Heron is a general-purpose, modular and extensible platform that can be > leveraged to support common, real-time analytics use cases. There is an > increasing demand for open-source, scalable real-time analytics systems. We > believe that Heron can be leveraged by other organizations to build > streaming applications that can benefit from its robustness, high > performance, adaptability to cloud environments and ease of use. Moreover, > we hope that open-sourcing Heron will help to further evolve the technology > as the project attracts contributors with diverse backgrounds and areas of > expertise. > > We believe the Apache foundation is a great fit as the long-term home for > Heron, as it provides an established process for community-driven > development and decision making by consensus. This is exactly the model we > want for future Heron development. > > = Initial Goals = > > * Move the existing codebase, website, documentation, and mailing lists to > Apache-hosted infrastructure. > * Integrate with the Apache development process. > * Ensure all dependencies are compliant with Apache License version 2.0. > * Incrementally develop and release per Apache guidelines. > > = Current Status = > > Heron is a stable project used in production at Twitter since 2014 and open > sourced under the ASL v2 license in 2016. The Heron source code is > currently hosted at github.com (https://github.com/twitter/heron), which > will seed the Apache git repository. > > = Meritocracy = > > By submitting this incubator proposal, we’re expressing our intent to build > a diverse developer community around Heron that will conduct itself > according to The Apache Way and use a meritocratic means of building it's > committer base. Several companies and universities have already expressed > interest in and contributed to Heron. Our goal is to grow the Heron > community by encouraging open communication, contribution and participation > of all types, and ensuring that contributors are recognized appropriately. > > = Community = > > Heron is currently being used by Twitter, Google, Machine Zone and > ndustrial.io and has received significant contributions by Microsoft and > Streamlio. By bringing Heron into the Apache ecosystem, we believe we can > attract even more developers who are interested in creating real-time > systems to build the project's contributor base. > > == Core Developers == > > Current core developers are engineers from Twitter, Google, Microsoft and > Streamlio. > > == Alignment == > > Heron utilizes a number of Apache technologies. Heron leverages Apache > ZooKeeper for coordination and has scheduler implementations to integrate > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF) > as well as spout implementations to integrate with Apache Kafka and metrics > implementations to integrate with Scribe. Heron also implements the Apache > Storm user-level API, which allows topologies written against Storm to run > in Heron. We believe that having Heron at Apache will help further the > growth of the streaming compute community, as well as encourage cooperation > and developer cross pollination with other Apache projects. > > = Known Risks = > > == Orphaned Products == > > The risk of the Heron project being abandoned is minimal. It is used in > production at Twitter and Google and other companies are evaluating or > adopting it for production use. > > == Inexperience with Open Source == > > All of the core contributors to the project have considerable experience > with open source software development. Bill Graham[2], Ashvin Agrawal[3] > and Supun Kamburugamuve[4], committers on the project, are PMCs on other > Apache projects and Bill and Ashvin have gone through the Apache incubator > process. Twitter has already donated numerous projects to the ASF (e.g., > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored > by experienced ASF members that can help with any roadblocks. > > == Homogenous Developers == > > Initial committers come from 5 separate organizations. Our intention is > increase the diversity of contributing developers and their affiliations. > To date github contributions have come from approximately 50 contributors > from outside the Twitter team. > > == Reliance on Salaried Developers == > > It is expected that Heron development will occur on both salaried time and > on volunteer time. The majority of initial committers are paid by their > employers to contribute to this project. We are committed to recruiting > additional committers from other organizations as well as non-salaried > committers to join project. > > == Relationships with Other Apache Products == > > As mentioned in the Alignment section, Heron implements the Apache Storm > API and integrates with multiple Apache schedulers (Apache Mesos, Apache > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache > Thrift. > > == An Excessive Fascination with the Apache Brand == > > Heron's popularity is growing in the streaming compute space and we are > long time supporters of the Apache brand. This proposal is not for the > purpose of generating publicity through. Rather, the primary benefits to > joining Apache are those of community building and open decision making > outlined in the Rationale section. > > == Documentation == > > This proposal exists online as > http://wiki.apache.org/incubator/HeronProposal. Extensive documentation > can > be found on github at https://twitter.github.io/heron and the source code > is well documented. > > == Source and Intellectual Property Submission Plan == > > The Heron codebase is currently hosted on Github: > https://github.com/twitter/heron. During incubation, the codebase will be > migrated to Apache infrastructure. The source code is already ASF 2.0 > licensed. > > == External Dependencies == > > All external libraries have ASF 2.0 compatible licenses except for pylint. > The pylint library is GPL licensed, but is only used for pre-build Python > style checks and is neither bundled with, nor relied upon by, the Heron > source or binary release artifacts. > > == Cryptography == > > Heron does not use any cryptography libraries. > > = Required Resources = > > == Mailing lists == > > * [hidden email] (with moderated subscriptions) > * [hidden email] > * [hidden email] > * [hidden email] > > == Subversion Directory == > > Git is the preferred source control system: git://git.apache.org/heron > > == Issue Tracking == > > JIRA: Heron (HERON) > > == Initial Committers == > > * Andrew Jorgensen (andrew at andrewjorgensen dot com) > * Ashvin Agrawal (ashvin at apache dot org)* > * Avrilia Floratou (avrilia dot floratou at gmail dot com) > * Bill Graham (billgraham at apache dot org)* > * Brian Hatfield (bmhatfield at gmail dot com) > * Chris Kellogg (cckellogg at gmail dot com) > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) > * Karthik Ramasamy (karthik at gmail dot com) > * Maosong Fu (maosongfu at gmail dot com) > * Neng Lu(freeneng at gmail dot com) > * Runhang Li (obj dot runhang at gmail dot com) > * Sanjeev Kulkarni (sanjeevrk at gmail dot com) > * Supun Kamburugamuve (supun at apache dot org)* > * Thomas Sun (tom dot ssf at gmail dot com) > * Yaliang Wang (yaliang dot w dot wang at ieee dot org) > > == Affiliations == > > * Andrew Jorgensen (Google) > * Ashvin Agrawal (Microsoft) > * Avrilia Floratou (Microsoft) > * Bill Graham (Twitter) > * Brian Hatfield (Google) > * Chris Kellogg (Twitter) > * Huijun Wu (Twitter) > * Karthik Ramasamy (Streamlio) > * Maosong Fu (Twitter) > * Neng Lu (Twitter) > * Runhang Li (Twitter) > * Sanjeev Kulkarni (Streamlio) > * Supun Kamburugamuve (Indiana University) > * Thomas Sun (Twitter) > * Yaliang Wang (Twitter) > > = Sponsors = > > == Champion == > > * Julien Le Dem (julien at apache dot org) > > == Nominated Mentors == > > * Jake Farrell (jfarrell at apache dot org) > * Jacques Nadeau (jacques at apache dot org) > * Julien Le Dem (julien at apache dot org) > * P. Taylor Goetz (ptgoetz at apache dot org) > > == Sponsoring Entity == > > The Apache Incubator > > == Footnotes == > > * 1 - Papers detailing Heron are available at > http://dl.acm.org/citation.cfm?id=2742788 and > http://sites.computer.org/debull/A15dec/p15.pdf. > * 2 - http://home.apache.org/phonebook.html?uid=billgraham > * 3 - http://home.apache.org/phonebook.html?uid=ashvin > * 4 - http://home.apache.org/phonebook.html?uid=supun > -- Cheers, Chris Aniszczyk http://aniszczyk.org +1 512 961 6719 |
Re: [VOTE] Heron to enter Apache Incubator
|
In reply to this post by Sanjeev Kulkarni
+1
Thanks, Supun.. On Fri, Jun 16, 2017 at 4:59 PM, Sanjeev Kulkarni <[hidden email]> wrote: > +1 > > On Fri, Jun 16, 2017 at 1:41 PM, Bill Graham <[hidden email]> wrote: > > > Hi, > > > > Based on the discussion on the incubator mailing list[1] I would like to > > call a vote to add Heron to the Apache Incubator. > > > > The full proposal is available below, and is also available on the Apache > > Incubator wiki at: > > https://wiki.apache.org/incubator/HeronProposal > > > > Please vote: > > [ ] +1, bring Heron into Incubator > > [ ] -1, do not bring Heron into Incubator, because... > > > > The vote will open for 7 days until Friday June 23 at 14:00 PT. > > > > Thank you > > > > 1 - > > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7 > > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E > > > > > > > > = Heron Proposal = > > > > = Abstract = > > Heron is a real-time, distributed, fault-tolerant stream processing > engine > > initially developed by Twitter. > > > > = Proposal = > > > > Heron is a real-time stream processing engine built for high performance, > > ease of manageability, performance predictability and developer > > productivity[1]. We wish to develop a community around Heron to increase > > contributions and see Heron thrive in an open forum. > > > > = Background = > > > > Heron provides the ability for developers to compose directed acyclic > > graphs (DAGs) of real-time query execution logic (i.e. a topology) and > > submit the topology to execute on a pluggable job scheduling system > (e.g., > > Apache Aurora, YARN, Marathon, etc). Users can employ either the native > > Heron API or the Apache Storm API to develop the topology. Heron supports > > the Storm API for ease of migration, but beyond that Heron’s architecture > > differs considerably from Storm’s. > > > > Users submit a topology to the scheduler using the Heron client, which > uses > > the Heron binary libraries to deploy all daemons required to run and > manage > > the topology. The topology therefore has no reliance on centrally managed > > Heron services, only on a generic job scheduling system, which lends > itself > > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among > > others). > > > > The scheduler runs each topology as a job consisting of multiple > > containers. One of the containers runs the topology master, responsible > for > > managing the topology. The remaining containers each runs a stream > manager > > responsible for data routing, a metrics manager that collects and reports > > various metrics and a number of processes called Heron instances which > run > > the user-defined logic on the stream of tuples. Parallelism is achieved > via > > process-based isolation of Heron instances, which provides predictable > > performance while simplifying debugging. The containers are allocated and > > managed by the scheduler framework based on resource availability of > nodes > > in the cluster. The metadata for the topology, such as the physical plan > > and execution details, are stored in the pluggable Heron State Manager > > (e.g. Apache ZooKeeper). > > > > = Rationale = > > > > Heron is a general-purpose, modular and extensible platform that can be > > leveraged to support common, real-time analytics use cases. There is an > > increasing demand for open-source, scalable real-time analytics systems. > We > > believe that Heron can be leveraged by other organizations to build > > streaming applications that can benefit from its robustness, high > > performance, adaptability to cloud environments and ease of use. > Moreover, > > we hope that open-sourcing Heron will help to further evolve the > technology > > as the project attracts contributors with diverse backgrounds and areas > of > > expertise. > > > > We believe the Apache foundation is a great fit as the long-term home for > > Heron, as it provides an established process for community-driven > > development and decision making by consensus. This is exactly the model > we > > want for future Heron development. > > > > = Initial Goals = > > > > * Move the existing codebase, website, documentation, and mailing lists > to > > Apache-hosted infrastructure. > > * Integrate with the Apache development process. > > * Ensure all dependencies are compliant with Apache License version 2.0. > > * Incrementally develop and release per Apache guidelines. > > > > = Current Status = > > > > Heron is a stable project used in production at Twitter since 2014 and > open > > sourced under the ASL v2 license in 2016. The Heron source code is > > currently hosted at github.com (https://github.com/twitter/heron), which > > will seed the Apache git repository. > > > > = Meritocracy = > > > > By submitting this incubator proposal, we’re expressing our intent to > build > > a diverse developer community around Heron that will conduct itself > > according to The Apache Way and use a meritocratic means of building it's > > committer base. Several companies and universities have already expressed > > interest in and contributed to Heron. Our goal is to grow the Heron > > community by encouraging open communication, contribution and > participation > > of all types, and ensuring that contributors are recognized > appropriately. > > > > = Community = > > > > Heron is currently being used by Twitter, Google, Machine Zone and > > ndustrial.io and has received significant contributions by Microsoft and > > Streamlio. By bringing Heron into the Apache ecosystem, we believe we can > > attract even more developers who are interested in creating real-time > > systems to build the project's contributor base. > > > > == Core Developers == > > > > Current core developers are engineers from Twitter, Google, Microsoft and > > Streamlio. > > > > == Alignment == > > > > Heron utilizes a number of Apache technologies. Heron leverages Apache > > ZooKeeper for coordination and has scheduler implementations to integrate > > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache > REEF) > > as well as spout implementations to integrate with Apache Kafka and > metrics > > implementations to integrate with Scribe. Heron also implements the > Apache > > Storm user-level API, which allows topologies written against Storm to > run > > in Heron. We believe that having Heron at Apache will help further the > > growth of the streaming compute community, as well as encourage > cooperation > > and developer cross pollination with other Apache projects. > > > > = Known Risks = > > > > == Orphaned Products == > > > > The risk of the Heron project being abandoned is minimal. It is used in > > production at Twitter and Google and other companies are evaluating or > > adopting it for production use. > > > > == Inexperience with Open Source == > > > > All of the core contributors to the project have considerable experience > > with open source software development. Bill Graham[2], Ashvin Agrawal[3] > > and Supun Kamburugamuve[4], committers on the project, are PMCs on other > > Apache projects and Bill and Ashvin have gone through the Apache > incubator > > process. Twitter has already donated numerous projects to the ASF (e.g., > > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored > > by experienced ASF members that can help with any roadblocks. > > > > == Homogenous Developers == > > > > Initial committers come from 5 separate organizations. Our intention is > > increase the diversity of contributing developers and their affiliations. > > To date github contributions have come from approximately 50 contributors > > from outside the Twitter team. > > > > == Reliance on Salaried Developers == > > > > It is expected that Heron development will occur on both salaried time > and > > on volunteer time. The majority of initial committers are paid by their > > employers to contribute to this project. We are committed to recruiting > > additional committers from other organizations as well as non-salaried > > committers to join project. > > > > == Relationships with Other Apache Products == > > > > As mentioned in the Alignment section, Heron implements the Apache Storm > > API and integrates with multiple Apache schedulers (Apache Mesos, Apache > > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache > > Thrift. > > > > == An Excessive Fascination with the Apache Brand == > > > > Heron's popularity is growing in the streaming compute space and we are > > long time supporters of the Apache brand. This proposal is not for the > > purpose of generating publicity through. Rather, the primary benefits to > > joining Apache are those of community building and open decision making > > outlined in the Rationale section. > > > > == Documentation == > > > > This proposal exists online as > > http://wiki.apache.org/incubator/HeronProposal. Extensive documentation > > can > > be found on github at https://twitter.github.io/heron and the source > code > > is well documented. > > > > == Source and Intellectual Property Submission Plan == > > > > The Heron codebase is currently hosted on Github: > > https://github.com/twitter/heron. During incubation, the codebase will > be > > migrated to Apache infrastructure. The source code is already ASF 2.0 > > licensed. > > > > == External Dependencies == > > > > All external libraries have ASF 2.0 compatible licenses except for > pylint. > > The pylint library is GPL licensed, but is only used for pre-build Python > > style checks and is neither bundled with, nor relied upon by, the Heron > > source or binary release artifacts. > > > > == Cryptography == > > > > Heron does not use any cryptography libraries. > > > > = Required Resources = > > > > == Mailing lists == > > > > * [hidden email] (with moderated subscriptions) > > * [hidden email] > > * [hidden email] > > * [hidden email] > > > > == Subversion Directory == > > > > Git is the preferred source control system: git://git.apache.org/heron > > > > == Issue Tracking == > > > > JIRA: Heron (HERON) > > > > == Initial Committers == > > > > * Andrew Jorgensen (andrew at andrewjorgensen dot com) > > * Ashvin Agrawal (ashvin at apache dot org)* > > * Avrilia Floratou (avrilia dot floratou at gmail dot com) > > * Bill Graham (billgraham at apache dot org)* > > * Brian Hatfield (bmhatfield at gmail dot com) > > * Chris Kellogg (cckellogg at gmail dot com) > > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) > > * Karthik Ramasamy (karthik at gmail dot com) > > * Maosong Fu (maosongfu at gmail dot com) > > * Neng Lu(freeneng at gmail dot com) > > * Runhang Li (obj dot runhang at gmail dot com) > > * Sanjeev Kulkarni (sanjeevrk at gmail dot com) > > * Supun Kamburugamuve (supun at apache dot org)* > > * Thomas Sun (tom dot ssf at gmail dot com) > > * Yaliang Wang (yaliang dot w dot wang at ieee dot org) > > > > == Affiliations == > > > > * Andrew Jorgensen (Google) > > * Ashvin Agrawal (Microsoft) > > * Avrilia Floratou (Microsoft) > > * Bill Graham (Twitter) > > * Brian Hatfield (Google) > > * Chris Kellogg (Twitter) > > * Huijun Wu (Twitter) > > * Karthik Ramasamy (Streamlio) > > * Maosong Fu (Twitter) > > * Neng Lu (Twitter) > > * Runhang Li (Twitter) > > * Sanjeev Kulkarni (Streamlio) > > * Supun Kamburugamuve (Indiana University) > > * Thomas Sun (Twitter) > > * Yaliang Wang (Twitter) > > > > = Sponsors = > > > > == Champion == > > > > * Julien Le Dem (julien at apache dot org) > > > > == Nominated Mentors == > > > > * Jake Farrell (jfarrell at apache dot org) > > * Jacques Nadeau (jacques at apache dot org) > > * Julien Le Dem (julien at apache dot org) > > * P. Taylor Goetz (ptgoetz at apache dot org) > > > > == Sponsoring Entity == > > > > The Apache Incubator > > > > == Footnotes == > > > > * 1 - Papers detailing Heron are available at > > http://dl.acm.org/citation.cfm?id=2742788 and > > http://sites.computer.org/debull/A15dec/p15.pdf. > > * 2 - http://home.apache.org/phonebook.html?uid=billgraham > > * 3 - http://home.apache.org/phonebook.html?uid=ashvin > > * 4 - http://home.apache.org/phonebook.html?uid=supun > > > -- Tech Lead, WSO2 Inc http://wso2.org supunk.blogspot.com |
|
+1 (non-binding)
On Fri, Jun 16, 2017 at 2:02 PM, Supun Kamburugamuva <[hidden email]> wrote: > +1 > > Thanks, > Supun.. > > On Fri, Jun 16, 2017 at 4:59 PM, Sanjeev Kulkarni <[hidden email]> > wrote: > > > +1 > > > > On Fri, Jun 16, 2017 at 1:41 PM, Bill Graham <[hidden email]> > wrote: > > > > > Hi, > > > > > > Based on the discussion on the incubator mailing list[1] I would like > to > > > call a vote to add Heron to the Apache Incubator. > > > > > > The full proposal is available below, and is also available on the > Apache > > > Incubator wiki at: > > > https://wiki.apache.org/incubator/HeronProposal > > > > > > Please vote: > > > [ ] +1, bring Heron into Incubator > > > [ ] -1, do not bring Heron into Incubator, because... > > > > > > The vote will open for 7 days until Friday June 23 at 14:00 PT. > > > > > > Thank you > > > > > > 1 - > > > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7 > > > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E > > > > > > > > > > > > = Heron Proposal = > > > > > > = Abstract = > > > Heron is a real-time, distributed, fault-tolerant stream processing > > engine > > > initially developed by Twitter. > > > > > > = Proposal = > > > > > > Heron is a real-time stream processing engine built for high > performance, > > > ease of manageability, performance predictability and developer > > > productivity[1]. We wish to develop a community around Heron to > increase > > > contributions and see Heron thrive in an open forum. > > > > > > = Background = > > > > > > Heron provides the ability for developers to compose directed acyclic > > > graphs (DAGs) of real-time query execution logic (i.e. a topology) and > > > submit the topology to execute on a pluggable job scheduling system > > (e.g., > > > Apache Aurora, YARN, Marathon, etc). Users can employ either the native > > > Heron API or the Apache Storm API to develop the topology. Heron > supports > > > the Storm API for ease of migration, but beyond that Heron’s > architecture > > > differs considerably from Storm’s. > > > > > > Users submit a topology to the scheduler using the Heron client, which > > uses > > > the Heron binary libraries to deploy all daemons required to run and > > manage > > > the topology. The topology therefore has no reliance on centrally > managed > > > Heron services, only on a generic job scheduling system, which lends > > itself > > > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN > (among > > > others). > > > > > > The scheduler runs each topology as a job consisting of multiple > > > containers. One of the containers runs the topology master, responsible > > for > > > managing the topology. The remaining containers each runs a stream > > manager > > > responsible for data routing, a metrics manager that collects and > reports > > > various metrics and a number of processes called Heron instances which > > run > > > the user-defined logic on the stream of tuples. Parallelism is achieved > > via > > > process-based isolation of Heron instances, which provides predictable > > > performance while simplifying debugging. The containers are allocated > and > > > managed by the scheduler framework based on resource availability of > > nodes > > > in the cluster. The metadata for the topology, such as the physical > plan > > > and execution details, are stored in the pluggable Heron State Manager > > > (e.g. Apache ZooKeeper). > > > > > > = Rationale = > > > > > > Heron is a general-purpose, modular and extensible platform that can be > > > leveraged to support common, real-time analytics use cases. There is an > > > increasing demand for open-source, scalable real-time analytics > systems. > > We > > > believe that Heron can be leveraged by other organizations to build > > > streaming applications that can benefit from its robustness, high > > > performance, adaptability to cloud environments and ease of use. > > Moreover, > > > we hope that open-sourcing Heron will help to further evolve the > > technology > > > as the project attracts contributors with diverse backgrounds and areas > > of > > > expertise. > > > > > > We believe the Apache foundation is a great fit as the long-term home > for > > > Heron, as it provides an established process for community-driven > > > development and decision making by consensus. This is exactly the model > > we > > > want for future Heron development. > > > > > > = Initial Goals = > > > > > > * Move the existing codebase, website, documentation, and mailing > lists > > to > > > Apache-hosted infrastructure. > > > * Integrate with the Apache development process. > > > * Ensure all dependencies are compliant with Apache License version > 2.0. > > > * Incrementally develop and release per Apache guidelines. > > > > > > = Current Status = > > > > > > Heron is a stable project used in production at Twitter since 2014 and > > open > > > sourced under the ASL v2 license in 2016. The Heron source code is > > > currently hosted at github.com (https://github.com/twitter/heron), > which > > > will seed the Apache git repository. > > > > > > = Meritocracy = > > > > > > By submitting this incubator proposal, we’re expressing our intent to > > build > > > a diverse developer community around Heron that will conduct itself > > > according to The Apache Way and use a meritocratic means of building > it's > > > committer base. Several companies and universities have already > expressed > > > interest in and contributed to Heron. Our goal is to grow the Heron > > > community by encouraging open communication, contribution and > > participation > > > of all types, and ensuring that contributors are recognized > > appropriately. > > > > > > = Community = > > > > > > Heron is currently being used by Twitter, Google, Machine Zone and > > > ndustrial.io and has received significant contributions by Microsoft > and > > > Streamlio. By bringing Heron into the Apache ecosystem, we believe we > can > > > attract even more developers who are interested in creating real-time > > > systems to build the project's contributor base. > > > > > > == Core Developers == > > > > > > Current core developers are engineers from Twitter, Google, Microsoft > and > > > Streamlio. > > > > > > == Alignment == > > > > > > Heron utilizes a number of Apache technologies. Heron leverages Apache > > > ZooKeeper for coordination and has scheduler implementations to > integrate > > > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache > > REEF) > > > as well as spout implementations to integrate with Apache Kafka and > > metrics > > > implementations to integrate with Scribe. Heron also implements the > > Apache > > > Storm user-level API, which allows topologies written against Storm to > > run > > > in Heron. We believe that having Heron at Apache will help further the > > > growth of the streaming compute community, as well as encourage > > cooperation > > > and developer cross pollination with other Apache projects. > > > > > > = Known Risks = > > > > > > == Orphaned Products == > > > > > > The risk of the Heron project being abandoned is minimal. It is used in > > > production at Twitter and Google and other companies are evaluating or > > > adopting it for production use. > > > > > > == Inexperience with Open Source == > > > > > > All of the core contributors to the project have considerable > experience > > > with open source software development. Bill Graham[2], Ashvin > Agrawal[3] > > > and Supun Kamburugamuve[4], committers on the project, are PMCs on > other > > > Apache projects and Bill and Ashvin have gone through the Apache > > incubator > > > process. Twitter has already donated numerous projects to the ASF > (e.g., > > > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be > mentored > > > by experienced ASF members that can help with any roadblocks. > > > > > > == Homogenous Developers == > > > > > > Initial committers come from 5 separate organizations. Our intention is > > > increase the diversity of contributing developers and their > affiliations. > > > To date github contributions have come from approximately 50 > contributors > > > from outside the Twitter team. > > > > > > == Reliance on Salaried Developers == > > > > > > It is expected that Heron development will occur on both salaried time > > and > > > on volunteer time. The majority of initial committers are paid by their > > > employers to contribute to this project. We are committed to recruiting > > > additional committers from other organizations as well as non-salaried > > > committers to join project. > > > > > > == Relationships with Other Apache Products == > > > > > > As mentioned in the Alignment section, Heron implements the Apache > Storm > > > API and integrates with multiple Apache schedulers (Apache Mesos, > Apache > > > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache > > > Thrift. > > > > > > == An Excessive Fascination with the Apache Brand == > > > > > > Heron's popularity is growing in the streaming compute space and we are > > > long time supporters of the Apache brand. This proposal is not for the > > > purpose of generating publicity through. Rather, the primary benefits > to > > > joining Apache are those of community building and open decision making > > > outlined in the Rationale section. > > > > > > == Documentation == > > > > > > This proposal exists online as > > > http://wiki.apache.org/incubator/HeronProposal. Extensive > documentation > > > can > > > be found on github at https://twitter.github.io/heron and the source > > code > > > is well documented. > > > > > > == Source and Intellectual Property Submission Plan == > > > > > > The Heron codebase is currently hosted on Github: > > > https://github.com/twitter/heron. During incubation, the codebase will > > be > > > migrated to Apache infrastructure. The source code is already ASF 2.0 > > > licensed. > > > > > > == External Dependencies == > > > > > > All external libraries have ASF 2.0 compatible licenses except for > > pylint. > > > The pylint library is GPL licensed, but is only used for pre-build > Python > > > style checks and is neither bundled with, nor relied upon by, the Heron > > > source or binary release artifacts. > > > > > > == Cryptography == > > > > > > Heron does not use any cryptography libraries. > > > > > > = Required Resources = > > > > > > == Mailing lists == > > > > > > * [hidden email] (with moderated subscriptions) > > > * [hidden email] > > > * [hidden email] > > > * [hidden email] > > > > > > == Subversion Directory == > > > > > > Git is the preferred source control system: git://git.apache.org/heron > > > > > > == Issue Tracking == > > > > > > JIRA: Heron (HERON) > > > > > > == Initial Committers == > > > > > > * Andrew Jorgensen (andrew at andrewjorgensen dot com) > > > * Ashvin Agrawal (ashvin at apache dot org)* > > > * Avrilia Floratou (avrilia dot floratou at gmail dot com) > > > * Bill Graham (billgraham at apache dot org)* > > > * Brian Hatfield (bmhatfield at gmail dot com) > > > * Chris Kellogg (cckellogg at gmail dot com) > > > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) > > > * Karthik Ramasamy (karthik at gmail dot com) > > > * Maosong Fu (maosongfu at gmail dot com) > > > * Neng Lu(freeneng at gmail dot com) > > > * Runhang Li (obj dot runhang at gmail dot com) > > > * Sanjeev Kulkarni (sanjeevrk at gmail dot com) > > > * Supun Kamburugamuve (supun at apache dot org)* > > > * Thomas Sun (tom dot ssf at gmail dot com) > > > * Yaliang Wang (yaliang dot w dot wang at ieee dot org) > > > > > > == Affiliations == > > > > > > * Andrew Jorgensen (Google) > > > * Ashvin Agrawal (Microsoft) > > > * Avrilia Floratou (Microsoft) > > > * Bill Graham (Twitter) > > > * Brian Hatfield (Google) > > > * Chris Kellogg (Twitter) > > > * Huijun Wu (Twitter) > > > * Karthik Ramasamy (Streamlio) > > > * Maosong Fu (Twitter) > > > * Neng Lu (Twitter) > > > * Runhang Li (Twitter) > > > * Sanjeev Kulkarni (Streamlio) > > > * Supun Kamburugamuve (Indiana University) > > > * Thomas Sun (Twitter) > > > * Yaliang Wang (Twitter) > > > > > > = Sponsors = > > > > > > == Champion == > > > > > > * Julien Le Dem (julien at apache dot org) > > > > > > == Nominated Mentors == > > > > > > * Jake Farrell (jfarrell at apache dot org) > > > * Jacques Nadeau (jacques at apache dot org) > > > * Julien Le Dem (julien at apache dot org) > > > * P. Taylor Goetz (ptgoetz at apache dot org) > > > > > > == Sponsoring Entity == > > > > > > The Apache Incubator > > > > > > == Footnotes == > > > > > > * 1 - Papers detailing Heron are available at > > > http://dl.acm.org/citation.cfm?id=2742788 and > > > http://sites.computer.org/debull/A15dec/p15.pdf. > > > * 2 - http://home.apache.org/phonebook.html?uid=billgraham > > > * 3 - http://home.apache.org/phonebook.html?uid=ashvin > > > * 4 - http://home.apache.org/phonebook.html?uid=supun > > > > > > > > > -- > Tech Lead, WSO2 Inc > http://wso2.org > supunk.blogspot.com > |
Re: [VOTE] Heron to enter Apache Incubator
|
|
In reply to this post by Bill Graham-4
+1 (non-binding)
On 2017-06-16 13:41 (-0700), Bill Graham <[hidden email]> wrote: > Hi, > > Based on the discussion on the incubator mailing list[1] I would like to > call a vote to add Heron to the Apache Incubator. > > The full proposal is available below, and is also available on the Apache > Incubator wiki at: > https://wiki.apache.org/incubator/HeronProposal > > Please vote: > [ ] +1, bring Heron into Incubator > [ ] -1, do not bring Heron into Incubator, because... > > The vote will open for 7 days until Friday June 23 at 14:00 PT. > > Thank you > > 1 - > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E > > > > = Heron Proposal = > > = Abstract = > Heron is a real-time, distributed, fault-tolerant stream processing engine > initially developed by Twitter. > > = Proposal = > > Heron is a real-time stream processing engine built for high performance, > ease of manageability, performance predictability and developer > productivity[1]. We wish to develop a community around Heron to increase > contributions and see Heron thrive in an open forum. > > = Background = > > Heron provides the ability for developers to compose directed acyclic > graphs (DAGs) of real-time query execution logic (i.e. a topology) and > submit the topology to execute on a pluggable job scheduling system (e.g., > Apache Aurora, YARN, Marathon, etc). Users can employ either the native > Heron API or the Apache Storm API to develop the topology. Heron supports > the Storm API for ease of migration, but beyond that Heronâs architecture > differs considerably from Stormâs. > > Users submit a topology to the scheduler using the Heron client, which uses > the Heron binary libraries to deploy all daemons required to run and manage > the topology. The topology therefore has no reliance on centrally managed > Heron services, only on a generic job scheduling system, which lends itself > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among > others). > > The scheduler runs each topology as a job consisting of multiple > containers. One of the containers runs the topology master, responsible for > managing the topology. The remaining containers each runs a stream manager > responsible for data routing, a metrics manager that collects and reports > various metrics and a number of processes called Heron instances which run > the user-defined logic on the stream of tuples. Parallelism is achieved via > process-based isolation of Heron instances, which provides predictable > performance while simplifying debugging. The containers are allocated and > managed by the scheduler framework based on resource availability of nodes > in the cluster. The metadata for the topology, such as the physical plan > and execution details, are stored in the pluggable Heron State Manager > (e.g. Apache ZooKeeper). > > = Rationale = > > Heron is a general-purpose, modular and extensible platform that can be > leveraged to support common, real-time analytics use cases. There is an > increasing demand for open-source, scalable real-time analytics systems. We > believe that Heron can be leveraged by other organizations to build > streaming applications that can benefit from its robustness, high > performance, adaptability to cloud environments and ease of use. Moreover, > we hope that open-sourcing Heron will help to further evolve the technology > as the project attracts contributors with diverse backgrounds and areas of > expertise. > > We believe the Apache foundation is a great fit as the long-term home for > Heron, as it provides an established process for community-driven > development and decision making by consensus. This is exactly the model we > want for future Heron development. > > = Initial Goals = > > * Move the existing codebase, website, documentation, and mailing lists to > Apache-hosted infrastructure. > * Integrate with the Apache development process. > * Ensure all dependencies are compliant with Apache License version 2.0. > * Incrementally develop and release per Apache guidelines. > > = Current Status = > > Heron is a stable project used in production at Twitter since 2014 and open > sourced under the ASL v2 license in 2016. The Heron source code is > currently hosted at github.com (https://github.com/twitter/heron), which > will seed the Apache git repository. > > = Meritocracy = > > By submitting this incubator proposal, weâre expressing our intent to build > a diverse developer community around Heron that will conduct itself > according to The Apache Way and use a meritocratic means of building it's > committer base. Several companies and universities have already expressed > interest in and contributed to Heron. Our goal is to grow the Heron > community by encouraging open communication, contribution and participation > of all types, and ensuring that contributors are recognized appropriately. > > = Community = > > Heron is currently being used by Twitter, Google, Machine Zone and > ndustrial.io and has received significant contributions by Microsoft and > Streamlio. By bringing Heron into the Apache ecosystem, we believe we can > attract even more developers who are interested in creating real-time > systems to build the project's contributor base. > > == Core Developers == > > Current core developers are engineers from Twitter, Google, Microsoft and > Streamlio. > > == Alignment == > > Heron utilizes a number of Apache technologies. Heron leverages Apache > ZooKeeper for coordination and has scheduler implementations to integrate > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF) > as well as spout implementations to integrate with Apache Kafka and metrics > implementations to integrate with Scribe. Heron also implements the Apache > Storm user-level API, which allows topologies written against Storm to run > in Heron. We believe that having Heron at Apache will help further the > growth of the streaming compute community, as well as encourage cooperation > and developer cross pollination with other Apache projects. > > = Known Risks = > > == Orphaned Products == > > The risk of the Heron project being abandoned is minimal. It is used in > production at Twitter and Google and other companies are evaluating or > adopting it for production use. > > == Inexperience with Open Source == > > All of the core contributors to the project have considerable experience > with open source software development. Bill Graham[2], Ashvin Agrawal[3] > and Supun Kamburugamuve[4], committers on the project, are PMCs on other > Apache projects and Bill and Ashvin have gone through the Apache incubator > process. Twitter has already donated numerous projects to the ASF (e.g., > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored > by experienced ASF members that can help with any roadblocks. > > == Homogenous Developers == > > Initial committers come from 5 separate organizations. Our intention is > increase the diversity of contributing developers and their affiliations. > To date github contributions have come from approximately 50 contributors > from outside the Twitter team. > > == Reliance on Salaried Developers == > > It is expected that Heron development will occur on both salaried time and > on volunteer time. The majority of initial committers are paid by their > employers to contribute to this project. We are committed to recruiting > additional committers from other organizations as well as non-salaried > committers to join project. > > == Relationships with Other Apache Products == > > As mentioned in the Alignment section, Heron implements the Apache Storm > API and integrates with multiple Apache schedulers (Apache Mesos, Apache > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache > Thrift. > > == An Excessive Fascination with the Apache Brand == > > Heron's popularity is growing in the streaming compute space and we are > long time supporters of the Apache brand. This proposal is not for the > purpose of generating publicity through. Rather, the primary benefits to > joining Apache are those of community building and open decision making > outlined in the Rationale section. > > == Documentation == > > This proposal exists online as > http://wiki.apache.org/incubator/HeronProposal. Extensive documentation can > be found on github at https://twitter.github.io/heron and the source code > is well documented. > > == Source and Intellectual Property Submission Plan == > > The Heron codebase is currently hosted on Github: > https://github.com/twitter/heron. During incubation, the codebase will be > migrated to Apache infrastructure. The source code is already ASF 2.0 > licensed. > > == External Dependencies == > > All external libraries have ASF 2.0 compatible licenses except for pylint. > The pylint library is GPL licensed, but is only used for pre-build Python > style checks and is neither bundled with, nor relied upon by, the Heron > source or binary release artifacts. > > == Cryptography == > > Heron does not use any cryptography libraries. > > = Required Resources = > > == Mailing lists == > > * [hidden email] (with moderated subscriptions) > * [hidden email] > * [hidden email] > * [hidden email] > > == Subversion Directory == > > Git is the preferred source control system: git://git.apache.org/heron > > == Issue Tracking == > > JIRA: Heron (HERON) > > == Initial Committers == > > * Andrew Jorgensen (andrew at andrewjorgensen dot com) > * Ashvin Agrawal (ashvin at apache dot org)* > * Avrilia Floratou (avrilia dot floratou at gmail dot com) > * Bill Graham (billgraham at apache dot org)* > * Brian Hatfield (bmhatfield at gmail dot com) > * Chris Kellogg (cckellogg at gmail dot com) > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) > * Karthik Ramasamy (karthik at gmail dot com) > * Maosong Fu (maosongfu at gmail dot com) > * Neng Lu(freeneng at gmail dot com) > * Runhang Li (obj dot runhang at gmail dot com) > * Sanjeev Kulkarni (sanjeevrk at gmail dot com) > * Supun Kamburugamuve (supun at apache dot org)* > * Thomas Sun (tom dot ssf at gmail dot com) > * Yaliang Wang (yaliang dot w dot wang at ieee dot org) > > == Affiliations == > > * Andrew Jorgensen (Google) > * Ashvin Agrawal (Microsoft) > * Avrilia Floratou (Microsoft) > * Bill Graham (Twitter) > * Brian Hatfield (Google) > * Chris Kellogg (Twitter) > * Huijun Wu (Twitter) > * Karthik Ramasamy (Streamlio) > * Maosong Fu (Twitter) > * Neng Lu (Twitter) > * Runhang Li (Twitter) > * Sanjeev Kulkarni (Streamlio) > * Supun Kamburugamuve (Indiana University) > * Thomas Sun (Twitter) > * Yaliang Wang (Twitter) > > = Sponsors = > > == Champion == > > * Julien Le Dem (julien at apache dot org) > > == Nominated Mentors == > > * Jake Farrell (jfarrell at apache dot org) > * Jacques Nadeau (jacques at apache dot org) > * Julien Le Dem (julien at apache dot org) > * P. Taylor Goetz (ptgoetz at apache dot org) > > == Sponsoring Entity == > > The Apache Incubator > > == Footnotes == > > * 1 - Papers detailing Heron are available at > http://dl.acm.org/citation.cfm?id=2742788 and > http://sites.computer.org/debull/A15dec/p15.pdf. > * 2 - http://home.apache.org/phonebook.html?uid=billgraham > * 3 - http://home.apache.org/phonebook.html?uid=ashvin > * 4 - http://home.apache.org/phonebook.html?uid=supun > --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
|
In reply to this post by Karthik Ramasamy
+1
On Fri, Jun 16, 2017 at 2:00 PM, Karthik Ramasamy <[hidden email]> wrote: > +1 > > On 2017-06-16 13:50 (-0700), Roy Lenferink <[hidden email]> wrote: > > +1 > > > > Op vr 16 jun. 2017 om 22:49 schreef Julien Le Dem < > [hidden email]> > > > > > +1 > > > > > > On Fri, Jun 16, 2017 at 1:46 PM, William Markito Oliveira < > > > [hidden email]> wrote: > > > > > > > +1 > > > > > > > > On Fri, Jun 16, 2017 at 3:45 PM, Debo Dutta (dedutta) < > [hidden email] > > > > > > > > wrote: > > > > > > > > > +1 > > > > > > > > > > On 6/16/17, 1:41 PM, "Bill Graham" <[hidden email]> wrote: > > > > > > > > > > Hi, > > > > > > > > > > Based on the discussion on the incubator mailing list[1] I > would > > > like > > > > > to > > > > > call a vote to add Heron to the Apache Incubator. > > > > > > > > > > The full proposal is available below, and is also available on > the > > > > > Apache > > > > > Incubator wiki at: > > > > > https://wiki.apache.org/incubator/HeronProposal > > > > > > > > > > Please vote: > > > > > [ ] +1, bring Heron into Incubator > > > > > [ ] -1, do not bring Heron into Incubator, because... > > > > > > > > > > The vote will open for 7 days until Friday June 23 at 14:00 PT. > > > > > > > > > > Thank you > > > > > > > > > > 1 - > > > > > > > > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7 > > > > > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E > > > > > > > > > > > > > > > > > > > > = Heron Proposal = > > > > > > > > > > = Abstract = > > > > > Heron is a real-time, distributed, fault-tolerant stream > processing > > > > > engine > > > > > initially developed by Twitter. > > > > > > > > > > = Proposal = > > > > > > > > > > Heron is a real-time stream processing engine built for high > > > > > performance, > > > > > ease of manageability, performance predictability and developer > > > > > productivity[1]. We wish to develop a community around Heron to > > > > > increase > > > > > contributions and see Heron thrive in an open forum. > > > > > > > > > > = Background = > > > > > > > > > > Heron provides the ability for developers to compose directed > > > acyclic > > > > > graphs (DAGs) of real-time query execution logic (i.e. a > topology) > > > > and > > > > > submit the topology to execute on a pluggable job scheduling > system > > > > > (e.g., > > > > > Apache Aurora, YARN, Marathon, etc). Users can employ either > the > > > > native > > > > > Heron API or the Apache Storm API to develop the topology. > Heron > > > > > supports > > > > > the Storm API for ease of migration, but beyond that Heron’s > > > > > architecture > > > > > differs considerably from Storm’s. > > > > > > > > > > Users submit a topology to the scheduler using the Heron > client, > > > > which > > > > > uses > > > > > the Heron binary libraries to deploy all daemons required to > run > > > and > > > > > manage > > > > > the topology. The topology therefore has no reliance on > centrally > > > > > managed > > > > > Heron services, only on a generic job scheduling system, which > > > lends > > > > > itself > > > > > well to be run on top of Apache Aurora/Mesos or Apache > Hadoop/YARN > > > > > (among > > > > > others). > > > > > > > > > > The scheduler runs each topology as a job consisting of > multiple > > > > > containers. One of the containers runs the topology master, > > > > > responsible for > > > > > managing the topology. The remaining containers each runs a > stream > > > > > manager > > > > > responsible for data routing, a metrics manager that collects > and > > > > > reports > > > > > various metrics and a number of processes called Heron > instances > > > > which > > > > > run > > > > > the user-defined logic on the stream of tuples. Parallelism is > > > > > achieved via > > > > > process-based isolation of Heron instances, which provides > > > > predictable > > > > > performance while simplifying debugging. The containers are > > > allocated > > > > > and > > > > > managed by the scheduler framework based on resource > availability > > > of > > > > > nodes > > > > > in the cluster. The metadata for the topology, such as the > physical > > > > > plan > > > > > and execution details, are stored in the pluggable Heron State > > > > Manager > > > > > (e.g. Apache ZooKeeper). > > > > > > > > > > = Rationale = > > > > > > > > > > Heron is a general-purpose, modular and extensible platform > that > > > can > > > > be > > > > > leveraged to support common, real-time analytics use cases. > There > > > is > > > > an > > > > > increasing demand for open-source, scalable real-time analytics > > > > > systems. We > > > > > believe that Heron can be leveraged by other organizations to > build > > > > > streaming applications that can benefit from its robustness, > high > > > > > performance, adaptability to cloud environments and ease of > use. > > > > > Moreover, > > > > > we hope that open-sourcing Heron will help to further evolve > the > > > > > technology > > > > > as the project attracts contributors with diverse backgrounds > and > > > > > areas of > > > > > expertise. > > > > > > > > > > We believe the Apache foundation is a great fit as the > long-term > > > home > > > > > for > > > > > Heron, as it provides an established process for > community-driven > > > > > development and decision making by consensus. This is exactly > the > > > > > model we > > > > > want for future Heron development. > > > > > > > > > > = Initial Goals = > > > > > > > > > > * Move the existing codebase, website, documentation, and > mailing > > > > > lists to > > > > > Apache-hosted infrastructure. > > > > > * Integrate with the Apache development process. > > > > > * Ensure all dependencies are compliant with Apache License > > > version > > > > > 2.0. > > > > > * Incrementally develop and release per Apache guidelines. > > > > > > > > > > = Current Status = > > > > > > > > > > Heron is a stable project used in production at Twitter since > 2014 > > > > and > > > > > open > > > > > sourced under the ASL v2 license in 2016. The Heron source > code is > > > > > currently hosted at github.com (https://github.com/twitter/ > heron), > > > > > which > > > > > will seed the Apache git repository. > > > > > > > > > > = Meritocracy = > > > > > > > > > > By submitting this incubator proposal, we’re expressing our > intent > > > to > > > > > build > > > > > a diverse developer community around Heron that will conduct > itself > > > > > according to The Apache Way and use a meritocratic means of > > > building > > > > > it's > > > > > committer base. Several companies and universities have already > > > > > expressed > > > > > interest in and contributed to Heron. Our goal is to grow the > Heron > > > > > community by encouraging open communication, contribution and > > > > > participation > > > > > of all types, and ensuring that contributors are recognized > > > > > appropriately. > > > > > > > > > > = Community = > > > > > > > > > > Heron is currently being used by Twitter, Google, Machine Zone > and > > > > > ndustrial.io and has received significant contributions by > > > Microsoft > > > > > and > > > > > Streamlio. By bringing Heron into the Apache ecosystem, we > believe > > > we > > > > > can > > > > > attract even more developers who are interested in creating > > > real-time > > > > > systems to build the project's contributor base. > > > > > > > > > > == Core Developers == > > > > > > > > > > Current core developers are engineers from Twitter, Google, > > > Microsoft > > > > > and > > > > > Streamlio. > > > > > > > > > > == Alignment == > > > > > > > > > > Heron utilizes a number of Apache technologies. Heron leverages > > > > Apache > > > > > ZooKeeper for coordination and has scheduler implementations to > > > > > integrate > > > > > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via > > > Apache > > > > > REEF) > > > > > as well as spout implementations to integrate with Apache > Kafka and > > > > > metrics > > > > > implementations to integrate with Scribe. Heron also > implements the > > > > > Apache > > > > > Storm user-level API, which allows topologies written against > Storm > > > > to > > > > > run > > > > > in Heron. We believe that having Heron at Apache will help > further > > > > the > > > > > growth of the streaming compute community, as well as encourage > > > > > cooperation > > > > > and developer cross pollination with other Apache projects. > > > > > > > > > > = Known Risks = > > > > > > > > > > == Orphaned Products == > > > > > > > > > > The risk of the Heron project being abandoned is minimal. It is > > > used > > > > in > > > > > production at Twitter and Google and other companies are > evaluating > > > > or > > > > > adopting it for production use. > > > > > > > > > > == Inexperience with Open Source == > > > > > > > > > > All of the core contributors to the project have considerable > > > > > experience > > > > > with open source software development. Bill Graham[2], Ashvin > > > > > Agrawal[3] > > > > > and Supun Kamburugamuve[4], committers on the project, are > PMCs on > > > > > other > > > > > Apache projects and Bill and Ashvin have gone through the > Apache > > > > > incubator > > > > > process. Twitter has already donated numerous projects to the > ASF > > > > > (e.g., > > > > > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to > be > > > > > mentored > > > > > by experienced ASF members that can help with any roadblocks. > > > > > > > > > > == Homogenous Developers == > > > > > > > > > > Initial committers come from 5 separate organizations. Our > > > intention > > > > is > > > > > increase the diversity of contributing developers and their > > > > > affiliations. > > > > > To date github contributions have come from approximately 50 > > > > > contributors > > > > > from outside the Twitter team. > > > > > > > > > > == Reliance on Salaried Developers == > > > > > > > > > > It is expected that Heron development will occur on both > salaried > > > > time > > > > > and > > > > > on volunteer time. The majority of initial committers are paid > by > > > > their > > > > > employers to contribute to this project. We are committed to > > > > recruiting > > > > > additional committers from other organizations as well as > > > > non-salaried > > > > > committers to join project. > > > > > > > > > > == Relationships with Other Apache Products == > > > > > > > > > > As mentioned in the Alignment section, Heron implements the > Apache > > > > > Storm > > > > > API and integrates with multiple Apache schedulers (Apache > Mesos, > > > > > Apache > > > > > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper > and > > > > Apache > > > > > Thrift. > > > > > > > > > > == An Excessive Fascination with the Apache Brand == > > > > > > > > > > Heron's popularity is growing in the streaming compute space > and we > > > > are > > > > > long time supporters of the Apache brand. This proposal is not > for > > > > the > > > > > purpose of generating publicity through. Rather, the primary > > > benefits > > > > > to > > > > > joining Apache are those of community building and open > decision > > > > making > > > > > outlined in the Rationale section. > > > > > > > > > > == Documentation == > > > > > > > > > > This proposal exists online as > > > > > http://wiki.apache.org/incubator/HeronProposal. Extensive > > > > > documentation can > > > > > be found on github at https://twitter.github.io/heron and the > > > source > > > > > code > > > > > is well documented. > > > > > > > > > > == Source and Intellectual Property Submission Plan == > > > > > > > > > > The Heron codebase is currently hosted on Github: > > > > > https://github.com/twitter/heron. During incubation, the > codebase > > > > > will be > > > > > migrated to Apache infrastructure. The source code is already > ASF > > > 2.0 > > > > > licensed. > > > > > > > > > > == External Dependencies == > > > > > > > > > > All external libraries have ASF 2.0 compatible licenses except > for > > > > > pylint. > > > > > The pylint library is GPL licensed, but is only used for > pre-build > > > > > Python > > > > > style checks and is neither bundled with, nor relied upon by, > the > > > > Heron > > > > > source or binary release artifacts. > > > > > > > > > > == Cryptography == > > > > > > > > > > Heron does not use any cryptography libraries. > > > > > > > > > > = Required Resources = > > > > > > > > > > == Mailing lists == > > > > > > > > > > * [hidden email] (with moderated > > > subscriptions) > > > > > * [hidden email] > > > > > * [hidden email] > > > > > * [hidden email] > > > > > > > > > > == Subversion Directory == > > > > > > > > > > Git is the preferred source control system: git:// > > > > git.apache.org/heron > > > > > > > > > > == Issue Tracking == > > > > > > > > > > JIRA: Heron (HERON) > > > > > > > > > > == Initial Committers == > > > > > > > > > > * Andrew Jorgensen (andrew at andrewjorgensen dot com) > > > > > * Ashvin Agrawal (ashvin at apache dot org)* > > > > > * Avrilia Floratou (avrilia dot floratou at gmail dot com) > > > > > * Bill Graham (billgraham at apache dot org)* > > > > > * Brian Hatfield (bmhatfield at gmail dot com) > > > > > * Chris Kellogg (cckellogg at gmail dot com) > > > > > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) > > > > > * Karthik Ramasamy (karthik at gmail dot com) > > > > > * Maosong Fu (maosongfu at gmail dot com) > > > > > * Neng Lu(freeneng at gmail dot com) > > > > > * Runhang Li (obj dot runhang at gmail dot com) > > > > > * Sanjeev Kulkarni (sanjeevrk at gmail dot com) > > > > > * Supun Kamburugamuve (supun at apache dot org)* > > > > > * Thomas Sun (tom dot ssf at gmail dot com) > > > > > * Yaliang Wang (yaliang dot w dot wang at ieee dot org) > > > > > > > > > > == Affiliations == > > > > > > > > > > * Andrew Jorgensen (Google) > > > > > * Ashvin Agrawal (Microsoft) > > > > > * Avrilia Floratou (Microsoft) > > > > > * Bill Graham (Twitter) > > > > > * Brian Hatfield (Google) > > > > > * Chris Kellogg (Twitter) > > > > > * Huijun Wu (Twitter) > > > > > * Karthik Ramasamy (Streamlio) > > > > > * Maosong Fu (Twitter) > > > > > * Neng Lu (Twitter) > > > > > * Runhang Li (Twitter) > > > > > * Sanjeev Kulkarni (Streamlio) > > > > > * Supun Kamburugamuve (Indiana University) > > > > > * Thomas Sun (Twitter) > > > > > * Yaliang Wang (Twitter) > > > > > > > > > > = Sponsors = > > > > > > > > > > == Champion == > > > > > > > > > > * Julien Le Dem (julien at apache dot org) > > > > > > > > > > == Nominated Mentors == > > > > > > > > > > * Jake Farrell (jfarrell at apache dot org) > > > > > * Jacques Nadeau (jacques at apache dot org) > > > > > * Julien Le Dem (julien at apache dot org) > > > > > * P. Taylor Goetz (ptgoetz at apache dot org) > > > > > > > > > > == Sponsoring Entity == > > > > > > > > > > The Apache Incubator > > > > > > > > > > == Footnotes == > > > > > > > > > > * 1 - Papers detailing Heron are available at > > > > > http://dl.acm.org/citation.cfm?id=2742788 and > > > > > http://sites.computer.org/debull/A15dec/p15.pdf. > > > > > * 2 - http://home.apache.org/phonebook.html?uid=billgraham > > > > > * 3 - http://home.apache.org/phonebook.html?uid=ashvin > > > > > * 4 - http://home.apache.org/phonebook.html?uid=supun > > > > > > > > > > > > > > > > > > > > > > > > > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [hidden email] > For additional commands, e-mail: [hidden email] > > |
Re: [VOTE] Heron to enter Apache Incubator
|
In reply to this post by Bill Graham-4
Go for it! +1 (binding)
Am .06.2017, 22:41 Uhr, schrieb Bill Graham <[hidden email]>: > Hi, > > Based on the discussion on the incubator mailing list[1] I would like to > call a vote to add Heron to the Apache Incubator. > > The full proposal is available below, and is also available on the Apache > Incubator wiki at: > https://wiki.apache.org/incubator/HeronProposal > > Please vote: > [ ] +1, bring Heron into Incubator > [ ] -1, do not bring Heron into Incubator, because... > > The vote will open for 7 days until Friday June 23 at 14:00 PT. > > Thank you > > 1 - > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E > > > > = Heron Proposal = > > = Abstract = > Heron is a real-time, distributed, fault-tolerant stream processing > engine > initially developed by Twitter. > > = Proposal = > > Heron is a real-time stream processing engine built for high performance, > ease of manageability, performance predictability and developer > productivity[1]. We wish to develop a community around Heron to increase > contributions and see Heron thrive in an open forum. > > = Background = > > Heron provides the ability for developers to compose directed acyclic > graphs (DAGs) of real-time query execution logic (i.e. a topology) and > submit the topology to execute on a pluggable job scheduling system > (e.g., > Apache Aurora, YARN, Marathon, etc). Users can employ either the native > Heron API or the Apache Storm API to develop the topology. Heron supports > the Storm API for ease of migration, but beyond that Heron’s architecture > differs considerably from Storm’s. > > Users submit a topology to the scheduler using the Heron client, which > uses > the Heron binary libraries to deploy all daemons required to run and > manage > the topology. The topology therefore has no reliance on centrally managed > Heron services, only on a generic job scheduling system, which lends > itself > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among > others). > > The scheduler runs each topology as a job consisting of multiple > containers. One of the containers runs the topology master, responsible > for > managing the topology. The remaining containers each runs a stream > manager > responsible for data routing, a metrics manager that collects and reports > various metrics and a number of processes called Heron instances which > run > the user-defined logic on the stream of tuples. Parallelism is achieved > via > process-based isolation of Heron instances, which provides predictable > performance while simplifying debugging. The containers are allocated and > managed by the scheduler framework based on resource availability of > nodes > in the cluster. The metadata for the topology, such as the physical plan > and execution details, are stored in the pluggable Heron State Manager > (e.g. Apache ZooKeeper). > > = Rationale = > > Heron is a general-purpose, modular and extensible platform that can be > leveraged to support common, real-time analytics use cases. There is an > increasing demand for open-source, scalable real-time analytics systems. > We > believe that Heron can be leveraged by other organizations to build > streaming applications that can benefit from its robustness, high > performance, adaptability to cloud environments and ease of use. > Moreover, > we hope that open-sourcing Heron will help to further evolve the > technology > as the project attracts contributors with diverse backgrounds and areas > of > expertise. > > We believe the Apache foundation is a great fit as the long-term home for > Heron, as it provides an established process for community-driven > development and decision making by consensus. This is exactly the model > we > want for future Heron development. > > = Initial Goals = > > * Move the existing codebase, website, documentation, and mailing lists > to > Apache-hosted infrastructure. > * Integrate with the Apache development process. > * Ensure all dependencies are compliant with Apache License version 2.0. > * Incrementally develop and release per Apache guidelines. > > = Current Status = > > Heron is a stable project used in production at Twitter since 2014 and > open > sourced under the ASL v2 license in 2016. The Heron source code is > currently hosted at github.com (https://github.com/twitter/heron), which > will seed the Apache git repository. > > = Meritocracy = > > By submitting this incubator proposal, we’re expressing our intent to > build > a diverse developer community around Heron that will conduct itself > according to The Apache Way and use a meritocratic means of building it's > committer base. Several companies and universities have already expressed > interest in and contributed to Heron. Our goal is to grow the Heron > community by encouraging open communication, contribution and > participation > of all types, and ensuring that contributors are recognized > appropriately. > > = Community = > > Heron is currently being used by Twitter, Google, Machine Zone and > ndustrial.io and has received significant contributions by Microsoft and > Streamlio. By bringing Heron into the Apache ecosystem, we believe we can > attract even more developers who are interested in creating real-time > systems to build the project's contributor base. > > == Core Developers == > > Current core developers are engineers from Twitter, Google, Microsoft and > Streamlio. > > == Alignment == > > Heron utilizes a number of Apache technologies. Heron leverages Apache > ZooKeeper for coordination and has scheduler implementations to integrate > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache > REEF) > as well as spout implementations to integrate with Apache Kafka and > metrics > implementations to integrate with Scribe. Heron also implements the > Apache > Storm user-level API, which allows topologies written against Storm to > run > in Heron. We believe that having Heron at Apache will help further the > growth of the streaming compute community, as well as encourage > cooperation > and developer cross pollination with other Apache projects. > > = Known Risks = > > == Orphaned Products == > > The risk of the Heron project being abandoned is minimal. It is used in > production at Twitter and Google and other companies are evaluating or > adopting it for production use. > > == Inexperience with Open Source == > > All of the core contributors to the project have considerable experience > with open source software development. Bill Graham[2], Ashvin Agrawal[3] > and Supun Kamburugamuve[4], committers on the project, are PMCs on other > Apache projects and Bill and Ashvin have gone through the Apache > incubator > process. Twitter has already donated numerous projects to the ASF (e.g., > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored > by experienced ASF members that can help with any roadblocks. > > == Homogenous Developers == > > Initial committers come from 5 separate organizations. Our intention is > increase the diversity of contributing developers and their affiliations. > To date github contributions have come from approximately 50 contributors > from outside the Twitter team. > > == Reliance on Salaried Developers == > > It is expected that Heron development will occur on both salaried time > and > on volunteer time. The majority of initial committers are paid by their > employers to contribute to this project. We are committed to recruiting > additional committers from other organizations as well as non-salaried > committers to join project. > > == Relationships with Other Apache Products == > > As mentioned in the Alignment section, Heron implements the Apache Storm > API and integrates with multiple Apache schedulers (Apache Mesos, Apache > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache > Thrift. > > == An Excessive Fascination with the Apache Brand == > > Heron's popularity is growing in the streaming compute space and we are > long time supporters of the Apache brand. This proposal is not for the > purpose of generating publicity through. Rather, the primary benefits to > joining Apache are those of community building and open decision making > outlined in the Rationale section. > > == Documentation == > > This proposal exists online as > http://wiki.apache.org/incubator/HeronProposal. Extensive documentation > can > be found on github at https://twitter.github.io/heron and the source code > is well documented. > > == Source and Intellectual Property Submission Plan == > > The Heron codebase is currently hosted on Github: > https://github.com/twitter/heron. During incubation, the codebase will be > migrated to Apache infrastructure. The source code is already ASF 2.0 > licensed. > > == External Dependencies == > > All external libraries have ASF 2.0 compatible licenses except for > pylint. > The pylint library is GPL licensed, but is only used for pre-build Python > style checks and is neither bundled with, nor relied upon by, the Heron > source or binary release artifacts. > > == Cryptography == > > Heron does not use any cryptography libraries. > > = Required Resources = > > == Mailing lists == > > * [hidden email] (with moderated subscriptions) > * [hidden email] > * [hidden email] > * [hidden email] > > == Subversion Directory == > > Git is the preferred source control system: git://git.apache.org/heron > > == Issue Tracking == > > JIRA: Heron (HERON) > > == Initial Committers == > > * Andrew Jorgensen (andrew at andrewjorgensen dot com) > * Ashvin Agrawal (ashvin at apache dot org)* > * Avrilia Floratou (avrilia dot floratou at gmail dot com) > * Bill Graham (billgraham at apache dot org)* > * Brian Hatfield (bmhatfield at gmail dot com) > * Chris Kellogg (cckellogg at gmail dot com) > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) > * Karthik Ramasamy (karthik at gmail dot com) > * Maosong Fu (maosongfu at gmail dot com) > * Neng Lu(freeneng at gmail dot com) > * Runhang Li (obj dot runhang at gmail dot com) > * Sanjeev Kulkarni (sanjeevrk at gmail dot com) > * Supun Kamburugamuve (supun at apache dot org)* > * Thomas Sun (tom dot ssf at gmail dot com) > * Yaliang Wang (yaliang dot w dot wang at ieee dot org) > > == Affiliations == > > * Andrew Jorgensen (Google) > * Ashvin Agrawal (Microsoft) > * Avrilia Floratou (Microsoft) > * Bill Graham (Twitter) > * Brian Hatfield (Google) > * Chris Kellogg (Twitter) > * Huijun Wu (Twitter) > * Karthik Ramasamy (Streamlio) > * Maosong Fu (Twitter) > * Neng Lu (Twitter) > * Runhang Li (Twitter) > * Sanjeev Kulkarni (Streamlio) > * Supun Kamburugamuve (Indiana University) > * Thomas Sun (Twitter) > * Yaliang Wang (Twitter) > > = Sponsors = > > == Champion == > > * Julien Le Dem (julien at apache dot org) > > == Nominated Mentors == > > * Jake Farrell (jfarrell at apache dot org) > * Jacques Nadeau (jacques at apache dot org) > * Julien Le Dem (julien at apache dot org) > * P. Taylor Goetz (ptgoetz at apache dot org) > > == Sponsoring Entity == > > The Apache Incubator > > == Footnotes == > > * 1 - Papers detailing Heron are available at > http://dl.acm.org/citation.cfm?id=2742788 and > http://sites.computer.org/debull/A15dec/p15.pdf. > * 2 - http://home.apache.org/phonebook.html?uid=billgraham > * 3 - http://home.apache.org/phonebook.html?uid=ashvin > * 4 - http://home.apache.org/phonebook.html?uid=supun -- My introduction https://youtu.be/Ln4vly5sxYU --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
Re: [VOTE] Heron to enter Apache Incubator
|
In reply to this post by Bill Graham-4
+1 binding
-Jake On Fri, Jun 16, 2017 at 4:41 PM, Bill Graham <[hidden email]> wrote: > Hi, > > Based on the discussion on the incubator mailing list[1] I would like to > call a vote to add Heron to the Apache Incubator. > > The full proposal is available below, and is also available on the Apache > Incubator wiki at: > https://wiki.apache.org/incubator/HeronProposal > > Please vote: > [ ] +1, bring Heron into Incubator > [ ] -1, do not bring Heron into Incubator, because... > > The vote will open for 7 days until Friday June 23 at 14:00 PT. > > Thank you > > 1 - > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7 > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E > > > > = Heron Proposal = > > = Abstract = > Heron is a real-time, distributed, fault-tolerant stream processing engine > initially developed by Twitter. > > = Proposal = > > Heron is a real-time stream processing engine built for high performance, > ease of manageability, performance predictability and developer > productivity[1]. We wish to develop a community around Heron to increase > contributions and see Heron thrive in an open forum. > > = Background = > > Heron provides the ability for developers to compose directed acyclic > graphs (DAGs) of real-time query execution logic (i.e. a topology) and > submit the topology to execute on a pluggable job scheduling system (e.g., > Apache Aurora, YARN, Marathon, etc). Users can employ either the native > Heron API or the Apache Storm API to develop the topology. Heron supports > the Storm API for ease of migration, but beyond that Heron’s architecture > differs considerably from Storm’s. > > Users submit a topology to the scheduler using the Heron client, which uses > the Heron binary libraries to deploy all daemons required to run and manage > the topology. The topology therefore has no reliance on centrally managed > Heron services, only on a generic job scheduling system, which lends itself > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among > others). > > The scheduler runs each topology as a job consisting of multiple > containers. One of the containers runs the topology master, responsible for > managing the topology. The remaining containers each runs a stream manager > responsible for data routing, a metrics manager that collects and reports > various metrics and a number of processes called Heron instances which run > the user-defined logic on the stream of tuples. Parallelism is achieved via > process-based isolation of Heron instances, which provides predictable > performance while simplifying debugging. The containers are allocated and > managed by the scheduler framework based on resource availability of nodes > in the cluster. The metadata for the topology, such as the physical plan > and execution details, are stored in the pluggable Heron State Manager > (e.g. Apache ZooKeeper). > > = Rationale = > > Heron is a general-purpose, modular and extensible platform that can be > leveraged to support common, real-time analytics use cases. There is an > increasing demand for open-source, scalable real-time analytics systems. We > believe that Heron can be leveraged by other organizations to build > streaming applications that can benefit from its robustness, high > performance, adaptability to cloud environments and ease of use. Moreover, > we hope that open-sourcing Heron will help to further evolve the technology > as the project attracts contributors with diverse backgrounds and areas of > expertise. > > We believe the Apache foundation is a great fit as the long-term home for > Heron, as it provides an established process for community-driven > development and decision making by consensus. This is exactly the model we > want for future Heron development. > > = Initial Goals = > > * Move the existing codebase, website, documentation, and mailing lists to > Apache-hosted infrastructure. > * Integrate with the Apache development process. > * Ensure all dependencies are compliant with Apache License version 2.0. > * Incrementally develop and release per Apache guidelines. > > = Current Status = > > Heron is a stable project used in production at Twitter since 2014 and open > sourced under the ASL v2 license in 2016. The Heron source code is > currently hosted at github.com (https://github.com/twitter/heron), which > will seed the Apache git repository. > > = Meritocracy = > > By submitting this incubator proposal, we’re expressing our intent to build > a diverse developer community around Heron that will conduct itself > according to The Apache Way and use a meritocratic means of building it's > committer base. Several companies and universities have already expressed > interest in and contributed to Heron. Our goal is to grow the Heron > community by encouraging open communication, contribution and participation > of all types, and ensuring that contributors are recognized appropriately. > > = Community = > > Heron is currently being used by Twitter, Google, Machine Zone and > ndustrial.io and has received significant contributions by Microsoft and > Streamlio. By bringing Heron into the Apache ecosystem, we believe we can > attract even more developers who are interested in creating real-time > systems to build the project's contributor base. > > == Core Developers == > > Current core developers are engineers from Twitter, Google, Microsoft and > Streamlio. > > == Alignment == > > Heron utilizes a number of Apache technologies. Heron leverages Apache > ZooKeeper for coordination and has scheduler implementations to integrate > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF) > as well as spout implementations to integrate with Apache Kafka and metrics > implementations to integrate with Scribe. Heron also implements the Apache > Storm user-level API, which allows topologies written against Storm to run > in Heron. We believe that having Heron at Apache will help further the > growth of the streaming compute community, as well as encourage cooperation > and developer cross pollination with other Apache projects. > > = Known Risks = > > == Orphaned Products == > > The risk of the Heron project being abandoned is minimal. It is used in > production at Twitter and Google and other companies are evaluating or > adopting it for production use. > > == Inexperience with Open Source == > > All of the core contributors to the project have considerable experience > with open source software development. Bill Graham[2], Ashvin Agrawal[3] > and Supun Kamburugamuve[4], committers on the project, are PMCs on other > Apache projects and Bill and Ashvin have gone through the Apache incubator > process. Twitter has already donated numerous projects to the ASF (e.g., > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored > by experienced ASF members that can help with any roadblocks. > > == Homogenous Developers == > > Initial committers come from 5 separate organizations. Our intention is > increase the diversity of contributing developers and their affiliations. > To date github contributions have come from approximately 50 contributors > from outside the Twitter team. > > == Reliance on Salaried Developers == > > It is expected that Heron development will occur on both salaried time and > on volunteer time. The majority of initial committers are paid by their > employers to contribute to this project. We are committed to recruiting > additional committers from other organizations as well as non-salaried > committers to join project. > > == Relationships with Other Apache Products == > > As mentioned in the Alignment section, Heron implements the Apache Storm > API and integrates with multiple Apache schedulers (Apache Mesos, Apache > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache > Thrift. > > == An Excessive Fascination with the Apache Brand == > > Heron's popularity is growing in the streaming compute space and we are > long time supporters of the Apache brand. This proposal is not for the > purpose of generating publicity through. Rather, the primary benefits to > joining Apache are those of community building and open decision making > outlined in the Rationale section. > > == Documentation == > > This proposal exists online as > http://wiki.apache.org/incubator/HeronProposal. Extensive documentation > can > be found on github at https://twitter.github.io/heron and the source code > is well documented. > > == Source and Intellectual Property Submission Plan == > > The Heron codebase is currently hosted on Github: > https://github.com/twitter/heron. During incubation, the codebase will be > migrated to Apache infrastructure. The source code is already ASF 2.0 > licensed. > > == External Dependencies == > > All external libraries have ASF 2.0 compatible licenses except for pylint. > The pylint library is GPL licensed, but is only used for pre-build Python > style checks and is neither bundled with, nor relied upon by, the Heron > source or binary release artifacts. > > == Cryptography == > > Heron does not use any cryptography libraries. > > = Required Resources = > > == Mailing lists == > > * [hidden email] (with moderated subscriptions) > * [hidden email] > * [hidden email] > * [hidden email] > > == Subversion Directory == > > Git is the preferred source control system: git://git.apache.org/heron > > == Issue Tracking == > > JIRA: Heron (HERON) > > == Initial Committers == > > * Andrew Jorgensen (andrew at andrewjorgensen dot com) > * Ashvin Agrawal (ashvin at apache dot org)* > * Avrilia Floratou (avrilia dot floratou at gmail dot com) > * Bill Graham (billgraham at apache dot org)* > * Brian Hatfield (bmhatfield at gmail dot com) > * Chris Kellogg (cckellogg at gmail dot com) > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) > * Karthik Ramasamy (karthik at gmail dot com) > * Maosong Fu (maosongfu at gmail dot com) > * Neng Lu(freeneng at gmail dot com) > * Runhang Li (obj dot runhang at gmail dot com) > * Sanjeev Kulkarni (sanjeevrk at gmail dot com) > * Supun Kamburugamuve (supun at apache dot org)* > * Thomas Sun (tom dot ssf at gmail dot com) > * Yaliang Wang (yaliang dot w dot wang at ieee dot org) > > == Affiliations == > > * Andrew Jorgensen (Google) > * Ashvin Agrawal (Microsoft) > * Avrilia Floratou (Microsoft) > * Bill Graham (Twitter) > * Brian Hatfield (Google) > * Chris Kellogg (Twitter) > * Huijun Wu (Twitter) > * Karthik Ramasamy (Streamlio) > * Maosong Fu (Twitter) > * Neng Lu (Twitter) > * Runhang Li (Twitter) > * Sanjeev Kulkarni (Streamlio) > * Supun Kamburugamuve (Indiana University) > * Thomas Sun (Twitter) > * Yaliang Wang (Twitter) > > = Sponsors = > > == Champion == > > * Julien Le Dem (julien at apache dot org) > > == Nominated Mentors == > > * Jake Farrell (jfarrell at apache dot org) > * Jacques Nadeau (jacques at apache dot org) > * Julien Le Dem (julien at apache dot org) > * P. Taylor Goetz (ptgoetz at apache dot org) > > == Sponsoring Entity == > > The Apache Incubator > > == Footnotes == > > * 1 - Papers detailing Heron are available at > http://dl.acm.org/citation.cfm?id=2742788 and > http://sites.computer.org/debull/A15dec/p15.pdf. > * 2 - http://home.apache.org/phonebook.html?uid=billgraham > * 3 - http://home.apache.org/phonebook.html?uid=ashvin > * 4 - http://home.apache.org/phonebook.html?uid=supun > |
|
In reply to this post by Bill Graham-4
+1 (non-binding)
On Fri, Jun 16, 2017 at 1:41 PM, Bill Graham <[hidden email]> wrote: > Hi, > > Based on the discussion on the incubator mailing list[1] I would like to > call a vote to add Heron to the Apache Incubator. > > The full proposal is available below, and is also available on the Apache > Incubator wiki at: > https://wiki.apache.org/incubator/HeronProposal > > Please vote: > [ ] +1, bring Heron into Incubator > [ ] -1, do not bring Heron into Incubator, because... > > The vote will open for 7 days until Friday June 23 at 14:00 PT. > > Thank you > > 1 - > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E > > > > = Heron Proposal = > > = Abstract = > Heron is a real-time, distributed, fault-tolerant stream processing engine > initially developed by Twitter. > > = Proposal = > > Heron is a real-time stream processing engine built for high performance, > ease of manageability, performance predictability and developer > productivity[1]. We wish to develop a community around Heron to increase > contributions and see Heron thrive in an open forum. > > = Background = > > Heron provides the ability for developers to compose directed acyclic > graphs (DAGs) of real-time query execution logic (i.e. a topology) and > submit the topology to execute on a pluggable job scheduling system (e.g., > Apache Aurora, YARN, Marathon, etc). Users can employ either the native > Heron API or the Apache Storm API to develop the topology. Heron supports > the Storm API for ease of migration, but beyond that Heron’s architecture > differs considerably from Storm’s. > > Users submit a topology to the scheduler using the Heron client, which uses > the Heron binary libraries to deploy all daemons required to run and manage > the topology. The topology therefore has no reliance on centrally managed > Heron services, only on a generic job scheduling system, which lends itself > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among > others). > > The scheduler runs each topology as a job consisting of multiple > containers. One of the containers runs the topology master, responsible for > managing the topology. The remaining containers each runs a stream manager > responsible for data routing, a metrics manager that collects and reports > various metrics and a number of processes called Heron instances which run > the user-defined logic on the stream of tuples. Parallelism is achieved via > process-based isolation of Heron instances, which provides predictable > performance while simplifying debugging. The containers are allocated and > managed by the scheduler framework based on resource availability of nodes > in the cluster. The metadata for the topology, such as the physical plan > and execution details, are stored in the pluggable Heron State Manager > (e.g. Apache ZooKeeper). > > = Rationale = > > Heron is a general-purpose, modular and extensible platform that can be > leveraged to support common, real-time analytics use cases. There is an > increasing demand for open-source, scalable real-time analytics systems. We > believe that Heron can be leveraged by other organizations to build > streaming applications that can benefit from its robustness, high > performance, adaptability to cloud environments and ease of use. Moreover, > we hope that open-sourcing Heron will help to further evolve the technology > as the project attracts contributors with diverse backgrounds and areas of > expertise. > > We believe the Apache foundation is a great fit as the long-term home for > Heron, as it provides an established process for community-driven > development and decision making by consensus. This is exactly the model we > want for future Heron development. > > = Initial Goals = > > * Move the existing codebase, website, documentation, and mailing lists to > Apache-hosted infrastructure. > * Integrate with the Apache development process. > * Ensure all dependencies are compliant with Apache License version 2.0. > * Incrementally develop and release per Apache guidelines. > > = Current Status = > > Heron is a stable project used in production at Twitter since 2014 and open > sourced under the ASL v2 license in 2016. The Heron source code is > currently hosted at github.com (https://github.com/twitter/heron), which > will seed the Apache git repository. > > = Meritocracy = > > By submitting this incubator proposal, we’re expressing our intent to build > a diverse developer community around Heron that will conduct itself > according to The Apache Way and use a meritocratic means of building it's > committer base. Several companies and universities have already expressed > interest in and contributed to Heron. Our goal is to grow the Heron > community by encouraging open communication, contribution and participation > of all types, and ensuring that contributors are recognized appropriately. > > = Community = > > Heron is currently being used by Twitter, Google, Machine Zone and > ndustrial.io and has received significant contributions by Microsoft and > Streamlio. By bringing Heron into the Apache ecosystem, we believe we can > attract even more developers who are interested in creating real-time > systems to build the project's contributor base. > > == Core Developers == > > Current core developers are engineers from Twitter, Google, Microsoft and > Streamlio. > > == Alignment == > > Heron utilizes a number of Apache technologies. Heron leverages Apache > ZooKeeper for coordination and has scheduler implementations to integrate > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF) > as well as spout implementations to integrate with Apache Kafka and metrics > implementations to integrate with Scribe. Heron also implements the Apache > Storm user-level API, which allows topologies written against Storm to run > in Heron. We believe that having Heron at Apache will help further the > growth of the streaming compute community, as well as encourage cooperation > and developer cross pollination with other Apache projects. > > = Known Risks = > > == Orphaned Products == > > The risk of the Heron project being abandoned is minimal. It is used in > production at Twitter and Google and other companies are evaluating or > adopting it for production use. > > == Inexperience with Open Source == > > All of the core contributors to the project have considerable experience > with open source software development. Bill Graham[2], Ashvin Agrawal[3] > and Supun Kamburugamuve[4], committers on the project, are PMCs on other > Apache projects and Bill and Ashvin have gone through the Apache incubator > process. Twitter has already donated numerous projects to the ASF (e.g., > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored > by experienced ASF members that can help with any roadblocks. > > == Homogenous Developers == > > Initial committers come from 5 separate organizations. Our intention is > increase the diversity of contributing developers and their affiliations. > To date github contributions have come from approximately 50 contributors > from outside the Twitter team. > > == Reliance on Salaried Developers == > > It is expected that Heron development will occur on both salaried time and > on volunteer time. The majority of initial committers are paid by their > employers to contribute to this project. We are committed to recruiting > additional committers from other organizations as well as non-salaried > committers to join project. > > == Relationships with Other Apache Products == > > As mentioned in the Alignment section, Heron implements the Apache Storm > API and integrates with multiple Apache schedulers (Apache Mesos, Apache > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache > Thrift. > > == An Excessive Fascination with the Apache Brand == > > Heron's popularity is growing in the streaming compute space and we are > long time supporters of the Apache brand. This proposal is not for the > purpose of generating publicity through. Rather, the primary benefits to > joining Apache are those of community building and open decision making > outlined in the Rationale section. > > == Documentation == > > This proposal exists online as > http://wiki.apache.org/incubator/HeronProposal. Extensive documentation can > be found on github at https://twitter.github.io/heron and the source code > is well documented. > > == Source and Intellectual Property Submission Plan == > > The Heron codebase is currently hosted on Github: > https://github.com/twitter/heron. During incubation, the codebase will be > migrated to Apache infrastructure. The source code is already ASF 2.0 > licensed. > > == External Dependencies == > > All external libraries have ASF 2.0 compatible licenses except for pylint. > The pylint library is GPL licensed, but is only used for pre-build Python > style checks and is neither bundled with, nor relied upon by, the Heron > source or binary release artifacts. > > == Cryptography == > > Heron does not use any cryptography libraries. > > = Required Resources = > > == Mailing lists == > > * [hidden email] (with moderated subscriptions) > * [hidden email] > * [hidden email] > * [hidden email] > > == Subversion Directory == > > Git is the preferred source control system: git://git.apache.org/heron > > == Issue Tracking == > > JIRA: Heron (HERON) > > == Initial Committers == > > * Andrew Jorgensen (andrew at andrewjorgensen dot com) > * Ashvin Agrawal (ashvin at apache dot org)* > * Avrilia Floratou (avrilia dot floratou at gmail dot com) > * Bill Graham (billgraham at apache dot org)* > * Brian Hatfield (bmhatfield at gmail dot com) > * Chris Kellogg (cckellogg at gmail dot com) > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) > * Karthik Ramasamy (karthik at gmail dot com) > * Maosong Fu (maosongfu at gmail dot com) > * Neng Lu(freeneng at gmail dot com) > * Runhang Li (obj dot runhang at gmail dot com) > * Sanjeev Kulkarni (sanjeevrk at gmail dot com) > * Supun Kamburugamuve (supun at apache dot org)* > * Thomas Sun (tom dot ssf at gmail dot com) > * Yaliang Wang (yaliang dot w dot wang at ieee dot org) > > == Affiliations == > > * Andrew Jorgensen (Google) > * Ashvin Agrawal (Microsoft) > * Avrilia Floratou (Microsoft) > * Bill Graham (Twitter) > * Brian Hatfield (Google) > * Chris Kellogg (Twitter) > * Huijun Wu (Twitter) > * Karthik Ramasamy (Streamlio) > * Maosong Fu (Twitter) > * Neng Lu (Twitter) > * Runhang Li (Twitter) > * Sanjeev Kulkarni (Streamlio) > * Supun Kamburugamuve (Indiana University) > * Thomas Sun (Twitter) > * Yaliang Wang (Twitter) > > = Sponsors = > > == Champion == > > * Julien Le Dem (julien at apache dot org) > > == Nominated Mentors == > > * Jake Farrell (jfarrell at apache dot org) > * Jacques Nadeau (jacques at apache dot org) > * Julien Le Dem (julien at apache dot org) > * P. Taylor Goetz (ptgoetz at apache dot org) > > == Sponsoring Entity == > > The Apache Incubator > > == Footnotes == > > * 1 - Papers detailing Heron are available at > http://dl.acm.org/citation.cfm?id=2742788 and > http://sites.computer.org/debull/A15dec/p15.pdf. > * 2 - http://home.apache.org/phonebook.html?uid=billgraham > * 3 - http://home.apache.org/phonebook.html?uid=ashvin > * 4 - http://home.apache.org/phonebook.html?uid=supun -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
Re: [VOTE] Heron to enter Apache Incubator
|
+1
On Jun 16, 2017 7:26 PM, "Ashish" <[hidden email]> wrote: +1 (non-binding) On Fri, Jun 16, 2017 at 1:41 PM, Bill Graham <[hidden email]> wrote: > Hi, > > Based on the discussion on the incubator mailing list[1] I would like to > call a vote to add Heron to the Apache Incubator. > > The full proposal is available below, and is also available on the Apache > Incubator wiki at: > https://wiki.apache.org/incubator/HeronProposal > > Please vote: > [ ] +1, bring Heron into Incubator > [ ] -1, do not bring Heron into Incubator, because... > > The vote will open for 7 days until Friday June 23 at 14:00 PT. > > Thank you > > 1 - > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7 > > > > = Heron Proposal = > > = Abstract = > Heron is a real-time, distributed, fault-tolerant stream processing engine > initially developed by Twitter. > > = Proposal = > > Heron is a real-time stream processing engine built for high performance, > ease of manageability, performance predictability and developer > productivity[1]. We wish to develop a community around Heron to increase > contributions and see Heron thrive in an open forum. > > = Background = > > Heron provides the ability for developers to compose directed acyclic > graphs (DAGs) of real-time query execution logic (i.e. a topology) and > submit the topology to execute on a pluggable job scheduling system (e.g., > Apache Aurora, YARN, Marathon, etc). Users can employ either the native > Heron API or the Apache Storm API to develop the topology. Heron supports > the Storm API for ease of migration, but beyond that Heron’s architecture > differs considerably from Storm’s. > > Users submit a topology to the scheduler using the Heron client, which > the Heron binary libraries to deploy all daemons required to run and manage > the topology. The topology therefore has no reliance on centrally managed > Heron services, only on a generic job scheduling system, which lends itself > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among > others). > > The scheduler runs each topology as a job consisting of multiple > containers. One of the containers runs the topology master, responsible for > managing the topology. The remaining containers each runs a stream manager > responsible for data routing, a metrics manager that collects and reports > various metrics and a number of processes called Heron instances which run > the user-defined logic on the stream of tuples. Parallelism is achieved via > process-based isolation of Heron instances, which provides predictable > performance while simplifying debugging. The containers are allocated and > managed by the scheduler framework based on resource availability of nodes > in the cluster. The metadata for the topology, such as the physical plan > and execution details, are stored in the pluggable Heron State Manager > (e.g. Apache ZooKeeper). > > = Rationale = > > Heron is a general-purpose, modular and extensible platform that can be > leveraged to support common, real-time analytics use cases. There is an > increasing demand for open-source, scalable real-time analytics systems. > believe that Heron can be leveraged by other organizations to build > streaming applications that can benefit from its robustness, high > performance, adaptability to cloud environments and ease of use. Moreover, > we hope that open-sourcing Heron will help to further evolve the technology > as the project attracts contributors with diverse backgrounds and areas of > expertise. > > We believe the Apache foundation is a great fit as the long-term home for > Heron, as it provides an established process for community-driven > development and decision making by consensus. This is exactly the model we > want for future Heron development. > > = Initial Goals = > > * Move the existing codebase, website, documentation, and mailing lists > Apache-hosted infrastructure. > * Integrate with the Apache development process. > * Ensure all dependencies are compliant with Apache License version 2.0. > * Incrementally develop and release per Apache guidelines. > > = Current Status = > > Heron is a stable project used in production at Twitter since 2014 and open > sourced under the ASL v2 license in 2016. The Heron source code is > currently hosted at github.com (https://github.com/twitter/heron), which > will seed the Apache git repository. > > = Meritocracy = > > By submitting this incubator proposal, we’re expressing our intent to build > a diverse developer community around Heron that will conduct itself > according to The Apache Way and use a meritocratic means of building it's > committer base. Several companies and universities have already expressed > interest in and contributed to Heron. Our goal is to grow the Heron > community by encouraging open communication, contribution and participation > of all types, and ensuring that contributors are recognized appropriately. > > = Community = > > Heron is currently being used by Twitter, Google, Machine Zone and > ndustrial.io and has received significant contributions by Microsoft and > Streamlio. By bringing Heron into the Apache ecosystem, we believe we can > attract even more developers who are interested in creating real-time > systems to build the project's contributor base. > > == Core Developers == > > Current core developers are engineers from Twitter, Google, Microsoft and > Streamlio. > > == Alignment == > > Heron utilizes a number of Apache technologies. Heron leverages Apache > ZooKeeper for coordination and has scheduler implementations to integrate > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache > as well as spout implementations to integrate with Apache Kafka and metrics > implementations to integrate with Scribe. Heron also implements the Apache > Storm user-level API, which allows topologies written against Storm to run > in Heron. We believe that having Heron at Apache will help further the > growth of the streaming compute community, as well as encourage cooperation > and developer cross pollination with other Apache projects. > > = Known Risks = > > == Orphaned Products == > > The risk of the Heron project being abandoned is minimal. It is used in > production at Twitter and Google and other companies are evaluating or > adopting it for production use. > > == Inexperience with Open Source == > > All of the core contributors to the project have considerable experience > with open source software development. Bill Graham[2], Ashvin Agrawal[3] > and Supun Kamburugamuve[4], committers on the project, are PMCs on other > Apache projects and Bill and Ashvin have gone through the Apache incubator > process. Twitter has already donated numerous projects to the ASF (e.g., > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored > by experienced ASF members that can help with any roadblocks. > > == Homogenous Developers == > > Initial committers come from 5 separate organizations. Our intention is > increase the diversity of contributing developers and their affiliations. > To date github contributions have come from approximately 50 contributors > from outside the Twitter team. > > == Reliance on Salaried Developers == > > It is expected that Heron development will occur on both salaried time and > on volunteer time. The majority of initial committers are paid by their > employers to contribute to this project. We are committed to recruiting > additional committers from other organizations as well as non-salaried > committers to join project. > > == Relationships with Other Apache Products == > > As mentioned in the Alignment section, Heron implements the Apache Storm > API and integrates with multiple Apache schedulers (Apache Mesos, Apache > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache > Thrift. > > == An Excessive Fascination with the Apache Brand == > > Heron's popularity is growing in the streaming compute space and we are > long time supporters of the Apache brand. This proposal is not for the > purpose of generating publicity through. Rather, the primary benefits to > joining Apache are those of community building and open decision making > outlined in the Rationale section. > > == Documentation == > > This proposal exists online as > http://wiki.apache.org/incubator/HeronProposal. Extensive documentation > be found on github at https://twitter.github.io/heron and the source code > is well documented. > > == Source and Intellectual Property Submission Plan == > > The Heron codebase is currently hosted on Github: > https://github.com/twitter/heron. During incubation, the codebase will be > migrated to Apache infrastructure. The source code is already ASF 2.0 > licensed. > > == External Dependencies == > > All external libraries have ASF 2.0 compatible licenses except for pylint. > The pylint library is GPL licensed, but is only used for pre-build Python > style checks and is neither bundled with, nor relied upon by, the Heron > source or binary release artifacts. > > == Cryptography == > > Heron does not use any cryptography libraries. > > = Required Resources = > > == Mailing lists == > > * [hidden email] (with moderated subscriptions) > * [hidden email] > * [hidden email] > * [hidden email] > > == Subversion Directory == > > Git is the preferred source control system: git://git.apache.org/heron > > == Issue Tracking == > > JIRA: Heron (HERON) > > == Initial Committers == > > * Andrew Jorgensen (andrew at andrewjorgensen dot com) > * Ashvin Agrawal (ashvin at apache dot org)* > * Avrilia Floratou (avrilia dot floratou at gmail dot com) > * Bill Graham (billgraham at apache dot org)* > * Brian Hatfield (bmhatfield at gmail dot com) > * Chris Kellogg (cckellogg at gmail dot com) > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) > * Karthik Ramasamy (karthik at gmail dot com) > * Maosong Fu (maosongfu at gmail dot com) > * Neng Lu(freeneng at gmail dot com) > * Runhang Li (obj dot runhang at gmail dot com) > * Sanjeev Kulkarni (sanjeevrk at gmail dot com) > * Supun Kamburugamuve (supun at apache dot org)* > * Thomas Sun (tom dot ssf at gmail dot com) > * Yaliang Wang (yaliang dot w dot wang at ieee dot org) > > == Affiliations == > > * Andrew Jorgensen (Google) > * Ashvin Agrawal (Microsoft) > * Avrilia Floratou (Microsoft) > * Bill Graham (Twitter) > * Brian Hatfield (Google) > * Chris Kellogg (Twitter) > * Huijun Wu (Twitter) > * Karthik Ramasamy (Streamlio) > * Maosong Fu (Twitter) > * Neng Lu (Twitter) > * Runhang Li (Twitter) > * Sanjeev Kulkarni (Streamlio) > * Supun Kamburugamuve (Indiana University) > * Thomas Sun (Twitter) > * Yaliang Wang (Twitter) > > = Sponsors = > > == Champion == > > * Julien Le Dem (julien at apache dot org) > > == Nominated Mentors == > > * Jake Farrell (jfarrell at apache dot org) > * Jacques Nadeau (jacques at apache dot org) > * Julien Le Dem (julien at apache dot org) > * P. Taylor Goetz (ptgoetz at apache dot org) > > == Sponsoring Entity == > > The Apache Incubator > > == Footnotes == > > * 1 - Papers detailing Heron are available at > http://dl.acm.org/citation.cfm?id=2742788 and > http://sites.computer.org/debull/A15dec/p15.pdf. > * 2 - http://home.apache.org/phonebook.html?uid=billgraham > * 3 - http://home.apache.org/phonebook.html?uid=ashvin > * 4 - http://home.apache.org/phonebook.html?uid=supun -- thanks ashish Blog: http://www.ashishpaliwal.com/blog My Photo Galleries: http://www.pbase.com/ashishpaliwal --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
Re: [VOTE] Heron to enter Apache Incubator
|
+1
On Fri, Jun 16, 2017 at 8:28 PM Jacques Nadeau <[hidden email]> wrote: > +1 > > On Jun 16, 2017 7:26 PM, "Ashish" <[hidden email]> wrote: > > +1 (non-binding) > > On Fri, Jun 16, 2017 at 1:41 PM, Bill Graham <[hidden email]> wrote: > > Hi, > > > > Based on the discussion on the incubator mailing list[1] I would like to > > call a vote to add Heron to the Apache Incubator. > > > > The full proposal is available below, and is also available on the Apache > > Incubator wiki at: > > https://wiki.apache.org/incubator/HeronProposal > > > > Please vote: > > [ ] +1, bring Heron into Incubator > > [ ] -1, do not bring Heron into Incubator, because... > > > > The vote will open for 7 days until Friday June 23 at 14:00 PT. > > > > Thank you > > > > 1 - > > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7 > 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E > > > > > > > > = Heron Proposal = > > > > = Abstract = > > Heron is a real-time, distributed, fault-tolerant stream processing > engine > > initially developed by Twitter. > > > > = Proposal = > > > > Heron is a real-time stream processing engine built for high performance, > > ease of manageability, performance predictability and developer > > productivity[1]. We wish to develop a community around Heron to increase > > contributions and see Heron thrive in an open forum. > > > > = Background = > > > > Heron provides the ability for developers to compose directed acyclic > > graphs (DAGs) of real-time query execution logic (i.e. a topology) and > > submit the topology to execute on a pluggable job scheduling system > (e.g., > > Apache Aurora, YARN, Marathon, etc). Users can employ either the native > > Heron API or the Apache Storm API to develop the topology. Heron supports > > the Storm API for ease of migration, but beyond that Heron’s architecture > > differs considerably from Storm’s. > > > > Users submit a topology to the scheduler using the Heron client, which > uses > > the Heron binary libraries to deploy all daemons required to run and > manage > > the topology. The topology therefore has no reliance on centrally managed > > Heron services, only on a generic job scheduling system, which lends > itself > > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among > > others). > > > > The scheduler runs each topology as a job consisting of multiple > > containers. One of the containers runs the topology master, responsible > for > > managing the topology. The remaining containers each runs a stream > manager > > responsible for data routing, a metrics manager that collects and reports > > various metrics and a number of processes called Heron instances which > run > > the user-defined logic on the stream of tuples. Parallelism is achieved > via > > process-based isolation of Heron instances, which provides predictable > > performance while simplifying debugging. The containers are allocated and > > managed by the scheduler framework based on resource availability of > nodes > > in the cluster. The metadata for the topology, such as the physical plan > > and execution details, are stored in the pluggable Heron State Manager > > (e.g. Apache ZooKeeper). > > > > = Rationale = > > > > Heron is a general-purpose, modular and extensible platform that can be > > leveraged to support common, real-time analytics use cases. There is an > > increasing demand for open-source, scalable real-time analytics systems. > We > > believe that Heron can be leveraged by other organizations to build > > streaming applications that can benefit from its robustness, high > > performance, adaptability to cloud environments and ease of use. > Moreover, > > we hope that open-sourcing Heron will help to further evolve the > technology > > as the project attracts contributors with diverse backgrounds and areas > of > > expertise. > > > > We believe the Apache foundation is a great fit as the long-term home for > > Heron, as it provides an established process for community-driven > > development and decision making by consensus. This is exactly the model > we > > want for future Heron development. > > > > = Initial Goals = > > > > * Move the existing codebase, website, documentation, and mailing lists > to > > Apache-hosted infrastructure. > > * Integrate with the Apache development process. > > * Ensure all dependencies are compliant with Apache License version 2.0. > > * Incrementally develop and release per Apache guidelines. > > > > = Current Status = > > > > Heron is a stable project used in production at Twitter since 2014 and > open > > sourced under the ASL v2 license in 2016. The Heron source code is > > currently hosted at github.com (https://github.com/twitter/heron), which > > will seed the Apache git repository. > > > > = Meritocracy = > > > > By submitting this incubator proposal, we’re expressing our intent to > build > > a diverse developer community around Heron that will conduct itself > > according to The Apache Way and use a meritocratic means of building it's > > committer base. Several companies and universities have already expressed > > interest in and contributed to Heron. Our goal is to grow the Heron > > community by encouraging open communication, contribution and > participation > > of all types, and ensuring that contributors are recognized > appropriately. > > > > = Community = > > > > Heron is currently being used by Twitter, Google, Machine Zone and > > ndustrial.io and has received significant contributions by Microsoft and > > Streamlio. By bringing Heron into the Apache ecosystem, we believe we can > > attract even more developers who are interested in creating real-time > > systems to build the project's contributor base. > > > > == Core Developers == > > > > Current core developers are engineers from Twitter, Google, Microsoft and > > Streamlio. > > > > == Alignment == > > > > Heron utilizes a number of Apache technologies. Heron leverages Apache > > ZooKeeper for coordination and has scheduler implementations to integrate > > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache > REEF) > > as well as spout implementations to integrate with Apache Kafka and > metrics > > implementations to integrate with Scribe. Heron also implements the > Apache > > Storm user-level API, which allows topologies written against Storm to > run > > in Heron. We believe that having Heron at Apache will help further the > > growth of the streaming compute community, as well as encourage > cooperation > > and developer cross pollination with other Apache projects. > > > > = Known Risks = > > > > == Orphaned Products == > > > > The risk of the Heron project being abandoned is minimal. It is used in > > production at Twitter and Google and other companies are evaluating or > > adopting it for production use. > > > > == Inexperience with Open Source == > > > > All of the core contributors to the project have considerable experience > > with open source software development. Bill Graham[2], Ashvin Agrawal[3] > > and Supun Kamburugamuve[4], committers on the project, are PMCs on other > > Apache projects and Bill and Ashvin have gone through the Apache > incubator > > process. Twitter has already donated numerous projects to the ASF (e.g., > > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored > > by experienced ASF members that can help with any roadblocks. > > > > == Homogenous Developers == > > > > Initial committers come from 5 separate organizations. Our intention is > > increase the diversity of contributing developers and their affiliations. > > To date github contributions have come from approximately 50 contributors > > from outside the Twitter team. > > > > == Reliance on Salaried Developers == > > > > It is expected that Heron development will occur on both salaried time > and > > on volunteer time. The majority of initial committers are paid by their > > employers to contribute to this project. We are committed to recruiting > > additional committers from other organizations as well as non-salaried > > committers to join project. > > > > == Relationships with Other Apache Products == > > > > As mentioned in the Alignment section, Heron implements the Apache Storm > > API and integrates with multiple Apache schedulers (Apache Mesos, Apache > > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache > > Thrift. > > > > == An Excessive Fascination with the Apache Brand == > > > > Heron's popularity is growing in the streaming compute space and we are > > long time supporters of the Apache brand. This proposal is not for the > > purpose of generating publicity through. Rather, the primary benefits to > > joining Apache are those of community building and open decision making > > outlined in the Rationale section. > > > > == Documentation == > > > > This proposal exists online as > > http://wiki.apache.org/incubator/HeronProposal. Extensive documentation > can > > be found on github at https://twitter.github.io/heron and the source > code > > is well documented. > > > > == Source and Intellectual Property Submission Plan == > > > > The Heron codebase is currently hosted on Github: > > https://github.com/twitter/heron. During incubation, the codebase will > be > > migrated to Apache infrastructure. The source code is already ASF 2.0 > > licensed. > > > > == External Dependencies == > > > > All external libraries have ASF 2.0 compatible licenses except for > pylint. > > The pylint library is GPL licensed, but is only used for pre-build Python > > style checks and is neither bundled with, nor relied upon by, the Heron > > source or binary release artifacts. > > > > == Cryptography == > > > > Heron does not use any cryptography libraries. > > > > = Required Resources = > > > > == Mailing lists == > > > > * [hidden email] (with moderated subscriptions) > > * [hidden email] > > * [hidden email] > > * [hidden email] > > > > == Subversion Directory == > > > > Git is the preferred source control system: git://git.apache.org/heron > > > > == Issue Tracking == > > > > JIRA: Heron (HERON) > > > > == Initial Committers == > > > > * Andrew Jorgensen (andrew at andrewjorgensen dot com) > > * Ashvin Agrawal (ashvin at apache dot org)* > > * Avrilia Floratou (avrilia dot floratou at gmail dot com) > > * Bill Graham (billgraham at apache dot org)* > > * Brian Hatfield (bmhatfield at gmail dot com) > > * Chris Kellogg (cckellogg at gmail dot com) > > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) > > * Karthik Ramasamy (karthik at gmail dot com) > > * Maosong Fu (maosongfu at gmail dot com) > > * Neng Lu(freeneng at gmail dot com) > > * Runhang Li (obj dot runhang at gmail dot com) > > * Sanjeev Kulkarni (sanjeevrk at gmail dot com) > > * Supun Kamburugamuve (supun at apache dot org)* > > * Thomas Sun (tom dot ssf at gmail dot com) > > * Yaliang Wang (yaliang dot w dot wang at ieee dot org) > > > > == Affiliations == > > > > * Andrew Jorgensen (Google) > > * Ashvin Agrawal (Microsoft) > > * Avrilia Floratou (Microsoft) > > * Bill Graham (Twitter) > > * Brian Hatfield (Google) > > * Chris Kellogg (Twitter) > > * Huijun Wu (Twitter) > > * Karthik Ramasamy (Streamlio) > > * Maosong Fu (Twitter) > > * Neng Lu (Twitter) > > * Runhang Li (Twitter) > > * Sanjeev Kulkarni (Streamlio) > > * Supun Kamburugamuve (Indiana University) > > * Thomas Sun (Twitter) > > * Yaliang Wang (Twitter) > > > > = Sponsors = > > > > == Champion == > > > > * Julien Le Dem (julien at apache dot org) > > > > == Nominated Mentors == > > > > * Jake Farrell (jfarrell at apache dot org) > > * Jacques Nadeau (jacques at apache dot org) > > * Julien Le Dem (julien at apache dot org) > > * P. Taylor Goetz (ptgoetz at apache dot org) > > > > == Sponsoring Entity == > > > > The Apache Incubator > > > > == Footnotes == > > > > * 1 - Papers detailing Heron are available at > > http://dl.acm.org/citation.cfm?id=2742788 and > > http://sites.computer.org/debull/A15dec/p15.pdf. > > * 2 - http://home.apache.org/phonebook.html?uid=billgraham > > * 3 - http://home.apache.org/phonebook.html?uid=ashvin > > * 4 - http://home.apache.org/phonebook.html?uid=supun > > > > -- > thanks > ashish > > Blog: http://www.ashishpaliwal.com/blog > My Photo Galleries: http://www.pbase.com/ashishpaliwal > > --------------------------------------------------------------------- > To unsubscribe, e-mail: [hidden email] > For additional commands, e-mail: [hidden email] > |
Re: [VOTE] Heron to enter Apache Incubator
|
+1 (binding)
> On Jun 16, 2017, at 8:43 PM, Nabarun Nag <[hidden email]> wrote: > > +1 > > On Fri, Jun 16, 2017 at 8:28 PM Jacques Nadeau <[hidden email]> wrote: > >> +1 >> >> On Jun 16, 2017 7:26 PM, "Ashish" <[hidden email]> wrote: >> >> +1 (non-binding) >> >> On Fri, Jun 16, 2017 at 1:41 PM, Bill Graham <[hidden email]> wrote: >>> Hi, >>> >>> Based on the discussion on the incubator mailing list[1] I would like to >>> call a vote to add Heron to the Apache Incubator. >>> >>> The full proposal is available below, and is also available on the Apache >>> Incubator wiki at: >>> https://wiki.apache.org/incubator/HeronProposal >>> >>> Please vote: >>> [ ] +1, bring Heron into Incubator >>> [ ] -1, do not bring Heron into Incubator, because... >>> >>> The vote will open for 7 days until Friday June 23 at 14:00 PT. >>> >>> Thank you >>> >>> 1 - >>> https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7 >> 786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E >>> >>> >>> >>> = Heron Proposal = >>> >>> = Abstract = >>> Heron is a real-time, distributed, fault-tolerant stream processing >> engine >>> initially developed by Twitter. >>> >>> = Proposal = >>> >>> Heron is a real-time stream processing engine built for high performance, >>> ease of manageability, performance predictability and developer >>> productivity[1]. We wish to develop a community around Heron to increase >>> contributions and see Heron thrive in an open forum. >>> >>> = Background = >>> >>> Heron provides the ability for developers to compose directed acyclic >>> graphs (DAGs) of real-time query execution logic (i.e. a topology) and >>> submit the topology to execute on a pluggable job scheduling system >> (e.g., >>> Apache Aurora, YARN, Marathon, etc). Users can employ either the native >>> Heron API or the Apache Storm API to develop the topology. Heron supports >>> the Storm API for ease of migration, but beyond that Heron’s architecture >>> differs considerably from Storm’s. >>> >>> Users submit a topology to the scheduler using the Heron client, which >> uses >>> the Heron binary libraries to deploy all daemons required to run and >> manage >>> the topology. The topology therefore has no reliance on centrally managed >>> Heron services, only on a generic job scheduling system, which lends >> itself >>> well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among >>> others). >>> >>> The scheduler runs each topology as a job consisting of multiple >>> containers. One of the containers runs the topology master, responsible >> for >>> managing the topology. The remaining containers each runs a stream >> manager >>> responsible for data routing, a metrics manager that collects and reports >>> various metrics and a number of processes called Heron instances which >> run >>> the user-defined logic on the stream of tuples. Parallelism is achieved >> via >>> process-based isolation of Heron instances, which provides predictable >>> performance while simplifying debugging. The containers are allocated and >>> managed by the scheduler framework based on resource availability of >> nodes >>> in the cluster. The metadata for the topology, such as the physical plan >>> and execution details, are stored in the pluggable Heron State Manager >>> (e.g. Apache ZooKeeper). >>> >>> = Rationale = >>> >>> Heron is a general-purpose, modular and extensible platform that can be >>> leveraged to support common, real-time analytics use cases. There is an >>> increasing demand for open-source, scalable real-time analytics systems. >> We >>> believe that Heron can be leveraged by other organizations to build >>> streaming applications that can benefit from its robustness, high >>> performance, adaptability to cloud environments and ease of use. >> Moreover, >>> we hope that open-sourcing Heron will help to further evolve the >> technology >>> as the project attracts contributors with diverse backgrounds and areas >> of >>> expertise. >>> >>> We believe the Apache foundation is a great fit as the long-term home for >>> Heron, as it provides an established process for community-driven >>> development and decision making by consensus. This is exactly the model >> we >>> want for future Heron development. >>> >>> = Initial Goals = >>> >>> * Move the existing codebase, website, documentation, and mailing lists >> to >>> Apache-hosted infrastructure. >>> * Integrate with the Apache development process. >>> * Ensure all dependencies are compliant with Apache License version 2.0. >>> * Incrementally develop and release per Apache guidelines. >>> >>> = Current Status = >>> >>> Heron is a stable project used in production at Twitter since 2014 and >> open >>> sourced under the ASL v2 license in 2016. The Heron source code is >>> currently hosted at github.com (https://github.com/twitter/heron), which >>> will seed the Apache git repository. >>> >>> = Meritocracy = >>> >>> By submitting this incubator proposal, we’re expressing our intent to >> build >>> a diverse developer community around Heron that will conduct itself >>> according to The Apache Way and use a meritocratic means of building it's >>> committer base. Several companies and universities have already expressed >>> interest in and contributed to Heron. Our goal is to grow the Heron >>> community by encouraging open communication, contribution and >> participation >>> of all types, and ensuring that contributors are recognized >> appropriately. >>> >>> = Community = >>> >>> Heron is currently being used by Twitter, Google, Machine Zone and >>> ndustrial.io and has received significant contributions by Microsoft and >>> Streamlio. By bringing Heron into the Apache ecosystem, we believe we can >>> attract even more developers who are interested in creating real-time >>> systems to build the project's contributor base. >>> >>> == Core Developers == >>> >>> Current core developers are engineers from Twitter, Google, Microsoft and >>> Streamlio. >>> >>> == Alignment == >>> >>> Heron utilizes a number of Apache technologies. Heron leverages Apache >>> ZooKeeper for coordination and has scheduler implementations to integrate >>> with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache >> REEF) >>> as well as spout implementations to integrate with Apache Kafka and >> metrics >>> implementations to integrate with Scribe. Heron also implements the >> Apache >>> Storm user-level API, which allows topologies written against Storm to >> run >>> in Heron. We believe that having Heron at Apache will help further the >>> growth of the streaming compute community, as well as encourage >> cooperation >>> and developer cross pollination with other Apache projects. >>> >>> = Known Risks = >>> >>> == Orphaned Products == >>> >>> The risk of the Heron project being abandoned is minimal. It is used in >>> production at Twitter and Google and other companies are evaluating or >>> adopting it for production use. >>> >>> == Inexperience with Open Source == >>> >>> All of the core contributors to the project have considerable experience >>> with open source software development. Bill Graham[2], Ashvin Agrawal[3] >>> and Supun Kamburugamuve[4], committers on the project, are PMCs on other >>> Apache projects and Bill and Ashvin have gone through the Apache >> incubator >>> process. Twitter has already donated numerous projects to the ASF (e.g., >>> Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored >>> by experienced ASF members that can help with any roadblocks. >>> >>> == Homogenous Developers == >>> >>> Initial committers come from 5 separate organizations. Our intention is >>> increase the diversity of contributing developers and their affiliations. >>> To date github contributions have come from approximately 50 contributors >>> from outside the Twitter team. >>> >>> == Reliance on Salaried Developers == >>> >>> It is expected that Heron development will occur on both salaried time >> and >>> on volunteer time. The majority of initial committers are paid by their >>> employers to contribute to this project. We are committed to recruiting >>> additional committers from other organizations as well as non-salaried >>> committers to join project. >>> >>> == Relationships with Other Apache Products == >>> >>> As mentioned in the Alignment section, Heron implements the Apache Storm >>> API and integrates with multiple Apache schedulers (Apache Mesos, Apache >>> Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache >>> Thrift. >>> >>> == An Excessive Fascination with the Apache Brand == >>> >>> Heron's popularity is growing in the streaming compute space and we are >>> long time supporters of the Apache brand. This proposal is not for the >>> purpose of generating publicity through. Rather, the primary benefits to >>> joining Apache are those of community building and open decision making >>> outlined in the Rationale section. >>> >>> == Documentation == >>> >>> This proposal exists online as >>> http://wiki.apache.org/incubator/HeronProposal. Extensive documentation >> can >>> be found on github at https://twitter.github.io/heron and the source >> code >>> is well documented. >>> >>> == Source and Intellectual Property Submission Plan == >>> >>> The Heron codebase is currently hosted on Github: >>> https://github.com/twitter/heron. During incubation, the codebase will >> be >>> migrated to Apache infrastructure. The source code is already ASF 2.0 >>> licensed. >>> >>> == External Dependencies == >>> >>> All external libraries have ASF 2.0 compatible licenses except for >> pylint. >>> The pylint library is GPL licensed, but is only used for pre-build Python >>> style checks and is neither bundled with, nor relied upon by, the Heron >>> source or binary release artifacts. >>> >>> == Cryptography == >>> >>> Heron does not use any cryptography libraries. >>> >>> = Required Resources = >>> >>> == Mailing lists == >>> >>> * [hidden email] (with moderated subscriptions) >>> * [hidden email] >>> * [hidden email] >>> * [hidden email] >>> >>> == Subversion Directory == >>> >>> Git is the preferred source control system: git://git.apache.org/heron >>> >>> == Issue Tracking == >>> >>> JIRA: Heron (HERON) >>> >>> == Initial Committers == >>> >>> * Andrew Jorgensen (andrew at andrewjorgensen dot com) >>> * Ashvin Agrawal (ashvin at apache dot org)* >>> * Avrilia Floratou (avrilia dot floratou at gmail dot com) >>> * Bill Graham (billgraham at apache dot org)* >>> * Brian Hatfield (bmhatfield at gmail dot com) >>> * Chris Kellogg (cckellogg at gmail dot com) >>> * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) >>> * Karthik Ramasamy (karthik at gmail dot com) >>> * Maosong Fu (maosongfu at gmail dot com) >>> * Neng Lu(freeneng at gmail dot com) >>> * Runhang Li (obj dot runhang at gmail dot com) >>> * Sanjeev Kulkarni (sanjeevrk at gmail dot com) >>> * Supun Kamburugamuve (supun at apache dot org)* >>> * Thomas Sun (tom dot ssf at gmail dot com) >>> * Yaliang Wang (yaliang dot w dot wang at ieee dot org) >>> >>> == Affiliations == >>> >>> * Andrew Jorgensen (Google) >>> * Ashvin Agrawal (Microsoft) >>> * Avrilia Floratou (Microsoft) >>> * Bill Graham (Twitter) >>> * Brian Hatfield (Google) >>> * Chris Kellogg (Twitter) >>> * Huijun Wu (Twitter) >>> * Karthik Ramasamy (Streamlio) >>> * Maosong Fu (Twitter) >>> * Neng Lu (Twitter) >>> * Runhang Li (Twitter) >>> * Sanjeev Kulkarni (Streamlio) >>> * Supun Kamburugamuve (Indiana University) >>> * Thomas Sun (Twitter) >>> * Yaliang Wang (Twitter) >>> >>> = Sponsors = >>> >>> == Champion == >>> >>> * Julien Le Dem (julien at apache dot org) >>> >>> == Nominated Mentors == >>> >>> * Jake Farrell (jfarrell at apache dot org) >>> * Jacques Nadeau (jacques at apache dot org) >>> * Julien Le Dem (julien at apache dot org) >>> * P. Taylor Goetz (ptgoetz at apache dot org) >>> >>> == Sponsoring Entity == >>> >>> The Apache Incubator >>> >>> == Footnotes == >>> >>> * 1 - Papers detailing Heron are available at >>> http://dl.acm.org/citation.cfm?id=2742788 and >>> http://sites.computer.org/debull/A15dec/p15.pdf. >>> * 2 - http://home.apache.org/phonebook.html?uid=billgraham >>> * 3 - http://home.apache.org/phonebook.html?uid=ashvin >>> * 4 - http://home.apache.org/phonebook.html?uid=supun >> >> >> >> -- >> thanks >> ashish >> >> Blog: http://www.ashishpaliwal.com/blog >> My Photo Galleries: http://www.pbase.com/ashishpaliwal >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [hidden email] >> For additional commands, e-mail: [hidden email] >> --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
Re: [VOTE] Heron to enter Apache Incubator
|
In reply to this post by Bill Graham-4
+1 (binding) -C
On Fri, Jun 16, 2017 at 1:41 PM, Bill Graham <[hidden email]> wrote: > Hi, > > Based on the discussion on the incubator mailing list[1] I would like to > call a vote to add Heron to the Apache Incubator. > > The full proposal is available below, and is also available on the Apache > Incubator wiki at: > https://wiki.apache.org/incubator/HeronProposal > > Please vote: > [ ] +1, bring Heron into Incubator > [ ] -1, do not bring Heron into Incubator, because... > > The vote will open for 7 days until Friday June 23 at 14:00 PT. > > Thank you > > 1 - > https://lists.apache.org/thread.html/fb91f527ef479bb5df45bf2c9d93b7786c3fa6cdbfeba3128599df79@%3Cgeneral.incubator.apache.org%3E > > > > = Heron Proposal = > > = Abstract = > Heron is a real-time, distributed, fault-tolerant stream processing engine > initially developed by Twitter. > > = Proposal = > > Heron is a real-time stream processing engine built for high performance, > ease of manageability, performance predictability and developer > productivity[1]. We wish to develop a community around Heron to increase > contributions and see Heron thrive in an open forum. > > = Background = > > Heron provides the ability for developers to compose directed acyclic > graphs (DAGs) of real-time query execution logic (i.e. a topology) and > submit the topology to execute on a pluggable job scheduling system (e.g., > Apache Aurora, YARN, Marathon, etc). Users can employ either the native > Heron API or the Apache Storm API to develop the topology. Heron supports > the Storm API for ease of migration, but beyond that Heron’s architecture > differs considerably from Storm’s. > > Users submit a topology to the scheduler using the Heron client, which uses > the Heron binary libraries to deploy all daemons required to run and manage > the topology. The topology therefore has no reliance on centrally managed > Heron services, only on a generic job scheduling system, which lends itself > well to be run on top of Apache Aurora/Mesos or Apache Hadoop/YARN (among > others). > > The scheduler runs each topology as a job consisting of multiple > containers. One of the containers runs the topology master, responsible for > managing the topology. The remaining containers each runs a stream manager > responsible for data routing, a metrics manager that collects and reports > various metrics and a number of processes called Heron instances which run > the user-defined logic on the stream of tuples. Parallelism is achieved via > process-based isolation of Heron instances, which provides predictable > performance while simplifying debugging. The containers are allocated and > managed by the scheduler framework based on resource availability of nodes > in the cluster. The metadata for the topology, such as the physical plan > and execution details, are stored in the pluggable Heron State Manager > (e.g. Apache ZooKeeper). > > = Rationale = > > Heron is a general-purpose, modular and extensible platform that can be > leveraged to support common, real-time analytics use cases. There is an > increasing demand for open-source, scalable real-time analytics systems. We > believe that Heron can be leveraged by other organizations to build > streaming applications that can benefit from its robustness, high > performance, adaptability to cloud environments and ease of use. Moreover, > we hope that open-sourcing Heron will help to further evolve the technology > as the project attracts contributors with diverse backgrounds and areas of > expertise. > > We believe the Apache foundation is a great fit as the long-term home for > Heron, as it provides an established process for community-driven > development and decision making by consensus. This is exactly the model we > want for future Heron development. > > = Initial Goals = > > * Move the existing codebase, website, documentation, and mailing lists to > Apache-hosted infrastructure. > * Integrate with the Apache development process. > * Ensure all dependencies are compliant with Apache License version 2.0. > * Incrementally develop and release per Apache guidelines. > > = Current Status = > > Heron is a stable project used in production at Twitter since 2014 and open > sourced under the ASL v2 license in 2016. The Heron source code is > currently hosted at github.com (https://github.com/twitter/heron), which > will seed the Apache git repository. > > = Meritocracy = > > By submitting this incubator proposal, we’re expressing our intent to build > a diverse developer community around Heron that will conduct itself > according to The Apache Way and use a meritocratic means of building it's > committer base. Several companies and universities have already expressed > interest in and contributed to Heron. Our goal is to grow the Heron > community by encouraging open communication, contribution and participation > of all types, and ensuring that contributors are recognized appropriately. > > = Community = > > Heron is currently being used by Twitter, Google, Machine Zone and > ndustrial.io and has received significant contributions by Microsoft and > Streamlio. By bringing Heron into the Apache ecosystem, we believe we can > attract even more developers who are interested in creating real-time > systems to build the project's contributor base. > > == Core Developers == > > Current core developers are engineers from Twitter, Google, Microsoft and > Streamlio. > > == Alignment == > > Heron utilizes a number of Apache technologies. Heron leverages Apache > ZooKeeper for coordination and has scheduler implementations to integrate > with Apache Mesos, Apache Aurora and Apache Hadoop's YARN (via Apache REEF) > as well as spout implementations to integrate with Apache Kafka and metrics > implementations to integrate with Scribe. Heron also implements the Apache > Storm user-level API, which allows topologies written against Storm to run > in Heron. We believe that having Heron at Apache will help further the > growth of the streaming compute community, as well as encourage cooperation > and developer cross pollination with other Apache projects. > > = Known Risks = > > == Orphaned Products == > > The risk of the Heron project being abandoned is minimal. It is used in > production at Twitter and Google and other companies are evaluating or > adopting it for production use. > > == Inexperience with Open Source == > > All of the core contributors to the project have considerable experience > with open source software development. Bill Graham[2], Ashvin Agrawal[3] > and Supun Kamburugamuve[4], committers on the project, are PMCs on other > Apache projects and Bill and Ashvin have gone through the Apache incubator > process. Twitter has already donated numerous projects to the ASF (e.g., > Apache Mesos, Apache Aurora, Apache Parquet). We also plan to be mentored > by experienced ASF members that can help with any roadblocks. > > == Homogenous Developers == > > Initial committers come from 5 separate organizations. Our intention is > increase the diversity of contributing developers and their affiliations. > To date github contributions have come from approximately 50 contributors > from outside the Twitter team. > > == Reliance on Salaried Developers == > > It is expected that Heron development will occur on both salaried time and > on volunteer time. The majority of initial committers are paid by their > employers to contribute to this project. We are committed to recruiting > additional committers from other organizations as well as non-salaried > committers to join project. > > == Relationships with Other Apache Products == > > As mentioned in the Alignment section, Heron implements the Apache Storm > API and integrates with multiple Apache schedulers (Apache Mesos, Apache > Aurora and Apache Hadoop's YARN) as well as Apache ZooKeeper and Apache > Thrift. > > == An Excessive Fascination with the Apache Brand == > > Heron's popularity is growing in the streaming compute space and we are > long time supporters of the Apache brand. This proposal is not for the > purpose of generating publicity through. Rather, the primary benefits to > joining Apache are those of community building and open decision making > outlined in the Rationale section. > > == Documentation == > > This proposal exists online as > http://wiki.apache.org/incubator/HeronProposal. Extensive documentation can > be found on github at https://twitter.github.io/heron and the source code > is well documented. > > == Source and Intellectual Property Submission Plan == > > The Heron codebase is currently hosted on Github: > https://github.com/twitter/heron. During incubation, the codebase will be > migrated to Apache infrastructure. The source code is already ASF 2.0 > licensed. > > == External Dependencies == > > All external libraries have ASF 2.0 compatible licenses except for pylint. > The pylint library is GPL licensed, but is only used for pre-build Python > style checks and is neither bundled with, nor relied upon by, the Heron > source or binary release artifacts. > > == Cryptography == > > Heron does not use any cryptography libraries. > > = Required Resources = > > == Mailing lists == > > * [hidden email] (with moderated subscriptions) > * [hidden email] > * [hidden email] > * [hidden email] > > == Subversion Directory == > > Git is the preferred source control system: git://git.apache.org/heron > > == Issue Tracking == > > JIRA: Heron (HERON) > > == Initial Committers == > > * Andrew Jorgensen (andrew at andrewjorgensen dot com) > * Ashvin Agrawal (ashvin at apache dot org)* > * Avrilia Floratou (avrilia dot floratou at gmail dot com) > * Bill Graham (billgraham at apache dot org)* > * Brian Hatfield (bmhatfield at gmail dot com) > * Chris Kellogg (cckellogg at gmail dot com) > * Huijun Wu (huijun dot wu dot 2010 at gmail dot com) > * Karthik Ramasamy (karthik at gmail dot com) > * Maosong Fu (maosongfu at gmail dot com) > * Neng Lu(freeneng at gmail dot com) > * Runhang Li (obj dot runhang at gmail dot com) > * Sanjeev Kulkarni (sanjeevrk at gmail dot com) > * Supun Kamburugamuve (supun at apache dot org)* > * Thomas Sun (tom dot ssf at gmail dot com) > * Yaliang Wang (yaliang dot w dot wang at ieee dot org) > > == Affiliations == > > * Andrew Jorgensen (Google) > * Ashvin Agrawal (Microsoft) > * Avrilia Floratou (Microsoft) > * Bill Graham (Twitter) > * Brian Hatfield (Google) > * Chris Kellogg (Twitter) > * Huijun Wu (Twitter) > * Karthik Ramasamy (Streamlio) > * Maosong Fu (Twitter) > * Neng Lu (Twitter) > * Runhang Li (Twitter) > * Sanjeev Kulkarni (Streamlio) > * Supun Kamburugamuve (Indiana University) > * Thomas Sun (Twitter) > * Yaliang Wang (Twitter) > > = Sponsors = > > == Champion == > > * Julien Le Dem (julien at apache dot org) > > == Nominated Mentors == > > * Jake Farrell (jfarrell at apache dot org) > * Jacques Nadeau (jacques at apache dot org) > * Julien Le Dem (julien at apache dot org) > * P. Taylor Goetz (ptgoetz at apache dot org) > > == Sponsoring Entity == > > The Apache Incubator > > == Footnotes == > > * 1 - Papers detailing Heron are available at > http://dl.acm.org/citation.cfm?id=2742788 and > http://sites.computer.org/debull/A15dec/p15.pdf. > * 2 - http://home.apache.org/phonebook.html?uid=billgraham > * 3 - http://home.apache.org/phonebook.html?uid=ashvin > * 4 - http://home.apache.org/phonebook.html?uid=supun --------------------------------------------------------------------- To unsubscribe, e-mail: [hidden email] For additional commands, e-mail: [hidden email] |
«
Return to Apache Incubator - General
|
1 view|%1 views
Loading...
| Free forum by Nabble | Edit this page |