[PROPOSAL] Pulsar - proposal for Apache Incubation

classic Classic list List threaded Threaded
35 messages Options
12
Reply | Threaded
Open this post in threaded view
|

[PROPOSAL] Pulsar - proposal for Apache Incubation

Joe Francis

Dear Apache Incubator Community,


We would like to submit the Pulsar proposal to the incubator.  Our draft is
available at:
    https://wiki.apache.org/incubator/PulsarProposal

A quick overview of Pulsar:

Pulsar is a highly scalable, low latency messaging platform running on
commodity hardware. It provides simple pub-sub semantics over topics,
guaranteed at-least-once delivery of messages, automatic cursor management for
subscribers, and cross-datacenter replication.

We are obviously looking for feedback and comments on the proposal, as well
as a few mentors.  Bryan Call has accepted to be our Champion.

Thank you,

-Joe Francis

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Pulsar - proposal for Apache Incubation

John D. Ament-2
Joe,

Its customary to bring in the entire body of the proposal in the discussion
thread.

Its good to see that someone stepped up to be Champion.  You still need
mentors.  Was Bryan planning to also mentor the project?  While he is a
foundation member, he is not on the IPMC.

Its great to see another messaging solution come to the ASF.  Does your
platform support any formats like AMQP, STOMP?  Are there chances to share
client libraries with ActiveMQ, Kafka or RocketMQ?

John

On Wed, Apr 26, 2017 at 5:38 PM Joe Francis <[hidden email]>
wrote:

>
> Dear Apache Incubator Community,
>
>
> We would like to submit the Pulsar proposal to the incubator.  Our draft is
> available at:
>     https://wiki.apache.org/incubator/PulsarProposal
>
> A quick overview of Pulsar:
>
> Pulsar is a highly scalable, low latency messaging platform running on
> commodity hardware. It provides simple pub-sub semantics over topics,
> guaranteed at-least-once delivery of messages, automatic cursor management
> for
> subscribers, and cross-datacenter replication.
>
> We are obviously looking for feedback and comments on the proposal, as well
> as a few mentors.  Bryan Call has accepted to be our Champion.
>
> Thank you,
>
> -Joe Francis
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Pulsar - proposal for Apache Incubation

Bryan Call-2
I wan’t planning on being a mentor, but I might be able to be convinced otherwise.

-Bryan


> On Apr 26, 2017, at 7:29 PM, John D. Ament <[hidden email]> wrote:
>
> Joe,
>
> Its customary to bring in the entire body of the proposal in the discussion
> thread.
>
> Its good to see that someone stepped up to be Champion.  You still need
> mentors.  Was Bryan planning to also mentor the project?  While he is a
> foundation member, he is not on the IPMC.
>
> Its great to see another messaging solution come to the ASF.  Does your
> platform support any formats like AMQP, STOMP?  Are there chances to share
> client libraries with ActiveMQ, Kafka or RocketMQ?
>
> John
>
> On Wed, Apr 26, 2017 at 5:38 PM Joe Francis <[hidden email]>
> wrote:
>
>>
>> Dear Apache Incubator Community,
>>
>>
>> We would like to submit the Pulsar proposal to the incubator.  Our draft is
>> available at:
>>    https://wiki.apache.org/incubator/PulsarProposal
>>
>> A quick overview of Pulsar:
>>
>> Pulsar is a highly scalable, low latency messaging platform running on
>> commodity hardware. It provides simple pub-sub semantics over topics,
>> guaranteed at-least-once delivery of messages, automatic cursor management
>> for
>> subscribers, and cross-datacenter replication.
>>
>> We are obviously looking for feedback and comments on the proposal, as well
>> as a few mentors.  Bryan Call has accepted to be our Champion.
>>
>> Thank you,
>>
>> -Joe Francis
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Pulsar - proposal for Apache Incubation

Dave Fisher-5
In reply to this post by John D. Ament-2
Hi,

Sent from my iPhone

> On Apr 26, 2017, at 7:29 PM, John D. Ament <[hidden email]> wrote:
>
> Joe,
>
> Its customary to bring in the entire body of the proposal in the discussion
> thread.
>
> Its good to see that someone stepped up to be Champion.  You still need
> mentors.  Was Bryan planning to also mentor the project?  While he is a
> foundation member, he is not on the IPMC.
>
> Its great to see another messaging solution come to the ASF.  Does your
> platform support any formats like AMQP, STOMP?  Are there chances to share
> client libraries with ActiveMQ, Kafka or RocketMQ?

Or not. Using other projects is not necessarily relevant to whether or not the proposed podling has a community.

I have to be careful about committing to Mentoring. I have some interest in such a project. I'd want gave a technical conversation.

Regards,
Dave


>
> John
>
> On Wed, Apr 26, 2017 at 5:38 PM Joe Francis <[hidden email]>
> wrote:
>
>>
>> Dear Apache Incubator Community,
>>
>>
>> We would like to submit the Pulsar proposal to the incubator.  Our draft is
>> available at:
>>    https://wiki.apache.org/incubator/PulsarProposal
>>
>> A quick overview of Pulsar:
>>
>> Pulsar is a highly scalable, low latency messaging platform running on
>> commodity hardware. It provides simple pub-sub semantics over topics,
>> guaranteed at-least-once delivery of messages, automatic cursor management
>> for
>> subscribers, and cross-datacenter replication.
>>
>> We are obviously looking for feedback and comments on the proposal, as well
>> as a few mentors.  Bryan Call has accepted to be our Champion.
>>
>> Thank you,
>>
>> -Joe Francis
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Pulsar - proposal for Apache Incubation

Matteo Merli
In reply to this post by John D. Ament-2
On Wed, Apr 26, 2017 at 7:30 PM John D. Ament <[hidden email]> wrote:

> Its great to see another messaging solution come to the ASF.  Does your
> platform support any formats like AMQP, STOMP?


There are currently many messaging APIs and wire protocol specifications,
but for Pulsar we have opted for a custom wire protocol and a client API.
The primary reasons for that are in the feature-set supported by each of
these APIs (each of them was lacking some feature we required) and how
using a particular API/protocol was enabling the implementation of a highly
scalable client/broker.

Building on top on the the native client API, we have implemented a
WebSocket based API and we plan to add support for MQTT as well. We would
be very happy to add support for additional protocol and APIs in the same
way.


> Are there chances to share
> client libraries with ActiveMQ, Kafka or RocketMQ?
>

Even though all these projects are in the "pub-sub" space, the data models
and semantics do have significant differences that make the task
complicated. I think that targeting a common client API is a much more
achievable goal and we would be very happy to contribute on that front.

Matteo
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Pulsar - proposal for Apache Incubation

Matteo Merli
In reply to this post by Dave Fisher-5
On Wed, Apr 26, 2017 at 7:49 PM Dave Fisher <[hidden email]> wrote:

> Or not. Using other projects is not necessarily relevant to whether or not
> the proposed podling has a community.
>
> I have to be careful about committing to Mentoring. I have some interest
> in such a project. I'd want gave a technical conversation.
>

Hi Dave,

I and other members of the team would be happy to have such technical
conversation, through any preferred channel.

Thanks,
Matteo
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Pulsar - proposal for Apache Incubation

Jochen Theodorou
In reply to this post by Joe Francis


On 26.04.2017 23:19, Joe Francis wrote:

>
> Dear Apache Incubator Community,
>
>
> We would like to submit the Pulsar proposal to the incubator.  Our draft is
> available at:
>     https://wiki.apache.org/incubator/PulsarProposal
>
> A quick overview of Pulsar:
>
> Pulsar is a highly scalable, low latency messaging platform running on
> commodity hardware. It provides simple pub-sub semantics over topics,
> guaranteed at-least-once delivery of messages, automatic cursor management for
> subscribers, and cross-datacenter replication.

Without really knowing the details of a messaging platform and even with
the danger of comparing apples and oranges... how would you compare for
example hazelcast based messaging to pulsar?

bye Jochen

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Pulsar - proposal for Apache Incubation

Xinyu Zhou
Hi Matteo,
Glad to see you and Pulsar are here.
John's idea is interesting and need to pay attention on it. There are so
many messaging systems in ASF or other organizations, share client
libraries between these products really make sense and will bring great
benefit for our user.
But as Matteo said, this task is very complicated, due to different models
and semantics. So we initiate a brand-new OpenMessaging specification, hope
this spec can establish a interconnected bridge between diverse messaging
solutions.
Regards, yukon Apache RocketMQ Committer
On Thu, Apr 27, 2017 at 4:41 PM, Jochen Theodorou <[hidden email]>
wrote:


On 26.04.2017 23:19, Joe Francis wrote:
 >
 > Dear Apache Incubator Community,
 >
 >
 > We would like to submit the Pulsar proposal to the incubator. Our draft
is
 > available at:
 > https://wiki.apache.org/incubator/PulsarProposal
 >
 > A quick overview of Pulsar:
 >
 > Pulsar is a highly scalable, low latency messaging platform running on
 > commodity hardware. It provides simple pub-sub semantics over topics,
 > guaranteed at-least-once delivery of messages, automatic cursor
management for
 > subscribers, and cross-datacenter replication.

Without really knowing the details of a messaging platform and even with
the danger of comparing apples and oranges... how would you compare for
example hazelcast based messaging to pulsar?

bye Jochen

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Pulsar - proposal for Apache Incubation

Jim Jagielski
In reply to this post by Joe Francis
This proposal looks like something I'd be interested in helping
out with... some basic questions: (1) How does the compare/contrast
w/ Apache Kafka and (2) What is seen as the most applicable use-case
for Pulsar?

Funny aside: the 1st new car I ever bought with my own money was
a Nissan Pulsar. It was a piece of crap :)

> On Apr 26, 2017, at 5:19 PM, Joe Francis <[hidden email]> wrote:
>
>
> Dear Apache Incubator Community,
>
>
> We would like to submit the Pulsar proposal to the incubator.  Our draft is
> available at:
>    https://wiki.apache.org/incubator/PulsarProposal
>
> A quick overview of Pulsar:
>
> Pulsar is a highly scalable, low latency messaging platform running on
> commodity hardware. It provides simple pub-sub semantics over topics,
> guaranteed at-least-once delivery of messages, automatic cursor management for
> subscribers, and cross-datacenter replication.
>
> We are obviously looking for feedback and comments on the proposal, as well
> as a few mentors.  Bryan Call has accepted to be our Champion.
>
> Thank you,
>
> -Joe Francis
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Pulsar - proposal for Apache Incubation

Bertrand Delacretaz
On Thu, Apr 27, 2017 at 2:20 PM, Jim Jagielski <[hidden email]> wrote:
> ...Funny aside: the 1st new car I ever bought with my own money was
> a Nissan Pulsar. It was a piece of crap :)...

Apache Pulsar (incubating): Making Pulsar Great Again ;-)

-Bertrand

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Pulsar - proposal for Apache Incubation

Jim Jagielski

> On Apr 27, 2017, at 8:29 AM, Bertrand Delacretaz <[hidden email]> wrote:
>
> On Thu, Apr 27, 2017 at 2:20 PM, Jim Jagielski <[hidden email]> wrote:
>> ...Funny aside: the 1st new car I ever bought with my own money was
>> a Nissan Pulsar. It was a piece of crap :)...
>
> Apache Pulsar (incubating): Making Pulsar Great Again ;-)
>

LOL.

"Apache Pulsar will never leave you stranded in the middle of
a tunnel at rush hour."



---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Pulsar - proposal for Apache Incubation

Pierre Smits
In reply to this post by Jim Jagielski
While considering the project/product interesting I am curious how the
proposers and champion see this becoming a success as an ASF
podling/project. Currently there seems to be only one adopter (Yahoo), and
for most the contributors are paid employees (of Yahoo).

Looking at the provided repo, and actions thereon, I see only 22
contributors referenced as contributors whose code has been committed. Are
not all interested in joining the podling, or don't they just want to be
referenced in the proposal as initial contributors/committers?

Looking at the list of initial contributors/committers I see Matteo Merli.
According to the proposal he is affiliated with an organisation called
Streaml.io. But according to LinkedIn he is affiliated with Yahoo. Which
one should be corrected.

Furthermore I see a lot of forks (250 up to today) of the main repo on
GitHub, yet I don't see a lot of activity (code changes) reverting back to
the main repo.

I fear this will be a podling with a very small base (adoption and
contribution wise).

Best regards,

Pierre Smits

ORRTIZ.COM <http://www.orrtiz.com>
OFBiz based solutions & services

OFBiz Extensions Marketplace
http://oem.ofbizci.net/oci-2/

On Thu, Apr 27, 2017 at 2:20 PM, Jim Jagielski <[hidden email]> wrote:

> This proposal looks like something I'd be interested in helping
> out with... some basic questions: (1) How does the compare/contrast
> w/ Apache Kafka and (2) What is seen as the most applicable use-case
> for Pulsar?
>
> Funny aside: the 1st new car I ever bought with my own money was
> a Nissan Pulsar. It was a piece of crap :)
>
> > On Apr 26, 2017, at 5:19 PM, Joe Francis <[hidden email]>
> wrote:
> >
> >
> > Dear Apache Incubator Community,
> >
> >
> > We would like to submit the Pulsar proposal to the incubator.  Our draft
> is
> > available at:
> >    https://wiki.apache.org/incubator/PulsarProposal
> >
> > A quick overview of Pulsar:
> >
> > Pulsar is a highly scalable, low latency messaging platform running on
> > commodity hardware. It provides simple pub-sub semantics over topics,
> > guaranteed at-least-once delivery of messages, automatic cursor
> management for
> > subscribers, and cross-datacenter replication.
> >
> > We are obviously looking for feedback and comments on the proposal, as
> well
> > as a few mentors.  Bryan Call has accepted to be our Champion.
> >
> > Thank you,
> >
> > -Joe Francis
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: [hidden email]
> > For additional commands, e-mail: [hidden email]
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Pulsar - proposal for Apache Incubation

Jim Jagielski
Certainly, this is one of the main aspects of Incubation surely,
to create a healthy community around a codebase and a project.

I personally don't think considerations of how large or small
a base is pertinent at this stage. The original Apache httpd
project had an *extremely* small initial base ;)

> On Apr 27, 2017, at 8:37 AM, Pierre Smits <[hidden email]> wrote:
>
> While considering the project/product interesting I am curious how the
> proposers and champion see this becoming a success as an ASF
> podling/project. Currently there seems to be only one adopter (Yahoo), and
> for most the contributors are paid employees (of Yahoo).
>
> Looking at the provided repo, and actions thereon, I see only 22
> contributors referenced as contributors whose code has been committed. Are
> not all interested in joining the podling, or don't they just want to be
> referenced in the proposal as initial contributors/committers?
>
> Looking at the list of initial contributors/committers I see Matteo Merli.
> According to the proposal he is affiliated with an organisation called
> Streaml.io. But according to LinkedIn he is affiliated with Yahoo. Which
> one should be corrected.
>
> Furthermore I see a lot of forks (250 up to today) of the main repo on
> GitHub, yet I don't see a lot of activity (code changes) reverting back to
> the main repo.
>
> I fear this will be a podling with a very small base (adoption and
> contribution wise).
>
> Best regards,
>
> Pierre Smits
>
> ORRTIZ.COM <http://www.orrtiz.com>
> OFBiz based solutions & services
>
> OFBiz Extensions Marketplace
> http://oem.ofbizci.net/oci-2/
>
> On Thu, Apr 27, 2017 at 2:20 PM, Jim Jagielski <[hidden email]> wrote:
>
>> This proposal looks like something I'd be interested in helping
>> out with... some basic questions: (1) How does the compare/contrast
>> w/ Apache Kafka and (2) What is seen as the most applicable use-case
>> for Pulsar?
>>
>> Funny aside: the 1st new car I ever bought with my own money was
>> a Nissan Pulsar. It was a piece of crap :)
>>
>>> On Apr 26, 2017, at 5:19 PM, Joe Francis <[hidden email]>
>> wrote:
>>>
>>>
>>> Dear Apache Incubator Community,
>>>
>>>
>>> We would like to submit the Pulsar proposal to the incubator.  Our draft
>> is
>>> available at:
>>>   https://wiki.apache.org/incubator/PulsarProposal
>>>
>>> A quick overview of Pulsar:
>>>
>>> Pulsar is a highly scalable, low latency messaging platform running on
>>> commodity hardware. It provides simple pub-sub semantics over topics,
>>> guaranteed at-least-once delivery of messages, automatic cursor
>> management for
>>> subscribers, and cross-datacenter replication.
>>>
>>> We are obviously looking for feedback and comments on the proposal, as
>> well
>>> as a few mentors.  Bryan Call has accepted to be our Champion.
>>>
>>> Thank you,
>>>
>>> -Joe Francis
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Pulsar - proposal for Apache Incubation

Bryan Call-2
In reply to this post by Pierre Smits
Having a lot of Yahoo employees on the Traffic Server Proposal back in 2009
was also a concern.  This is something we were able to easily overcome in
incubation.  Building a healthy diverse community is one of the project's top
priorities.

-Bryan

> On Apr 27, 2017, at 5:37 AM, Pierre Smits <[hidden email]> wrote:
>
> While considering the project/product interesting I am curious how the
> proposers and champion see this becoming a success as an ASF
> podling/project. Currently there seems to be only one adopter (Yahoo), and
> for most the contributors are paid employees (of Yahoo).
>
> Looking at the provided repo, and actions thereon, I see only 22
> contributors referenced as contributors whose code has been committed. Are
> not all interested in joining the podling, or don't they just want to be
> referenced in the proposal as initial contributors/committers?
>
> Looking at the list of initial contributors/committers I see Matteo Merli.
> According to the proposal he is affiliated with an organisation called
> Streaml.io. But according to LinkedIn he is affiliated with Yahoo. Which
> one should be corrected.
>
> Furthermore I see a lot of forks (250 up to today) of the main repo on
> GitHub, yet I don't see a lot of activity (code changes) reverting back to
> the main repo.
>
> I fear this will be a podling with a very small base (adoption and
> contribution wise).
>
> Best regards,
>
> Pierre Smits
>
> ORRTIZ.COM <http://www.orrtiz.com>
> OFBiz based solutions & services
>
> OFBiz Extensions Marketplace
> http://oem.ofbizci.net/oci-2/
>
> On Thu, Apr 27, 2017 at 2:20 PM, Jim Jagielski <[hidden email]> wrote:
>
>> This proposal looks like something I'd be interested in helping
>> out with... some basic questions: (1) How does the compare/contrast
>> w/ Apache Kafka and (2) What is seen as the most applicable use-case
>> for Pulsar?
>>
>> Funny aside: the 1st new car I ever bought with my own money was
>> a Nissan Pulsar. It was a piece of crap :)
>>
>>> On Apr 26, 2017, at 5:19 PM, Joe Francis <[hidden email]>
>> wrote:
>>>
>>>
>>> Dear Apache Incubator Community,
>>>
>>>
>>> We would like to submit the Pulsar proposal to the incubator.  Our draft
>> is
>>> available at:
>>>   https://wiki.apache.org/incubator/PulsarProposal
>>>
>>> A quick overview of Pulsar:
>>>
>>> Pulsar is a highly scalable, low latency messaging platform running on
>>> commodity hardware. It provides simple pub-sub semantics over topics,
>>> guaranteed at-least-once delivery of messages, automatic cursor
>> management for
>>> subscribers, and cross-datacenter replication.
>>>
>>> We are obviously looking for feedback and comments on the proposal, as
>> well
>>> as a few mentors.  Bryan Call has accepted to be our Champion.
>>>
>>> Thank you,
>>>
>>> -Joe Francis
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: [hidden email]
>>> For additional commands, e-mail: [hidden email]
>>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Pulsar - proposal for Apache Incubation

Joe Francis
In reply to this post by Joe Francis
Dave,
Would be happy to have a technical conversation, let us know how you would prefer to have it
Joe

    On Wednesday, April 26, 2017 7:49 PM, Dave Fisher <[hidden email]> wrote:
 

 Hi,

Sent from my iPhone

> On Apr 26, 2017, at 7:29 PM, John D. Ament <[hidden email]> wrote:
>
> Joe,
>
> Its customary to bring in the entire body of the proposal in the discussion
> thread.
>
> Its good to see that someone stepped up to be Champion.  You still need
> mentors.  Was Bryan planning to also mentor the project?  While he is a
> foundation member, he is not on the IPMC.
>
> Its great to see another messaging solution come to the ASF.  Does your
> platform support any formats like AMQP, STOMP?  Are there chances to share
> client libraries with ActiveMQ, Kafka or RocketMQ?

Or not. Using other projects is not necessarily relevant to whether or not the proposed podling has a community.

I have to be careful about committing to Mentoring. I have some interest in such a project. I'd want gave a technical conversation.

Regards,
Dave


>
> John
>
> On Wed, Apr 26, 2017 at 5:38 PM Joe Francis <[hidden email]>
> wrote:
>
>>
>> Dear Apache Incubator Community,
>>
>>
>> We would like to submit the Pulsar proposal to the incubator.  Our draft is
>> available at:
>>    https://wiki.apache.org/incubator/PulsarProposal
>>
>> A quick overview of Pulsar:
>>
>> Pulsar is a highly scalable, low latency messaging platform running on
>> commodity hardware. It provides simple pub-sub semantics over topics,
>> guaranteed at-least-once delivery of messages, automatic cursor management
>> for
>> subscribers, and cross-datacenter replication.
>>
>> We are obviously looking for feedback and comments on the proposal, as well
>> as a few mentors.  Bryan Call has accepted to be our Champion.
>>
>> Thank you,
>>
>> -Joe Francis
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]


   
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Pulsar - proposal for Apache Incubation

Leif Hedstrom-3
In reply to this post by Bryan Call-2

> On Apr 27, 2017, at 11:41 AM, Bryan Call <[hidden email]> wrote:
>
> Having a lot of Yahoo employees on the Traffic Server Proposal back in 2009
> was also a concern.  This is something we were able to easily overcome in
> incubation.  Building a healthy diverse community is one of the project's top
> priorities.


It’s worth pointing out that Yahoo! and Yahoo! Japan are two entirely separate entities, so the project already has two large organizations backing the project, and a few others as well. I see no real issue here with diversity.

Cheers,

— Leif


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Pulsar - proposal for Apache Incubation

Joe Francis
In reply to this post by Jim Jagielski




On Thursday, April 27, 2017 5:20 AM, Jim Jagielski <[hidden email]> wrote:



> This proposal looks like something I'd be interested in helping
> out with... some basic questions: (1) How does the compare/contrast
> w/ Apache Kafka and (2) What is seen as the most applicable use-case
> for Pulsar?

There are few design choices in which Pulsar differs from Kafka. These choices were primarily dictated by requirements for Pulsar to be a core low-level platform backing serving systems:
    * Multi-Tenant platform
    * Scale  to ~1M topics in one cluster
    * Guaranteed durability. Everything must be replicated and flushed to disk before acknowledgment.
    * Low publish latency (99pct <5ms) with that durability, even under conditions where dispatch has to do cold reads from disk
    * Geo-Replication as a 1st class feature

Some of these requirements are made possible by using Apache BookKeeper as the storage for Pulsar topics, and separating brokers from storage. There are several advantages in having 2 different layers (brokers + storage) in the system:
    * Shift traffic across different brokers (automatic load balancer, rapid failover)
    * Easily scale up each layer independently
    * New machines automatically start serving traffic
    * Use disk I/O effectively

> (2) What is seen as the most applicable use-case for Pulsar?

At Yahoo, Pulsar has been supporting a broad range of applications. Main usage patterns could be summarized as :
    * Message queues
    * Notification / Application Integration
    * Data source and sink for stream processing
    * Durable message bus for other platforms (storage, dbs)

In particular the low-publish latency and high-availability properties, make Pulsar suitable to be used in the critical data path of online event processing/serving. The multi-tenancy allows one organization to operate a single cluster serving multiple applications.


> Funny aside: the 1st new car I ever bought with my own money was

> a Nissan Pulsar. It was a piece of crap :)

Our intention is certainly to do better than that :)


We have been running Pulsar  in production at scale for about 2 years now, and more detailed write-up (though a little dated) is available here  https://yahooeng.tumblr.com/post/150078336821/open-sourcing-pulsar-pub-sub-messaging-at-scale

Cheers,
Joe

> On Apr 26, 2017, at 5:19 PM, Joe Francis <[hidden email]> wrote:
>
>
> Dear Apache Incubator Community,
>
>
> We would like to submit the Pulsar proposal to the incubator.  Our draft is
> available at:
>    https://wiki.apache.org/incubator/PulsarProposal
>
> A quick overview of Pulsar:
>
> Pulsar is a highly scalable, low latency messaging platform running on
> commodity hardware. It provides simple pub-sub semantics over topics,
> guaranteed at-least-once delivery of messages, automatic cursor management for
> subscribers, and cross-datacenter replication.
>
> We are obviously looking for feedback and comments on the proposal, as well
> as a few mentors.  Bryan Call has accepted to be our Champion.
>
> Thank you,
>
> -Joe Francis
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]

>

---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Pulsar - proposal for Apache Incubation

Dave Fisher-5
In reply to this post by Jochen Theodorou
Hi -

I am in California and could do a webex this Monday.

Regards,
Dave

Sent from my iPhone

> On Apr 27, 2017, at 1:41 AM, Jochen Theodorou <[hidden email]> wrote:
>
>
>
>> On 26.04.2017 23:19, Joe Francis wrote:
>>
>> Dear Apache Incubator Community,
>>
>>
>> We would like to submit the Pulsar proposal to the incubator.  Our draft is
>> available at:
>>    https://wiki.apache.org/incubator/PulsarProposal
>>
>> A quick overview of Pulsar:
>>
>> Pulsar is a highly scalable, low latency messaging platform running on
>> commodity hardware. It provides simple pub-sub semantics over topics,
>> guaranteed at-least-once delivery of messages, automatic cursor management for
>> subscribers, and cross-datacenter replication.
>
> Without really knowing the details of a messaging platform and even with the danger of comparing apples and oranges... how would you compare for example hazelcast based messaging to pulsar?
>
> bye Jochen
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [hidden email]
For additional commands, e-mail: [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Pulsar - proposal for Apache Incubation

Pierre Smits
In reply to this post by Leif Hedstrom-3
Yahoo! and Yahoo! Japan are NOT two entirely separate entities. According
to financial information (see [1] of 2016 Yahoo! held/holds 25.6% of the
companies shares. In some areas that is enough to constitute directional
power. Maybe, in this case, there are enough Chinese walls between the two
entities to consider the two separate. We'll probably never know...

Another aspect that might be more worrying is the recent deal between
Verizon and Yahoo! regarding the sale of the internet activities of Yahoo!
to Verizon. I wonder what the effect of this transaction will be on the
parties from Yahoo! intended to participate in this project. If the changes
mean that the parties now listed with Yahoo! won't be paid to do
development work on the product of the incubating project, this could mean
that the project loses more than half of initial contributor/committer base.



[1] https://about.yahoo.co.jp/ir/en/holder/status.html

Best regards,

Pierre

Pierre Smits

ORRTIZ.COM <http://www.orrtiz.com>
OFBiz based solutions & services

OFBiz Extensions Marketplace
http://oem.ofbizci.net/oci-2/

On Thu, Apr 27, 2017 at 10:35 PM, Leif Hedstrom <[hidden email]> wrote:

>
> > On Apr 27, 2017, at 11:41 AM, Bryan Call <[hidden email]> wrote:
> >
> > Having a lot of Yahoo employees on the Traffic Server Proposal back in
> 2009
> > was also a concern.  This is something we were able to easily overcome in
> > incubation.  Building a healthy diverse community is one of the
> project's top
> > priorities.
>
>
> It’s worth pointing out that Yahoo! and Yahoo! Japan are two entirely
> separate entities, so the project already has two large organizations
> backing the project, and a few others as well. I see no real issue here
> with diversity.
>
> Cheers,
>
> — Leif
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [hidden email]
> For additional commands, e-mail: [hidden email]
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [PROPOSAL] Pulsar - proposal for Apache Incubation

Pierre Smits
I meant to say 35.6% of the shares in my previous posting.

Pierre Smits

ORRTIZ.COM <http://www.orrtiz.com>
OFBiz based solutions & services

OFBiz Extensions Marketplace
http://oem.ofbizci.net/oci-2/

On Fri, Apr 28, 2017 at 9:12 AM, Pierre Smits <[hidden email]>
wrote:

> Yahoo! and Yahoo! Japan are NOT two entirely separate entities. According
> to financial information (see [1] of 2016 Yahoo! held/holds 25.6% of the
> companies shares. In some areas that is enough to constitute directional
> power. Maybe, in this case, there are enough Chinese walls between the two
> entities to consider the two separate. We'll probably never know...
>
> Another aspect that might be more worrying is the recent deal between
> Verizon and Yahoo! regarding the sale of the internet activities of Yahoo!
> to Verizon. I wonder what the effect of this transaction will be on the
> parties from Yahoo! intended to participate in this project. If the changes
> mean that the parties now listed with Yahoo! won't be paid to do
> development work on the product of the incubating project, this could mean
> that the project loses more than half of initial contributor/committer base.
>
>
>
> [1] https://about.yahoo.co.jp/ir/en/holder/status.html
>
> Best regards,
>
> Pierre
>
> Pierre Smits
>
> ORRTIZ.COM <http://www.orrtiz.com>
> OFBiz based solutions & services
>
> OFBiz Extensions Marketplace
> http://oem.ofbizci.net/oci-2/
>
> On Thu, Apr 27, 2017 at 10:35 PM, Leif Hedstrom <[hidden email]> wrote:
>
>>
>> > On Apr 27, 2017, at 11:41 AM, Bryan Call <[hidden email]> wrote:
>> >
>> > Having a lot of Yahoo employees on the Traffic Server Proposal back in
>> 2009
>> > was also a concern.  This is something we were able to easily overcome
>> in
>> > incubation.  Building a healthy diverse community is one of the
>> project's top
>> > priorities.
>>
>>
>> It’s worth pointing out that Yahoo! and Yahoo! Japan are two entirely
>> separate entities, so the project already has two large organizations
>> backing the project, and a few others as well. I see no real issue here
>> with diversity.
>>
>> Cheers,
>>
>> — Leif
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: [hidden email]
>> For additional commands, e-mail: [hidden email]
>>
>>
>
12