Create a contest to encourage users to test new SOVRYN features on testnet

I understand what you are explaining.

Maybe the system can exercise a natural selection of the best bug hunters. Surely it is important to be practical.

Let’s look at other opinions from other forum members…

Thank you!

1 Like

I think there’s a really good proposal in the making here, good job @juliomoros… Forgive my ignorance but how long are the Sovryn sprints (makes a difference in assessing how reasonable the prizes are)?

My only suggestion is to consider a small additional prize, determined by a lottery of all addresses that submitted a report of sufficient quality (there would need to be a minimum number of entries and the lottery can’t be won by those already claiming a prize)… Just think there’s value in providing a small incentive to those who look but don’t necessarily find.

1 Like

A sprint last two weeks… The lottery would imply a smart contract to be developed… And an additional budget, perhaps for a version N° 2 of this proposal.

1 Like

I love it! thank you for giving us another opportunity team!!

Well I think is time to be preparing the formal SIP… wish me luck! :nerd_face:

3 Likes

i think the sip should include a link to the discord channel #user-feedback or #bounties for reporting

1 Like

A branch for this proposal has been created. I hope to receive some comments (SIP in the DRAFT Stage)

The proposal is this:

The amount of work to be done by the jurors will climb into hours if not dozens of hours per month as time goes on, it will be hard to motivate the jurors to properly do the due diligence the check every report, then criticize the bug report, score every aspect of the bug report, check if the user signed a message, then talk to the other jurors and once you finally come to a conclusion you have to sign a multisig to send the bounty.

Point is that you are asking alot from these jurors for what is basically volunteer work, and at that you are giving 3 people the power to distribute 440 SOV a month (currently around 2k usd). If they are not financially incentivized, collusion might be an issue.

I agree with you that should be there an incentive for jurors; and the common answer I’ve received from most of the people I’ve asked in private to be one of them is that they are too busy… We are all supposed to be very busy, and this extra activity as jurors, is outside our duties within Sovryn day-by-day work.

By the other hand, I know the Sovryn team, and all of us are amazing… (my personal opinion), and I think all of us love our work, and the reason why we are here, and I see a collution as something very unlikely; however, the extra effort it would take should be incentivized; as @yago mentioned in the last community call here: https://youtu.be/X7_rqBdAQIo?t=2719

I may suggest to award each jury member with 100 SOV tokens per month, for a total of 300 SOV tokens per month for jury members.

Another important thing is to make this a proposal just for 1 quarter; e.g. from April the 1st to June the 30th. Things in Sovryn are changins too fast as to pretent set a monthly undefinite program like this.

I’d like to see some comments before include these changes in the SIP-0013 draft. Thank you very much.

2 Likes

Disclosure: @juliomoros asked me to be one of the jurors, and these are the questions i raised which he suggested i post to the forum for consideration by others:

  1. There DOES seem to be quite a bit of focus on form and process, are you sure you’re not going a bit overboard? Could this be simplified to decrease the workload, as mentioned by sway? Regardless of compensation incentives, this could really be a lot of work that could create conflicting priorities in a juror’s responsibilities.

    • how many bug bounty reports do you estimate receiving realistically per sprint?
    • could we combine the process to only happen every second sprint, but combine the sprints in the judging?
  2. are you SURE that we are better running a bug bounty program ourselves rather than using gitcoin?

    • i guess an argument for running a testnet bounty contest is that we identify bugs and inconsistencies in the UI / UX earlier, and identify exploits earlier saving the project time and money when it’s a mainnet deploy, but why would we use such a program, if in place, only for testnet?
    • One of the things you mention is using the BBP as a hiring pipeline for obviously talented people, and indeed, that has bounty programs have proven to be one of the best ways for onboarding talent in the cryptosphere, which makes me think that your proposal would best be serve sovryn by being expanded in scope, rather than minimized to a single aspect of our products / code.
    • @yago reached out to @stefan regarding using gitcoin as a hiring pipeline last week, who reached out to me. The thing about running bounties over gitcoin is that it needs someone dedicated to managing the user account, process, content, as well as promoting it’s existence on social media channels etc to get traction and attention. Just “having” bounties doesn’t mean they get looked at - even here one faces competition for available resources and attention.

So, when i see your dedication to (and the quality of) this proposal, i’m starting to think that it should be expanded in scope and budget, and run over gitcoin as a monthly program incorporating the two sprints

Would you be willing to be responsible for this, if we did so? Would you have the TIME to do so amongst your other responsibilities for Sovryn?

Take a look at the gitcoin process here: Quickstart | Gitcoin , b/c it covers all of the necessary steps we’d have to duplicate for an internal program.

3 Likes
  1. Running an inhouse test-net bounty program is a project in itself that needs a dedicated team to analyse any incoming reports. Nevertheless, it highly desirable to have one such program running in order to prevent any errors that might affect the project reputation
  2. It makes more sense to award the prizes on a monthly basis. Firstly, that is a standard promotion structure used in various industries across the board and, secondly, it is easier to manage it from the operational perspective. In order to sweeten the deal, the prize structure can be expanded to 6 monthly rewards.
  3. I would not market such a “competition” as a permanent one right from the start. I would rather announce it to last for a couple of months and reconsider it further down the line.
  4. Gitcoin is a platform that might make the entire process easier for the reviewing team. Also, that will facilitate the program expansion to all bugs categories. But, that needs an active and tech-savvy admin.

Definitely agree with this proposal. Regular public testing should be encouraged and incentivised. I like the revised reward amounts but agree with other comments that there should be a scoring system, with a minimum threshold to pass eligibility for a reward.
Whether this is the right approach or not, I believe there should be some sort of test system in place going forward.

As for the comic sans. I would like to suggest a separate proposal SIP to make all of Sovryn use comic sans from now on.

2 Likes

Questions

  • Do users actually need this incentive? How has the volume and quality of user-generated bug reports been so far, without this incentive?

  • Could the need for high quality bug reports be adequately fulfilled with an internal QA role that we do not have to micromanage as much as this contest program?

  • If in any case we want to involve the community in the bug hunting process and reward their participation, would they be just as motivated with nonmonetary rewards e.g. unique NFTs, special Discord badges, etc?

Alternative workflow

I echo some of the other comments that suggest the workflow seems a bit overengineered for the task at hand. I don’t mean to denigrate the proposal, I think it’s well thought out it’s just that for such low amounts of value on the line I think there may be too much work involved for the effort to really be worth it.

Here’s an alternate workflow I came up with that might be “good enough” to at least test and validate the concept before putting further effort into refinement:

  • Users are given a reporting period during which they can find and submit bug reports as new issues in the appropriate GitHub repo, with their RSK address included at the bottom of the report.

  • A dev will apply the label “validated” if the bug can be reproduced.

  • Bugs that cannot be reproduced will have their issues closed. Duplicate issues will also be closed, although if they include some helpful information that the first issue did not have then they may be considered for the prize pool as well.

  • At the end of the reporting period, the RSK addresses included on all validated bug reports (and duplicates considered helpful) will go into a random drawing for 1st, 2nd, and 3rd place prizes to be sent to the winning RSK addresses.

4 Likes

Hi, here are my thoughts:

Based on the significant activity observed in the Discord server where users are currently reporting issues that are probably the results of their own tests directly in Discord without any special reward, we must think very carefully on what type of issues we would like to receive through this initiative.

This is very relevant because once you put SOVs as a reward, and if this is on the high side, you will have or may have a lot of submissions and that means, a lot of work for the jury to thoroughly review. What will be the reward to the Jury and why would the jury invest those hours there instead of contributing to the platform?.

If we think there is a special type of test that does not exactly fit the bug bounty requirement and does not exactly fit the type of issues reported through the discord channel, and we all believe are very relevant to the platform then I would agree with thinking of some sort of incentive that can be SOVs or NFts or something else

But at this point, I’m not sure what type of tests or issues we expect to find that is very different than the regular reports in #tech-support or #user-feedback channels for example?. If there are such then finding incentives is totally fine

1 Like

Hi, I have carefully read all the posts, very valuable. I am writing these lines to you:

I believe that the formality and efforts required to file a bug report are so demanding that anyone will not know how to do it and want to do it unless they are very committed. Indeed, I believe that few would do it if they were not incentivised to do so. Sovryn needs to incentivise users to find those bugs as soon as possible, in my view.

It’s one thing to report a bug on discord simply in a few words, but it’s quite another to explain it properly by following Julio’s proposed instructions.

I don’t think there will be many bug reports, because the developers are doing their work very well, and as these problems are discovered and fixed, the occurrence of bugs will decrease. However, if I am wrong and there are many bug reports, they are welcome, and that will justify the resources invested in discovering them, assessing them and solving them.

Regarding the quantity and form (SOV, NFT) or the frequency of the awards, I have full confidence in the judgement of the Sovryn team. I also trust the system they choose to value them. Actually for me it is more important to solve them. In my opinion, it would be good if the user could choose the incentive they want, whether monetary or not, if given the option.

On the other hand, the only thing that seems odd to me is to create periods with a start and an end for the reporting of a bug. From my point of view a bug that is discovered today should be reported today and fixed quickly. If I take a fortnight to write a brilliant report, for me it is not as important as revealing it as soon as possible, even if the way I make it public is not as spectacular.

On a day-to-day basis, perhaps a separate B-feedback channel could be opened in the discord, to differentiate normal queries and comments related to the use of the platform, from those caused by bugs in the platform. Perhaps some users will make simple alerts in that new B channel, which can be dealt with quickly before it leads to a reward report. Or if someone has already reported it before, the users of that channel themselves can help by explaining that the problem is known, has been reported, or is in the process of being reported and fixed. This reduces the possibility of ghost bugs, already reported bugs, or repeated bug reports, and avoids extra work for the jury by reducing background noise.

All in all, I consider this bug-finding programme to be very good, because it allows users to help Sovryn and be rewarded by Sovryn.

2 Likes

I think the key message some of us were trying to share is that, this effort would be worth if the type of bugs we are referring to are different than the ones included in the Bug Bounty that already exist and also different than the bugs that are being reported daily on Discord for example.
I’ve seen people doing the math on how much they were expecting to get and reporting that in Discord when numbers were not adding up etc.

So if, you foresee an area with possible issues not being covered between those two programs, then I agree there is an opportunity for another program but we may need to:

  • Agree that there is an important type of issue not covered
  • Define a template so the user that submits the report does a thorough job which at the same time filters out most of the spam
  • I think that users can submit at anytime, but the SOV reward should be given with a certain cadence so there are enough reports to be compared and selected as winners
  • Define who would be doing that job of reviewing and replicating those reports. Having the template will be critical because the reviewer should need to be able to replicate the error

But again, this is only valid if we think there are a number of issues not included in the bug bounty and as part of the error/support logs we get in Discord

2 Likes

I have been reading the proposal and each comment carefully and these are some thoughts.

-I totally agree with compensating early users who take their time to test the platform or who were victims of some bugs of it.
It is the early users who take risks and give value to the platform with their criticisms and discoveries.
It is vitally important to discover and correct all bugs or improve the platform in all aspects before being exposed to the public and running the risk of being exploited or criticized. Reputation is essential in the valuation of any platform.

-I disagree with some comments that suggest that users should not be rewarded financially for their discoveries or only be rewarded with NFT (speculative asset), because time invested is money. Time is the most valuable asset.
Maybe not many bugs/improvements are reported yet because not many people are actually using the platform.
I see a lot of people desperate for the SOV token but not many asking questions about the platform itself.
Myself, at the beginning of February, have had a 5X long with almost no loan, a short with an “opening price” lower than what is shown in the interface, and a few days ago a 5X short that should have more than 25% of gains that all the time it was negative it was negative because the “current price” (2,3k higher than market price) is not the one that really uses the platform (also higher) and other things that I was reporting.

-There is an aspect that I think it is the most important:
Users will only report bugs that harm them, they will NEVER report bugs that they can take advantage of, unless they have a greater incentive to report it (sufficient amount of SOVs). Game theory.

-Regarding the workload for the juries, an intermediate level could be included that is in charge of filtering the reports (duplicates, badly written, etc.) so that the jury receives fewer reports.
“Filtrator”'s tasks should include quick review of reports, discard the repeated ones, maintain a shared spreadsheet with incidents already reported, select those that qualify for the contest and forward them to the jury.

1 Like

Who would you suggest having as this intermediary level?

Someone who cares about the project and is properly SOV-monetarily incentivized…

Thank you very much for your comments; I really appreciate them very much!

It gives me to understand that all of you actually took time to understand it and who appreciate the importance of this proposal.

1.- Part of why it took me a while to answer is because I have carefully read all the objections, in detail. I took the time to analyze each response and each recommendation.

Unfortunately many of the suggestions do not solve the challenges that this proposal seeks to address.

2.- As a conclusion I have determined that the following worrisome issues do exist:

  • What happens if too many reports are presented?
    It is an unknown parameter and the fear of receiving let’s say 3,000 reports for a contest is not an unfounded fear.

However: A “RANDOM SELECTION” OF REPORTS exposes us to discard high-quality ones, to simply reduce the number of reports to read.

THAT’S WHY: It makes sense to include a “Leaker” in the proposal with an incentive of 100 SOV per month.

If at the end of the Quarter no more than 30 reports per contest are received, this position will be eliminated.

  • These contests are based on a continuous process.
    There is a more or less continuous rate of issue reports that is expected to be translated into a continuous rate of user reports.

THAT IS WHY A MONTHLY CONTEST will only make us read twice as many reports as in a bi-weekly contest; In addition, we expose ourselves to receiving reports far from the opportune moment. It is not desirable to receive a report of something that should have been tested 10 days ago. Sovryn is evolving at a very fast pace to expose us to that risk.

WHILE THERE ARE INCENTIVES to produce more reports by offering rewards for them; there is also the extra effort of the user to make a minimally acceptable report.

These are forces that tend to cancel each other out, producing as a result a number of reports not likely higher than the number of incidents that users report to us on a daily basis.

  • A challenge posed by this proposal is to take advantage of user experience to anticipate the discovery of bugs in immunefi, which may results in significant savings for Sovryn.

THIS IS WHY GIT COIN does not provide added value:

a. Git Coin cannot offer rewards on SOV token, and they will charge their commission.
b. We already have a payment mechanism with the MultiSigWallet. We don’t need the one from Git Coin.
c. Git Coin requires an additional learning curve and admin functions (more seemingly unnecessary work, for Sovryn).
d. Git Coin is a tool, not an extra helper. (Git Coin doesn’t come with a funny dwarf in a box ready to do our work for us.)
e. Git Coin is ANOTHER platform with ITS own problems, with which we are going to bring our users face to face, instead of receiving them directly through our means; that complicates the picture more, especially for the user.

  • It is not intended to create another one bug hunt, but a more complete user feedback; every day there are reports of issues brought by users, which will always be there because they are complaints about malfunctions of the platform, which is under evolving process.

These reports are free, yes, and they are so at the expense of having to chat with the user (which is very time consuming) to find out what is going on. The reports do not replace this, but they are expected to mitigate the situation. Reports can also enrich our Wiki with additional information.

A report saves Sovryn staff chat time. But, in any case, attending the user is a job that we cannot avoid.

3.- I’d love to have the time to respond to all the others concerns that were expressed, especially those of @light , @stefan and @exiledsurfer ; I am very grateful for your suggestions; unfortunately, for the reasons I have just mentioned and others, in my opinion those suggestions would not contribute to a real improvement towards the efficiency of the process already proposed.

4.-I am going to mention the latest modifications of SIP-0013.
(I know that due the number, no one will want to take me away the label … :grin:)

4.1. It is indicated (I did not do it originally) that the report must be made in English.

4.2. The report must include an Abstract, of no more than 300 words, with the headlines very summarized of what was covered in the report.

4.3. The report must include a conclusion, of no more than 400 words, with highly summarized headlines about what was discovered or recommended in the report.

4.4. A Leaker will be on the work team on a trial basis only for this quarter period. The aim of this “Leaker” is to save work to the jury.

4.5. The main responsibility of the leaker will be to maintain an excel sheet with the list of reported incidents

4.6. The Leaker will also have the responsibility to discard reports that do not meet the minimum required quality.

4.7. The Leaker will discard repeated reports, giving priority to the first to appear.

The Leaker can be any member of our community and will be selected from among those who show interest in carrying out the task, by the members of the jury.

Thanks to everyone for your interest

1 Like