From 7d1ec110720ae27b229e4e5b69d7b084439bcb8a Mon Sep 17 00:00:00 2001 From: paulgear Date: Sat, 18 Mar 2006 12:36:41 +0000 Subject: [PATCH] I think am done with this. Feedback before i release it on an unsuspecting public, please! git-svn-id: https://shorewall.svn.sourceforge.net/svnroot/shorewall/trunk@3695 fbd18981-670d-0410-9b5c-8dc0c1a9a2bb --- docs/survey-200603.xml | 317 ++++++++++++++++++++--------------------- 1 file changed, 157 insertions(+), 160 deletions(-) diff --git a/docs/survey-200603.xml b/docs/survey-200603.xml index beb2ead39..d72a48897 100644 --- a/docs/survey-200603.xml +++ b/docs/survey-200603.xml @@ -38,75 +38,86 @@ Background In early March 2006, i embarked on the journey of surveying - Shorewall users. Initially this sprang from my own curiosity: it seemed to - me that some of the systems at work on which i was using Shorewall were - bigger and more complex than a lot of the ones others were using, and i - wanted to find out if there were people out there who used it like i did. - As started thinking about the questions i would ask, i realised that i - could ask a few more questions that might help us as a project to - understand a bit more about all of our users. + Shorewall users. Initially this sprang from my own curiosity: i thought + that some of the systems at work on which i use Shorewall may be bigger + and more complex than most others, and i wanted to find out if there are + people out there who use Shorewall like i do. As started thinking about + the questions i would ask, i realised that if i asked the right questions, + i could create a survey that might help the Shorewall project better to + understand its users. I used Zoomerang to - create the survey. It has a number of tools that make it really easy to - create useful surveys. To get the most benefit out of Zoomerang, you have - to subscribe to their professional version. In the long term, it would be - great to see a practical free software alternative that we could - self-host. A number of free content management systems such as Drupal have a survey module, but when i - last looked at them, they were much more limited and harder to use than - Zoomerang. + create the survey. It has a number of tools that make it easy to create + useful surveys. To get the most benefit out of Zoomerang, you have to + subscribe to their professional version. + + In the long term, it would be great to have a practical free + software alternative that could be self-hosted. A number of free content + management systems such as Drupal + have a survey module, but when i last looked at them, they were more + limited and harder to use than Zoomerang.
- Take the survey + Survey and results links The survey is still open as of this writing, and can be accessed at the Zoomerang survey page. Further participation is encouraged. The - figures quoted in this document reflect the latest results at the time - of writing. -
- -
- Survey results + figures quoted in this document reflect the results at the time of + writing. The public - results of the survey are also available. If you complete the - survey, a link to the results is provided on the thank you page. + results of the survey are available. If you complete the survey, + a link to the results is provided on the thank you page. +
+ +
+ Sample size + + An important note about this survey is that it has a small sample + size (103 complete responses at the time of writing), so any conclusions + drawn should be considered tentative. + + To speculate on the overall number of users that this sample + represents, the Debian popularity + contest reports 478 installations of Shorewall, 285 of which are + in active use. Assuming that the popularity contest represents 30% of + the Debian installed base (likely ridiculously optimistic), this would + make the number of active Shorewall systems approximately: + + 285 / 0.3 (percentage of Debian systems) / 0.26 (percentage Debian + holds of all distributions) = 3654 (rounding up the numbers to the + nearest whole, and assuming the percentages extrapolate + regularly) + + This means that our survey represents a maximum of 2.8% of the + installed base, likely far less. +
+ +
+ Other possible inaccuracies + + Additionally, since the survey was open to multiple responses, it + could be that some people answered the questions about themselves more + than once, despite instructions to the contrary in the introduction + page. + + There is an error in the released version of the survey for + question 15 (RAM size): it was a multiple choice question rather than + single choice, and thus there were more results than expected. The + number of errors doesn't seem to be significant. + + If you notice any errors in this analysis, or have any suggestions + about how to improve it, please contact the author at pgear@shorewall.net.
Results analysis - An important note about this survey is that it has a small sample - size (103 complete responses at the time of writing), so any conclusions - drawn should be considered tentative. - - To speculate on the overall number of users that this sample - represents, the Debian popularity - contest reports 478 installations of Shorewall, 285 of which are - in active use. Assuming that the popularity contest represents 30% of the - Debian installed base (likely ridiculously optimistic), this would make - the number of active Shorewall systems approximately: - - 285 / 0.3 (percentage of Debian systems) / 0.26 (percentage Debian - holds of all distributions) = 3654 (rounding up the numbers to the nearest - whole, and assuming the percentages extrapolate regularly) - - This means that our survey represents a maximum of 2.8% of the - installed base, likely far less. - - Additionally, since the survey was open to multiple responses, it - could be that some people answered the questions about themselves more - than once, despite instructions to the contrary in the introduction - page. - - If you notice any errors in this analysis, or have any suggestions - about how to improve it, please contact the author at pgear@shorewall.net. -
Organisations @@ -222,9 +233,9 @@ hardware, with a further 6% running it on x86-64/EM64T platforms. One response was received indicating use of Shorewall on MIPS (Linksys WRT platform). No responses were received for any other hardware platform. - While it is not surprising that Intel architectures would be first, - given their marketplace dominance, it seems a little skewed not to have - any representatives of other distributions. + While it is not surprising that Intel architectures would be dominant, + given their market share, it seems a little skewed not to have any + representatives of other distributions. A good spread of CPU power is shown in the survey responses. The largest group was 400-999 MHz (30%), with only 16% of responses @@ -238,11 +249,8 @@ bias towards higher RAM figures. The majority (52%) of systems have between 256 and 1023 MB; only 11% of systems have less than 128 MB; 28% have 1024 MB or more. This reflects the more server-oriented workload - that many Shorewall systems run (see the section on server roles below). - (Note that there is an error in the released version of the survey for - this question: it was a multiple choice question rather than single - choice, and thus there were more results than expected. However, the - number of errors doesn't seem to be significant.) + that many Shorewall systems run (see the section on server roles + below). Shorewall systems on the whole tend toward smaller OS hard disks, with 42% having disks 39 GB or smaller. The largest group by a small @@ -277,9 +285,9 @@ The message about maintaining an up-to-date Shorewall system seems to have gotten through, with 61% of respondents running the latest - version (3.0), and an additional 22% running the previous stable version - (2.4). Only 14% of users are running unsupported versions (2.2 and - older). + stable version (3.0), and an additional 22% running the previous stable + version (2.4). Only 14% of users are running unsupported older versions + (2.2 and previous). The most common roles played by Shorewall systems are: @@ -309,8 +317,8 @@
Comments from users - Here's a sample of the comments we received about the survey - - they have been carefully sanitised to make us look good. ;-) + Following is a sample of the comments we received about the survey + - they have been carefully sanitised to make us look good. ;-) @@ -357,6 +365,76 @@
+
+ Lessons learned about survey technique + +
+ Treat surveys like releasing free software + + + + test on a small group before you go public + + + + release early and often + + + + make branches (copies) when you release alpha and beta + versions + + + + merge the changes from branches (lessons you learned in those + versions) into the main trunk + + +
+ +
+ Start small and work towards what you want to know with specific, + concrete questions + + I tried to do everything in one survey, and ended up confusing + some people. For example, despite the fact that the survey's start page + clearly says "Please answer the questions for only ONE SYSTEM running + Shorewall", i received multiple comments saying that they couldn't + answer accurately because they ran more than Shorewall system. + + It would have been better to have two surveys: one about the + people who use Shorewall, and another about the systems they run it on. + Better still would be for Shorewall to automatically collect appropriate + information about systems and request permission to send it to a central + location for statistical analysis. How to do this and maintain users' + privacy and obtain their permission efficiently is not an easy problem + with a product like Shorewall, which doesn't actually stay running on + user systems, and doesn't present a user interface per se. +
+ +
+ Be prepared beforehand + + Within hours of the survey's release, 50% of the results were in. + Within 3 days, it hit the Zoomerang basic survey limit of 100 responses. + I had not planned for such an enthusiastic response, and also was too + busy to download all of the results before the survey's time limit + expired. Fortunately, i was able to obtain funding to allow a Zoomerang + "pro" subscription to be purchased and thus provide advanced analysis, + and complete downloads of the results. +
+ +
+ Incrementally improve your surveys + + The final version of this survey was released still with a few + bugs. The released version was just a copy of my master survey, and i + continued to maintain the master after the final survey was released + (and during this analysis), and i'm sure the next version will be even + better. +
+
+
Possible implications for the Shorewall project @@ -370,7 +448,7 @@ Shorewall seems to be predominantly used in small, i386-based environments such as home LANs and small businesses. It seems to be frequently combined with a number of other basic functions, such as DNS, - DHCP, NTP, VPN. Integration with (or perhaps providing a plugin module + DHCP, NTP, VPN. Integration with (or perhaps providing a plug-in module for) a dedicated gateway distribution such as ipcop, Smoothwall, or Clark Connect might be a good way to serve the needs of our users.
@@ -378,100 +456,19 @@
Possible implications for other free software projects - -
+ + + The essence of free software is software by the people, for the + people. Knowing who the people are and what their needs are is + critical to this process. + -
- Lessons learned about surveys - - - -
- What i did right - - - - - - Treat surveys like releasing free software: - - - - release early and often - - - - test on a small group before you go public - - - - make branches (copies) when you release alpha and beta - versions - - - - merge the changes from branches (lessons you learned in - those versions) into the main trunk - - - - -
- -
- What i did wrong - - - - - - Start small and work towards what you want to know. - - I tried to do everything in one survey, and ended up confusing - some people. For example, despite the fact that the survey's start - page clearly says "Please answer the questions for only ONE SYSTEM - running Shorewall", i received multiple comments saying that they - couldn't answer accurately because they ran more than Shorewall - system. - - It would have been better to have two surveys: one about the - people who use Shorewall, and another about the systems they run it - on. Better still would be for Shorewall to automatically collect - appropriate information about systems and request permission to send - it to a central location for statistical analysis. How to do this - and maintain users' privacy and obtain their permission efficiently - is not an easy problem with a product like Shorewall, which doesn't - actually stay running on user systems, and doesn't present a user - interface per-se. - - - - Be prepared beforehand. - - Within hours of the survey's release, 50% of the results were - in. Within 3 days, it hit the Zoomerang basic survey limit of 100 - responses. I had not planned for such an enthusiastic response, and - also was too busy to download all of the results before the survey's - time limit expired. Fortunately, i was able to obtain funding to - allow a Zoomerang "pro" subscription to be purchased and thus - provide advanced analysis, and complete downloads of the - results. - - - - -
-
- -
- - - -
- -
- - - + + If at all possible, build statistics gathering into your + application, and find a way to encourage people to use it. This + concrete data will help confirm the results of any surveys you might + conduct. + +
\ No newline at end of file