From 1c5f525b5807c9b0225f3eea2f36b16cd3907503 Mon Sep 17 00:00:00 2001 From: Garret Wassermann Date: Fri, 10 Feb 2017 16:42:40 -0500 Subject: [PATCH 01/25] Update doc index to say Django 1.8+ required, instead of 1.7+ --- docs/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/index.rst b/docs/index.rst index 483d5d84..e2fab98a 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -70,6 +70,6 @@ Dependencies ------------ 1. Python 2.7+ (or 3.3+) -2. Django (1.7 or newer) +2. Django (1.8 or newer) 3. An existing **working** Django project with database etc. If you cannot log into the Admin, you won't get this product working! This means you **must** run `syncdb` **before** you add ``helpdesk`` to your ``INSTALLED_APPS``. From ea30d1845e69c3982dd16e65669c4e5d204dc0c1 Mon Sep 17 00:00:00 2001 From: Garret Wassermann Date: Fri, 10 Feb 2017 23:20:59 -0500 Subject: [PATCH 02/25] Moved contributing.rst to CONTRIBUTING in the top-level directory, added language on required tests for new pull requests, for #449 --- docs/contributing.rst => CONTRIBUTING | 27 +++++++++++++++++++-------- 1 file changed, 19 insertions(+), 8 deletions(-) rename docs/contributing.rst => CONTRIBUTING (57%) diff --git a/docs/contributing.rst b/CONTRIBUTING similarity index 57% rename from docs/contributing.rst rename to CONTRIBUTING index e4cff162..d5201216 100644 --- a/docs/contributing.rst +++ b/CONTRIBUTING @@ -6,15 +6,13 @@ django-helpdesk is an open-source project and as such contributions from the com Licensing --------- -All contributions to django-helpdesk must be under the BSD license documented in our :doc:`license` page. All code submitted (in any way: via e-mail, via GitHub forks, attachments, etc) are assumed to be open-source and licensed under the BSD license. +All contributions to django-helpdesk must be under the BSD license documented in the LICENSE file in the top-level directory of this project. By submitting a contribution to this project (in any way: via e-mail, via GitHub forks, attachments, etc), you acknowledge that your contribution is open-source and licensed under the BSD license. If you or your organisation does not accept these license terms then we cannot accept your contribution. Please reconsider! Translations ------------ -.. image:: http://www.transifex.net/projects/p/django-helpdesk/resource/core/chart/image_png - Although django-helpdesk has originally been written for the English language, there are already multiple translations to Spanish, Polish, and German and more translations are welcomed. Translations are handled using the excellent Transifex service which is much easier for most users than manually editing .po files. It also allows collaborative translation. If you want to help translate django-helpdesk into languages other than English, we encourage you to make use of our Transifex project: @@ -26,22 +24,35 @@ Once you have translated content via Transifex, please raise an issue on the pro Code changes ------------ -Please fork the project on GitHub, make your changes, and log a pull request to get the changes pulled back into my repository. +Please fork the project on GitHub, make your changes, and log a pull request to get the changes pulled back into the +master branch of the django-helpdesk repository. Wherever possible please break git commits up into small chunks that are specific to a single bit of functionality. For example, a commit should not contain both new functionality *and* a bugfix; the new function and the bugfix should be separate commits wherever possible. Commit messages should also explain *what*, precisely, has been changed. -If you have any questions, please start a discussion on the GitHub issue tracker at https://github.com/django-helpdesk/django-helpdesk/issues +All commits should include appropriate new or updated tests; see the Tests section below for more details. + +If you have any questions, please start a discussion on the GitHub issue tracker at + +https://github.com/django-helpdesk/django-helpdesk/issues Tests ----- -Currently, test coverage is very low. We're working on increasing this, and to make life easier we are using `Travis CI`_ for continuous integration. This means that the test suite is run every time a code change is made, so we can try and make sure we avoid basic bugs and other regressions. +Currently, test coverage is very low. We're working on increasing this, and to make life easier we are using `Travis CI` (http://travis-ci.org/) for continuous integration. This means that the test suite is run every time a code change is made, so we can try and make sure we avoid basic bugs and other regressions. + +As a general policy, we will only accept new feature commits if they are accompanied by appropriate unit/functional tests (that is, tests for the functionality you just added). Bugfixes should also include new unit tests to ensure the bug has been fixed. + +More significant code refactoring must also include proper integration or validation tests, to be committed BEFORE the refactoring patches. This is to ensure that the refactored code produces the same results as the previous code base. + +Any further integration or validation tests (tests for the entire django-helpdesk application) are not required but greatly appreciated until we can improve our overall test coverage. Please include tests in the ``tests/`` folder when committing code changes. -.. _Travis CI: http://travis-ci.org/ +If you have any questions about creating or maintaining proper tests, please start a discussion on the GitHub issue tracker at + +https://github.com/django-helpdesk/django-helpdesk/issues Database schema changes ----------------------- @@ -50,6 +61,6 @@ As well as making your normal code changes to ``models.py``, please generate a D ./manage.py migrate helpdesk --auto [migration_name] -Make sure that ``migration_name`` is a sensible single-string explanation of what this migration does, such as *add_priority_options* or *add_basket_table* +Make sure that ``migration_name`` is a sensible single-string explanation of what this migration does, such as *add_priority_options* or *add_basket_table*. This will add a file to the ``migrations/`` folder, which must be committed to git with your other code changes. From 6fec1015c39ee7447dc2e9f7705b2c1c66cb08d5 Mon Sep 17 00:00:00 2001 From: Garret Wassermann Date: Wed, 15 Feb 2017 18:51:20 -0500 Subject: [PATCH 03/25] Work in progress on unicode chars --- .travis.yml | 3 ++- helpdesk/management/commands/get_email.py | 23 +++++++++------- helpdesk/tests/test_get_email.py | 33 +++++++++++++++++++---- 3 files changed, 43 insertions(+), 16 deletions(-) diff --git a/.travis.yml b/.travis.yml index 62879e4d..37e94b6b 100644 --- a/.travis.yml +++ b/.travis.yml @@ -4,6 +4,7 @@ python: - "2.7" - 3.4.4 - "3.5" + - "3.6" env: - DJANGO=1.8.17 @@ -18,7 +19,7 @@ install: before_script: - "pep8 --exclude=migrations --ignore=E501 helpdesk" -script: +script: - coverage run --source='.' quicktest.py helpdesk after_success: diff --git a/helpdesk/management/commands/get_email.py b/helpdesk/management/commands/get_email.py index 97b5b8cc..68e74ecb 100644 --- a/helpdesk/management/commands/get_email.py +++ b/helpdesk/management/commands/get_email.py @@ -10,6 +10,7 @@ scripts/get_email.py - Designed to be run from cron, this script checks the helpdesk, creating tickets from the new messages (or adding to existing tickets if needed) """ +from __future__ import unicode_literals from datetime import timedelta import email @@ -158,7 +159,7 @@ def process_queue(q, logger): msgNum = msg.split(" ")[0] logger.info("Processing message %s" % msgNum) - full_message = "\n".join(server.retr(msgNum)[1]) + full_message = encoding.force_text("\n".join(server.retr(msgNum)[1]), errors='ignore') ticket = ticket_from_message(message=full_message, queue=q, logger=logger) if ticket: @@ -198,7 +199,8 @@ def process_queue(q, logger): for num in msgnums: logger.info("Processing message %s" % num) status, data = server.fetch(num, '(RFC822)') - ticket = ticket_from_message(message=encoding.smart_text(data[0][1], errors='replace'), queue=q, logger=logger) + full_message = encoding.force_text(data[0][1], errors='ignore') + ticket = ticket_from_message(message=full_message, queue=q, logger=logger) if ticket: server.store(num, '+FLAGS', '\\Deleted') logger.info("Successfully processed message %s, deleted from IMAP server" % num) @@ -218,7 +220,8 @@ def process_queue(q, logger): for i, m in enumerate(mail, 1): logger.info("Processing message %d" % i) with open(m, 'r') as f: - ticket = ticket_from_message(message=f.read(), queue=q, logger=logger) + full_message = encoding.force_text(f.read(), errors='ignore') + ticket = ticket_from_message(message=full_message, queue=q, logger=logger) if ticket: logger.info("Successfully processed message %d, ticket/comment created." % i) try: @@ -243,10 +246,10 @@ def decodeUnknown(charset, string): if type(string) is not str: if not charset: try: - return str(string, encoding='utf-8', errors='replace') + return str(string, encoding='utf-8', errors='ignore') except: - return str(string, encoding='iso8859-1', errors='replace') - return str(string, encoding=charset, errors='replace') + return str(string, encoding='iso8859-1', errors='ignore') + return str(string, encoding=charset, errors='ignore') return string @@ -255,7 +258,7 @@ def decode_mail_headers(string): if six.PY2: return u' '.join([unicode(msg, charset or 'utf-8') for msg, charset in decoded]) elif six.PY3: - return u' '.join([str(msg, encoding=charset, errors='replace') if charset else str(msg) for msg, charset in decoded]) + return u' '.join([str(msg, encoding=charset, errors='ignore') if charset else str(msg) for msg, charset in decoded]) def ticket_from_message(message, queue, logger): @@ -302,9 +305,9 @@ def ticket_from_message(message, queue, logger): if part.get_content_maintype() == 'text' and name is None: if part.get_content_subtype() == 'plain': - body = EmailReplyParser.parse_reply( - decodeUnknown(part.get_content_charset(), part.get_payload(decode=True)) - ) + body = encoding.force_text(EmailReplyParser.parse_reply( + decodeUnknown(part.get_content_charset(), encoding.force_text(part.get_payload(decode=True))) + )) logger.debug("Discovered plain text MIME part") else: files.append( diff --git a/helpdesk/tests/test_get_email.py b/helpdesk/tests/test_get_email.py index 3a296514..87eddffd 100644 --- a/helpdesk/tests/test_get_email.py +++ b/helpdesk/tests/test_get_email.py @@ -1,3 +1,5 @@ +from __future__ import unicode_literals + from helpdesk.models import Queue, Ticket from django.test import TestCase from django.core.management import call_command @@ -40,7 +42,7 @@ class GetEmailCommonTests(TestCase): class GetEmailParametricTemplate(object): - """TestCase that checks email functionality accross methods and socks configs.""" + """TestCase that checks email functionality across methods and socks configs.""" def setUp(self): @@ -52,7 +54,8 @@ class GetEmailParametricTemplate(object): "allow_email_submission": True, "email_box_type": self.method, "logging_dir": self.temp_logdir, - "logging_type": 'none'} + "logging_type": 'none' + } if self.method == 'local': kwargs["email_box_local_dir"] = '/var/lib/mail/helpdesk/' @@ -74,10 +77,13 @@ class GetEmailParametricTemplate(object): def test_read_email(self): """Tests reading emails from a queue and creating tickets. For each email source supported, we mock the backend to provide - authenticly formatted responses containing our test data.""" + authentically formatted responses containing our test data.""" test_email = "To: update.public@example.com\nFrom: comment@example.com\nSubject: Some Comment\n\nThis is the helpdesk comment via email." test_mail_len = len(test_email) + test_unicode_email = "To: update.public@example.com\nFrom: comment@example.com\nSubject: Some Unicode Comment\n\nThis is the helpdesk comment via email with unicode chars \u2013 inserted for testing purposes." + test_unicode_email_len = len(test_unicode_email) + if self.socks: from socks import ProxyConnectionError with self.assertRaisesRegexp(ProxyConnectionError, '%s:%s' % (unrouted_socks_server, unused_port)): @@ -98,13 +104,25 @@ class GetEmailParametricTemplate(object): mocked_isfile.assert_any_call('/var/lib/mail/helpdesk/filename1') mocked_isfile.assert_any_call('/var/lib/mail/helpdesk/filename2') + with mock.patch('helpdesk.management.commands.get_email.listdir') as mocked_listdir, \ + mock.patch('helpdesk.management.commands.get_email.isfile') as mocked_isfile, \ + mock.patch('builtins.open' if six.PY3 else '__builtin__.open', mock.mock_open(read_data=test_unicode_email)): + mocked_isfile.return_value = True + mocked_listdir.return_value = ['filename3'] + + call_command('get_email') + + mocked_listdir.assert_called_with('/var/lib/mail/helpdesk/') + mocked_isfile.assert_any_call('/var/lib/mail/helpdesk/filename3') + elif self.method == 'pop3': # mock poplib.POP3's list and retr methods to provide responses as per RFC 1939 pop3_emails = { '1': ("+OK", test_email.split('\n')), '2': ("+OK", test_email.split('\n')), + '3': ("+OK", test_unicode_email.split('\n')), } - pop3_mail_list = ("+OK 2 messages", ("1 %d" % test_mail_len, "2 %d" % test_mail_len)) + pop3_mail_list = ("+OK 3 messages", ("1 %d" % test_mail_len, "2 %d" % test_mail_len, "3 %d" % test_unicode_email_len)) mocked_poplib_server = mock.Mock() mocked_poplib_server.list = mock.Mock(return_value=pop3_mail_list) mocked_poplib_server.retr = mock.Mock(side_effect=lambda x: pop3_emails[x]) @@ -117,8 +135,9 @@ class GetEmailParametricTemplate(object): imap_emails = { "1": ("OK", (("1", test_email),)), "2": ("OK", (("2", test_email),)), + "3": ("OK", (("3", test_unicode_email),)), } - imap_mail_list = ("OK", ("1 2",)) + imap_mail_list = ("OK", ("1 2 3",)) mocked_imaplib_server = mock.Mock() mocked_imaplib_server.search = mock.Mock(return_value=imap_mail_list) @@ -136,6 +155,10 @@ class GetEmailParametricTemplate(object): self.assertEqual(ticket2.ticket_for_url, "QQ-%s" % ticket2.id) self.assertEqual(ticket2.description, "This is the helpdesk comment via email.") + ticket3 = get_object_or_404(Ticket, pk=3) + self.assertEqual(ticket3.ticket_for_url, "QQ-%s" % ticket3.id) + self.assertEqual(ticket3.description, "This is the helpdesk comment via email with unicode chars \u2013 inserted for testing purposes.") + # build matrix of test cases case_methods = [c[0] for c in Queue._meta.get_field('email_box_type').choices] From 9dd558d71fb54b301b2e56ebb645ea9d7011be6f Mon Sep 17 00:00:00 2001 From: Garret Wassermann Date: Wed, 15 Feb 2017 18:51:55 -0500 Subject: [PATCH 04/25] Update gitignore to ignore KDE .directory files --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 7f84b87d..3a412d80 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,4 @@ docs/doctrees/* .coverage .project .pydevproject +.directory From d5e3432483f6d914320b79d8a9c1b056c93d486c Mon Sep 17 00:00:00 2001 From: Garret Wassermann Date: Wed, 15 Feb 2017 19:59:59 -0500 Subject: [PATCH 05/25] Apply word wrap to CONTRIBUTING to make it easier for viewing in text editor --- CONTRIBUTING | 71 +++++++++++++++++++++++++++++++++++++++------------- 1 file changed, 53 insertions(+), 18 deletions(-) diff --git a/CONTRIBUTING b/CONTRIBUTING index d5201216..063c6284 100644 --- a/CONTRIBUTING +++ b/CONTRIBUTING @@ -1,66 +1,101 @@ Contributing ============ -django-helpdesk is an open-source project and as such contributions from the community are welcomed and encouraged. +django-helpdesk is an open-source project and as such contributions from the +community are welcomed and encouraged. Licensing --------- -All contributions to django-helpdesk must be under the BSD license documented in the LICENSE file in the top-level directory of this project. By submitting a contribution to this project (in any way: via e-mail, via GitHub forks, attachments, etc), you acknowledge that your contribution is open-source and licensed under the BSD license. +All contributions to django-helpdesk must be under the BSD license documented in +the LICENSE file in the top-level directory of this project. By submitting a +contribution to this project (in any way: via e-mail, via GitHub forks, +attachments, etc), you acknowledge that your contribution is open-source and +licensed under the BSD license. -If you or your organisation does not accept these license terms then we cannot accept your contribution. Please reconsider! +If you or your organisation does not accept these license terms then we cannot +accept your contribution. Please reconsider! Translations ------------ -Although django-helpdesk has originally been written for the English language, there are already multiple translations to Spanish, Polish, and German and more translations are welcomed. +Although django-helpdesk has originally been written for the English language, +there are already multiple translations to Spanish, Polish, and German and more +translations are welcomed. -Translations are handled using the excellent Transifex service which is much easier for most users than manually editing .po files. It also allows collaborative translation. If you want to help translate django-helpdesk into languages other than English, we encourage you to make use of our Transifex project: +Translations are handled using the excellent Transifex service which is much +easier for most users than manually editing .po files. It also allows +collaborative translation. If you want to help translate django-helpdesk into +languages other than English, we encourage you to make use of our Transifex +project: http://www.transifex.net/projects/p/django-helpdesk/resource/core/ -Once you have translated content via Transifex, please raise an issue on the project Github page to let us know it's ready to import. +Once you have translated content via Transifex, please raise an issue on the +project Github page to let us know it's ready to import. Code changes ------------ -Please fork the project on GitHub, make your changes, and log a pull request to get the changes pulled back into the +Please fork the project on GitHub, make your changes, and log a pull request to +get the changes pulled back into the master branch of the django-helpdesk repository. -Wherever possible please break git commits up into small chunks that are specific to a single bit of functionality. For example, a commit should not contain both new functionality *and* a bugfix; the new function and the bugfix should be separate commits wherever possible. +Wherever possible please break git commits up into small chunks that are +specific to a single bit of functionality. For example, a commit should not +contain both new functionality *and* a bugfix; the new function and the bugfix +should be separate commits wherever possible. Commit messages should also explain *what*, precisely, has been changed. -All commits should include appropriate new or updated tests; see the Tests section below for more details. +All commits should include appropriate new or updated tests; see the Tests +section below for more details. -If you have any questions, please start a discussion on the GitHub issue tracker at +If you have any questions, please start a discussion on the GitHub issue tracker +at https://github.com/django-helpdesk/django-helpdesk/issues Tests ----- -Currently, test coverage is very low. We're working on increasing this, and to make life easier we are using `Travis CI` (http://travis-ci.org/) for continuous integration. This means that the test suite is run every time a code change is made, so we can try and make sure we avoid basic bugs and other regressions. +Currently, test coverage is very low. We're working on increasing this, and to +make life easier we are using `Travis CI` (http://travis-ci.org/) for continuous +integration. This means that the test suite is run every time a code change is +made, so we can try and make sure we avoid basic bugs and other regressions. -As a general policy, we will only accept new feature commits if they are accompanied by appropriate unit/functional tests (that is, tests for the functionality you just added). Bugfixes should also include new unit tests to ensure the bug has been fixed. +As a general policy, we will only accept new feature commits if they are +accompanied by appropriate unit/functional tests (that is, tests for the +functionality you just added). Bugfixes should also include new unit tests to +ensure the bug has been fixed. -More significant code refactoring must also include proper integration or validation tests, to be committed BEFORE the refactoring patches. This is to ensure that the refactored code produces the same results as the previous code base. +More significant code refactoring must also include proper integration or +validation tests, to be committed BEFORE the refactoring patches. This is to +ensure that the refactored code produces the same results as the previous code +base. -Any further integration or validation tests (tests for the entire django-helpdesk application) are not required but greatly appreciated until we can improve our overall test coverage. +Any further integration or validation tests (tests for the entire +django-helpdesk application) are not required but greatly appreciated until we +can improve our overall test coverage. Please include tests in the ``tests/`` folder when committing code changes. -If you have any questions about creating or maintaining proper tests, please start a discussion on the GitHub issue tracker at +If you have any questions about creating or maintaining proper tests, please +start a discussion on the GitHub issue tracker at https://github.com/django-helpdesk/django-helpdesk/issues Database schema changes ----------------------- -As well as making your normal code changes to ``models.py``, please generate a Django migration file and commit it with your code. You will want to use a command similar to the following:: +As well as making your normal code changes to ``models.py``, please generate a +Django migration file and commit it with your code. You will want to use a +command similar to the following:: ./manage.py migrate helpdesk --auto [migration_name] -Make sure that ``migration_name`` is a sensible single-string explanation of what this migration does, such as *add_priority_options* or *add_basket_table*. +Make sure that ``migration_name`` is a sensible single-string explanation of +what this migration does, such as *add_priority_options* or *add_basket_table*. -This will add a file to the ``migrations/`` folder, which must be committed to git with your other code changes. +This will add a file to the ``migrations/`` folder, which must be committed to +git with your other code changes. From ec38150d1fd3a543c9670c15cbb1e92e897631f7 Mon Sep 17 00:00:00 2001 From: Garret Wassermann Date: Wed, 15 Feb 2017 20:00:40 -0500 Subject: [PATCH 06/25] Ignore KDE .desktop files in .gitignore --- .gitignore | 1 + 1 file changed, 1 insertion(+) diff --git a/.gitignore b/.gitignore index 7f84b87d..90452818 100644 --- a/.gitignore +++ b/.gitignore @@ -6,3 +6,4 @@ docs/doctrees/* .coverage .project .pydevproject +.desktop From 492e25c77bb16ada3f2fa5ee0d6f9f8775c1a417 Mon Sep 17 00:00:00 2001 From: Garret Wassermann Date: Wed, 15 Feb 2017 20:03:08 -0500 Subject: [PATCH 07/25] Update docs to say Python 3.4+ since that's what we current do testing against --- docs/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/index.rst b/docs/index.rst index e2fab98a..ae9e3ead 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -69,7 +69,7 @@ django-helpdesk is released under the BSD license, however it packages 3rd party Dependencies ------------ -1. Python 2.7+ (or 3.3+) +1. Python 2.7+ (or 3.4+) 2. Django (1.8 or newer) 3. An existing **working** Django project with database etc. If you cannot log into the Admin, you won't get this product working! This means you **must** run `syncdb` **before** you add ``helpdesk`` to your ``INSTALLED_APPS``. From 7bd7b42788c07eafe5a539e955b8a2c7489b018a Mon Sep 17 00:00:00 2001 From: Garret Wassermann Date: Wed, 15 Feb 2017 20:04:32 -0500 Subject: [PATCH 08/25] Ignore KDE .directory files in .gitignore; totally used the wrong name (.desktop) last time, oops --- .gitignore | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/.gitignore b/.gitignore index 90452818..3a412d80 100644 --- a/.gitignore +++ b/.gitignore @@ -6,4 +6,4 @@ docs/doctrees/* .coverage .project .pydevproject -.desktop +.directory From 18557aad3871e595744ec86aea29987d50e8ceb3 Mon Sep 17 00:00:00 2001 From: Garret Wassermann Date: Sat, 25 Feb 2017 04:07:20 -0500 Subject: [PATCH 09/25] Add Python 3.6 to Travis CI configuration --- .travis.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/.travis.yml b/.travis.yml index 62879e4d..37e94b6b 100644 --- a/.travis.yml +++ b/.travis.yml @@ -4,6 +4,7 @@ python: - "2.7" - 3.4.4 - "3.5" + - "3.6" env: - DJANGO=1.8.17 @@ -18,7 +19,7 @@ install: before_script: - "pep8 --exclude=migrations --ignore=E501 helpdesk" -script: +script: - coverage run --source='.' quicktest.py helpdesk after_success: From dfdb12e354182acd8695cf769bc9d38e23b6e37a Mon Sep 17 00:00:00 2001 From: "Yuego [Home]" Date: Wed, 8 Mar 2017 23:01:31 +0400 Subject: [PATCH 10/25] Fix link translation --- helpdesk/templates/helpdesk/kb_category.html | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/helpdesk/templates/helpdesk/kb_category.html b/helpdesk/templates/helpdesk/kb_category.html index f508aecd..e3859c90 100644 --- a/helpdesk/templates/helpdesk/kb_category.html +++ b/helpdesk/templates/helpdesk/kb_category.html @@ -22,7 +22,9 @@

{{ item.question }}