Issue #6942 - Add XTM integration in CMS

cms/bin/xtm_translations/xtm_api.py

+# This file is part of the Adblock Plus web scripts,
+# Copyright (C) 2006-present eyeo GmbH
+#
+# Adblock Plus is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License version 3 as
+# published by the Free Software Foundation.
+#
+# Adblock Plus is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with Adblock Plus.  If not, see <http://www.gnu.org/licenses/>.
+
+from __future__ import unicode_literals
+
+import requests

[Vasily Kuznetsov, 2018/09/26 15:45:24]

Imports order should be stdlib, 3rd party, local with an empty line between
them. Requests is 3rd party.

[Tudor Avram, 2018/10/04 06:48:07]

Done.

+import json
+
+__all__ = [
+    'XTMCloudAPI', 'XTMCloudException', 'get_token',
+]
+
+_BASE_URL = 'https://wstest2.xtm-intl.com/rest-api/'

[Vasily Kuznetsov, 2018/09/26 15:45:24]

Shouldn't this URL be configurable somehow?

[Tudor Avram, 2018/10/04 06:48:09]

Going to discuss the level of configurability with Winsley later this week. 

[Vasily Kuznetsov, 2018/10/05 10:56:25]

Acknowledged.

+
+
+class XTMCloudException(Exception):
+    _BASE_MESSAGE = ('Error: XTM Cloud API failed while {0}, with error '
+                     'code {1}: {2}')
+
+    def __init__(self, code, msg, action):
+        """Constructor.
+
+        Parameters
+        ----------
+        code: int
+            The error code returned by the API.
+        msg: str
+            The error message returned by the API.
+        action: str
+            The action that caused the exception.
+
+        """
+        full_message = self._BASE_MESSAGE.format(action, code, msg)
+        super(XTMCloudException, self).__init__(full_message)

[Vasily Kuznetsov, 2018/09/26 15:45:24]

Do you think somebody might be interested in having the code, message and action
as separate attributes on this exception for inspection? Maybe this is
premature, but in principle we could assign them to attributes...

[Tudor Avram, 2018/10/04 06:48:08]

Done.

+
+
+class XTMCloudAPI(object):
+
+    _AUTHORIZATION_TMP = 'XTM-Basic {0}'
+    _URL_PATHS = {
+        'create': 'projects',

[Vasily Kuznetsov, 2018/09/26 15:45:24]

So I think I understand my biggest problem with having these dicts of constants
like this. Usually we use constants to avoid magical numbers or strings. So that
instead of foo.bar("create") we have foo.bar(CREATE) and then people reading the
code are not wondering what this string means.

But when we use dicts for storing constants we kind of end up with other magical
strings. For example, instead of:

    url = self.base_url + 'projects'

we now have:

    url = self.base_url + self._URL_PATHS['create']

But then it seems we have a magical string here again. Do you think it would
make sense to convert these dicts to "namespaces" too?

[Tudor Avram, 2018/10/04 06:48:08]

Done.

+        'download_multiple': 'projects/{0}/files/download',
+        'downlload_single': 'projects/{0}/files/{1}/download',
+        'upload': 'projects/{0}/files/upload',
+        'get_target_lang': 'projects/{0}/metrics',
+        'add_target_lang': 'projects/{0}/target-languages',
+    }
+    _DEFAULT_CONTENT_TYPE = 'multipart/form-data'

[Vasily Kuznetsov, 2018/09/26 15:45:24]

Are we using this constant anywhere?

[Tudor Avram, 2018/10/04 06:48:10]

Nope. Was using that initially, forgot to remove id :D 

Done.

+    _MATCH_TYPE = {
+        'upload': 'NO_MATCH',
+        'update': 'MATCH_NAMES',
+    }
+    _URL_PARAMS = {
+        'download': '?fileType={1}',
+    }
+    _SUCCESS_CODES = {
+        'create': 201,
+        'upload': 200,
+        'download': 200,
+        'get_target_lang': 200,
+        'add_target_lang': 200,
+    }
+
+    def __init__(self, token):
+        """Constructor.
+
+        Parameters
+        ----------
+        token: str
+            Token used to authenticate with the API.
+
+        """
+        self._token = token
+        self.base_url = _BASE_URL
+
+    def _execute(self, url, data=None, files=None, stream=False,
+                 params=None, headers=None):
+        """Private method executing requests and returning the result.

[Vasily Kuznetsov, 2018/09/26 15:45:24]

Nit: I think we don't really need to include the information that is already
obvious from the method name. Usually the best docstring for a method is
something like """<verb> [<noun phrase>] ...""" and here it could be just
"""Send request to the API and return the response."""

[Tudor Avram, 2018/10/04 06:48:09]

Done.

+
+        Parameters
+        ----------
+        url: str
+            The url we're making the request to.
+        data: dict
+            The data to be sent to the API. If provided, the request will be
+            a POST request, otherwise GET. Default None.
+        files: dict
+            The files to be uploaded(if any). Default None.
+        params: dict
+            The URL parameters to be specified.
+        stream: bool
+            Whether using the stream option when executing the request or not.
+        headers: dict
+            Default headers to be sent with the request.
+
+        Returns
+        -------
+        The contents of the API response.
+
+        """
+        auth_header = {
+            'Authorization': self._AUTHORIZATION_TMP.format(self._token),
+        }
+
+        if headers:
+            headers.update(auth_header)
+        else:
+            headers = auth_header
+
+        if data:
+            response = requests.post(url, data=data, files=files,
+                                     headers=headers, params=params,
+                                     stream=stream)
+        else:
+            response = requests.get(url, headers=headers, stream=True,
+                                    params=params)
+
+        return response
+
+    def _construct_files_dict(self, files, param_tmp):
+        """Private method converting a list of files to a uploadable format.
+
+        Parameters
+        ----------
+        files: iterable
+            Of the files to be uploaded. See create_project() for expected
+            format.
+
+        Returns
+        -------
+        dict
+            With the names of the files to be uploaded.
+        dict
+            With the files in the correct format for upload.
+
+        """
+        files_to_upload = {}
+        file_names = {}
+
+        for idx in range(len(files)):
+            if files[idx][0]:
+                file_names[(param_tmp + '.name').format(idx)] = files[idx][0]
+            files_to_upload[(param_tmp + '.file').format(idx)] = files[idx][1]
+
+        return file_names, files_to_upload
+
+    def create_project(self, name, description, reference_id, target_languages,
+                       customer_id, workflow_id, source_language='en_US',
+                       files=None):
+        """Create a new project with XTM Cloud.
+
+        Parameters
+        ----------
+        name: str
+            The name of the project we want to create.
+        description: str
+            The description of the project.
+        reference_id: str
+            The referenceID of the project.
+        target_languages: iterable
+            The target language(s) for the project.
+        customer_id: int
+            The id of the customer creating the project
+        workflow_id: int
+            The id of the main workflow
+        source_language: str
+            The source language for the project.
+        files: iterable
+            With the files to be uploaded on creation Expects a list of
+            tuples of the form:
+            (<file_name>, <file_content>).
+
+            If <file_name> is None, it will be ignored.
+
+        Returns
+        -------
+        int
+            The id of the created project.
+        iterable
+            Containing the information for all the jobs that were initiated.
+            See docs for upload_files() for the fully documented format.
+
+        Raises
+        ------
+        XTMCloudException
+            If the creation of the project was not successful.
+
+        """
+        data = {
+            'name': name,
+            'description': description,
+            'referenceId': reference_id,
+            'customerId': customer_id,
+            'workflowId': workflow_id,
+            'sourceLanguage': source_language,
+            'targetLanguages': target_languages,
+        }
+
+        if files:
+            file_names, files_to_upload = self._construct_files_dict(
+                files, 'translationFiles[{}]')
+            data.update(file_names)
+        else:
+            # Hacky way to go around 415 error code
+            files_to_upload = {'a': 'b'}
+
+        url = self.base_url + self._URL_PATHS['create']
+
+        response = self._execute(url, data=data, files=files_to_upload)
+
+        if response.status_code != self._SUCCESS_CODES['create']:
+            # The creation was not successful
+            raise XTMCloudException(response.status_code,
+                                    response.text.decode('utf-8'),
+                                    'creating job')
+
+        data = json.loads(response.text.encode('utf-8'))
+
+        return data['projectId'], data['jobs']
+
+    def upload_files(self, files, project_id, overwrite=True):
+        """Upload a set of files to a project.
+
+        Parameters
+        ----------
+        files: iterable
+            Of the files to be uploaded.
+            Expects a list of tuples with the following format:
+                (<file_name>, <file_content>)
+            If <file_name> is None, it will be ignored.
+
+        project_id: int
+            The id of the project we're uploading to.
+        overwrite: bool
+            Whether the files to be uploaded are going to overwrite the files
+            that are already in the XTM project or not. Default False.
+
+        Returns
+        -------
+        iterable
+            With the upload results. Each element will have the form:
+
+                {
+                    'fileName': <the name of the uploaded file>,
+                    'jobId': <ID of the job associated with the file>,
+                    'sourceLanguage': <source language of the uploaded file>,
+                    'targetLanguage': <target language of the uploaded file>,
+                }
+
+        Raises
+        ------
+        XTMCloudException
+            If the API operation fails.
+
+        """
+        file_names, files_to_upload = self._construct_files_dict(
+            files, 'files[{}]',
+        )
+
+        if len(files_to_upload.keys()) == 0:
+            raise Exception('Error: No files provided for upload.')
+
+        data = {
+            'matchType': self._MATCH_TYPE['update'] if overwrite else

[Vasily Kuznetsov, 2018/09/26 15:45:24]

Maybe it would make more sense to have True and False as the keys of this dict
so that you could just index it with `overwrite`.

[Tudor Avram, 2018/10/04 06:48:09]

Done.

+            self._MATCH_TYPE['upload'],
+        }
+        data.update(file_names)
+
+        url = self.base_url + self._URL_PATHS['upload'].format(project_id)
+
+        response = self._execute(url, data=data, files=files_to_upload)
+
+        if response.status_code != self._SUCCESS_CODES['upload']:
+            raise XTMCloudException(response.status_code,
+                                    response.text.encode('utf-8'),
+                                    'uploading files')
+
+        return json.loads(response.text.encode('utf-8'))['jobs']
+
+    def download_files(self, project_id, files_type='TARGET'):
+        """Download files for a specific project.
+
+        Parameters
+        ----------
+        project_id: int
+            The id of the project to download from.
+        files_type: str
+            The type of the files we want to download. Default TARGET
+
+        Returns
+        -------
+        bytes
+            The contents of the zip file returned by the API
+
+        Raises
+        ------
+        XTMCloudException
+            If the request is flawed in any way.
+
+        """
+        url = (self.base_url + self._URL_PATHS['download_multiple']).format(
+            project_id,
+        )
+
+        exception_msg = {
+            400: 'Invalid request',
+            401: 'Authentication failed',
+            500: 'Internal server error',
+            404: 'Project not found.',
+        }
+
+        response = self._execute(url, params={'fileType': files_type},
+                                 stream=True)
+
+        if response.status_code != self._SUCCESS_CODES['download']:
+            raise XTMCloudException(
+                response.status_code,
+                exception_msg[response.status_code],
+                'downloading files from {}'.format(project_id),
+            )
+
+        return response.content
+
+    def get_target_languages(self, project_id):
+        """Get all the target languages for a specific project.
+
+        Parameters
+        ----------
+        project_id: int
+            The id if the project in question.
+
+        Returns
+        -------
+        iterable
+            With the target languages for this project.
+
+        Raises
+        ------
+        XTMCloudException
+            If the request is unsuccessful.
+
+        """
+        url = (self.base_url + self._URL_PATHS['get_target_lang']).format(
+            project_id,
+        )
+
+        response = self._execute(url, stream=True)
+
+        if response.status_code != self._SUCCESS_CODES['get_target_lang']:
+            raise XTMCloudException(response.status_code, response.content,
+                                    'extracting target languages')
+
+        data = json.loads(response.content.encode('utf-8'))
+        return {item['targetLanguage'] for item in data}
+
+    def add_target_languages(self, project_id, target_languages):
+        """Add target languages to a project.
+
+        Parameters
+        ----------
+        project_id: int
+            The id of the project in question.
+        target_languages: iterable
+            The languages to be added.
+
+        Raises
+        -------
+        XTMCloudException
+            If the request to the API was not successful.
+
+        """
+        data = json.dumps({
+            'targetLanguages': target_languages,
+        })
+        url = (self.base_url + self._URL_PATHS['add_target_lang']).format(
+            project_id,
+        )
+        headers = {'content-type': 'application/json'}
+
+        response = self._execute(url, data=data, headers=headers)
+
+        if response.status_code != self._SUCCESS_CODES['add_target_lang']:
+            raise XTMCloudException(response.status_code, response.content,
+                                    'adding target languages to project')
+
+
+def get_token(username, password, user_id):

[Vasily Kuznetsov, 2018/09/26 15:45:25]

Maybe this should also be a method of XTMCloudAPI class?

[Tudor Avram, 2018/10/04 06:48:08]

We have already discussed this in person. As I mentioned then, I feel like
`get_token()` doesn't really fit into all the things from `XTMCloudAPI`, as it
generates the token, rather than using it. And all the methods of that class
make use of the token to make requests.

[Vasily Kuznetsov, 2018/10/05 10:56:25]

Allright, makes sense.

+    """Generate an API token from username and password.
+
+    Parameters
+    ----------
+    username: str
+        The username used to generate the token.
+    password: str
+        The password used to generate the token.
+    user_id: int
+        The user ID used to generate the token.
+
+    Returns
+    -------
+    str
+        The resulting token generated by the API.
+
+    Raises
+    ------
+    XTMCloudException
+        If the credentials provided were invalid.
+
+    """
+    request_body = json.dumps({
+        'client': username,
+        'password': password,
+        'userId': user_id,
+    })
+
+    url = _BASE_URL + 'auth/token'
+
+    headers = {'content-type': 'application/json'}
+
+    response = requests.post(url, data=request_body, headers=headers)
+
+    if response.status_code == 200:
+        return json.loads(response.text)['token'].encode()
+
+    raise XTMCloudException(response.status_code,
+                            response.text.encode('utf-8'),
+                            'generating token')