crowdin: update for Q

1. Update the ReadMe to reflect changes for Q.

2. Squash updates from LineageOS:
We need to update our script to make it work properly again
and add recent changes from LineageOS that were authored
by "Michael W <baddaemon87@gmail.com>" (and a bit adapted).

https://review.lineageos.org/c/LineageOS/cm_crowdin/+/268087
crowdin: Make downloading a parallel thing

* Instead of relying on the API, call the download for several languages
  at once
* Limit the amount of simultaneous calls to a maximum of 20 as per
  https://support.crowdin.com/api/api-integration-setup/#rate-limits

https://review.lineageos.org/c/LineageOS/cm_crowdin/+/267360/1
crowdin: Improve console output

* Reduce the amount of output lines - this makes it more readable
* Add a signal handler to handle aborts - we don't need a stacktrace here

https://review.lineageos.org/c/LineageOS/cm_crowdin/+/253482
crowdin: Don't clean non-xml and handle errors for xml-files

* For non-xml files, just don't touch the file
* For xml-files, create a backup:
  - create the same folder structure from /res(.*)/(.*) and place the
    wrong file in there, appending a number if already existing
  - then checkout the original xml file to get rid of the errors

https://review.lineageos.org/c/LineageOS/cm_crowdin/+/250431
crowdin: Remove invalid strings

* aapt2 fails when linking resources which contain strings with a product
  attribute but don't have the same string with product=default set
* Search for all strings in the relevant files which have a product
  attribute and look for the corresponding default one
* Remove all strings with that name if it doesn't exist

https://review.lineageos.org/c/LineageOS/cm_crowdin/+/250488
crowdin: Improve empty resource file removal

* Instead of maintaining a list of all variants of empty resource tags,
  just count the number of sub elements
* Equal to zero -> No child elements -> remove

https://review.lineageos.org/c/LineageOS/cm_crowdin/+/244152
crowdin: Fix for subprojects

* If a project is stored within another projects folder structure,
  the current implementation will treat it as "already known" and therefore
  never process the project itself
* Search for the biggest path match before continuing

https://review.lineageos.org/c/LineageOS/cm_crowdin/+/244128
crowdin: Fix for devicesettings

* Due to a split at /res, the devicesettings-repo was always treated wrong
* Check the length of the list after split and treat that case properly

https://review.lineageos.org/c/LineageOS/cm_crowdin/+/243912
crowdin: Add missing import

* Forgot "re", will result in an error when trying to download from crowdin

https://review.lineageos.org/c/LineageOS/cm_crowdin/+/242731
crowdin: Re-add removing empty xml files

* Process every file and remove all comments
* If only the root tags are left, remove the whole file

https://review.lineageos.org/c/LineageOS/cm_crowdin/+/242685
crowdin: Only push the translation changes

* Instead of commiting all the changed files, parse the config files
  and only commit changes made to files which are mentioned there
* This ensures that changes to other files which exist locally aren't
  added to the commits and have to be manually removed again

Change-Id: I51697a3421308eeff9110243869383d336e960ae

2 files changed, 202 insertions, 71 deletions

diff --git a/README.mkdn b/README.mkdn
index 607f3d1..05fb05e 100644..100755
--- a/README.mkdn
+++ b/README.mkdn
@@ -17,16 +17,18 @@ Prerequisites
 
 4. The package "python-git" is used for Git integration and must be installed (also see: https://gitpython.readthedocs.io/en/stable/).
 
-5. The prebuilt java version of crowdin-cli >= 2.0.x (see: https://support.crowdin.com/cli-tool/) is required for this to work.
+5. The installs of the package "gitdb" with its dependency via <code>pip install gitdb</code> for python 2.7.x.
+
+6. The prebuilt java version of crowdin-cli >= 2.0.x (see: https://support.crowdin.com/cli-tool/) is required for this to work.
    It can be download from here: https://downloads.crowdin.com/cli/v2/crowdin-cli.zip
 
    Follow the installation instructions on the web page and set up the cli-tool correctly.
 
    *Note: The current limitation is that the JAR-file must be accessible via the path "/usr/local/bin/". You can of course change that in the python script file.*
 
-6. Currently the crowdin-cli tool requires either Linux or macOS and an appropiate Java version >= 1.8.xx to work.
+7. Currently the crowdin-cli tool requires either Linux or macOS and an appropiate Java version >= 1.8.xx to work.
 
-7. <code>/config/AICP-version_extra_packages.xml</code> must be copied to <code>.repo/local_manifests</code> of each "AICP-version" tree.
+8. <code>/config/AICP-version_extra_packages.xml</code> must be copied to <code>.repo/local_manifests</code> of each "AICP-version" tree.
    This makes sure you will sync all the extra packages not included in the main manifest.
    Please remember that you should comment out the packages you already have in your local aicp_manifest.xml file
    as these are device dependent.
@@ -43,20 +45,20 @@ Needed directory structure:
 * /your_username
  * /aicp
    * /config/.yaml  #all YAML config files go in here!
-   * /p9.0
+   * /q10.0
      * /.repo
-       * /local_manifests/p9.0_extra_packages.xml
+       * /local_manifests/q10.0_extra_packages.xml
    * crowdin_sync.py
 
 Enviroment variables to export:
 
     export AICP_CROWDIN_API_KEY=aicp_api_key
-    export AICP_CROWDIN_BASE_PATH_p9_0=your_base_path_and_branch
+    export AICP_CROWDIN_BASE_PATH_q10_0=your_base_path_and_branch
 
 Example:
 
     export AICP_CROWDIN_API_KEY=54e01e81--your-api-key--f6a2724a  #Can be found in your project settings page!
-    export AICP_CROWDIN_BASE_PATH_p9_0=/home/your_username/aicp/p9.0  #This is dependent on the real path to your source tree
+    export AICP_CROWDIN_BASE_PATH_q10_0=/home/your_username/aicp/q10.0  #This is dependent on the real path to your source tree
 
 Execute:
 The python script "crowdin_sync.py" and the "config" directory should be copied into the base folder structure, e.g. /home/your_username/aicp, like shown above.
@@ -86,27 +88,27 @@ optional arguments:<br />
 
 Examples:
 
-<code>./crowdin_sync.py --username GerritName --branch p9.0 --upload-sources</code>
+<code>./crowdin_sync.py --username GerritName --branch q10.0 --upload-sources</code>
 
 Will upload specified local files from the YAML-config to Crowdin. Translations already there will be preserved.
 
-<code>./crowdin_sync.py --username GerritName --branch p9.0 --upload-translations</code>
+<code>./crowdin_sync.py --username GerritName --branch q10.0 --upload-translations</code>
 
 Will upload local translations to Crowdin, based on YAML-config and from your local sources.
 
-<code>./crowdin_sync.py --username GerritName --branch p9.0 --download</code>
+<code>./crowdin_sync.py --username GerritName --branch q10.0 --download</code>
 
-Will download translations from Crowdin of the specified branch (p9.0), based on YAML-config, to your local sources,
+Will download translations from Crowdin of the specified branch (q10.0), based on YAML-config, to your local sources,
 delete empty translations and upload updated or new translations to AICP Gerrit for review.
 
-<code>./crowdin_sync.py -username GerritName --branch p9.0 --local-download</code>
+<code>./crowdin_sync.py --branch q10.0 --local-download</code>
 
-Will download translations from Crowdin of the specified branch (p9.0), based on YAML-config, to your local sources
+Will download translations from Crowdin of the specified branch (q10.0), based on YAML-config, to your local sources
 and delete empty translations. This is useful to perform local builds and test the imported translations.
 
-<code>./crowdin_sync.py -username GerritName --branch p9.0 --submit</code>
+<code>./crowdin_sync.py --username GerritName --branch q10.0 --submit</code>
 
-Will search for open translations commits on AICP's Gerrit on the specified branch (p9.0) and
+Will search for open translations commits on AICP's Gerrit on the specified branch (q10.0) and
 automatically review, verify and submit them into the repositories. This is useful after successful builds and requires
 Gerrit Admin rights to preform this action.
 
diff --git a/crowdin_sync.py b/crowdin_sync.py
index 8f4bfbe..787c64b 100755
--- a/crowdin_sync.py
+++ b/crowdin_sync.py
@@ -1,12 +1,12 @@
 #!/usr/bin/env python
 # -*- coding: utf-8 -*-
-# crowdin-cli_sync.py
+# crowdin_sync.py
 #
 # Updates Crowdin source translations and pushes translations
 # directly to AICP Gerrit.
 #
 # Copyright (C) 2014-2016 The CyanogenMod Project
-# Copyright (C) 2017-2019 The LineageOS Project
+# Copyright (C) 2017-2020 The LineageOS Project
 # This code has been modified.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -30,11 +30,14 @@ import json
 import git
 import os
 import re
+import shutil
 import subprocess
 import sys
 import yaml
 
+from itertools import islice
 from lxml import etree
+from signal import signal, SIGINT
 from xml.dom import minidom
 
 # ################################# GLOBALS ################################## #
@@ -83,7 +86,7 @@ def add_target_paths(config_files, repo, base_path, project_path):
 
     # Strip all comments
     for f in file_paths:
-        clean_file(base_path, project_path, f)
+        clean_xml_file(base_path, project_path, f)
 
     # Modified and untracked files
     modified = repo.git.ls_files(m=True, o=True)
@@ -127,7 +130,7 @@ def get_target_path(pattern, source, lang, project_path):
     return target_path
 
 
-def clean_file(base_path, project_path, filename):
+def clean_xml_file(base_path, project_path, filename):
     path = base_path + '/' + project_path + '/' + filename
 
     # We don't want to create every file, just work with those already existing
@@ -141,7 +144,38 @@ def clean_file(base_path, project_path, filename):
         return
 
     XML = fh.read()
-    tree = etree.fromstring(XML)
+    try:
+        tree = etree.fromstring(XML)
+    except etree.XMLSyntaxError as err:
+        print('%s: XML Error: %s' % (filename, err.error_log))
+        filename, ext = os.path.splitext(path)
+        if ext == '.xml':
+            reset_file(path, repo)
+        return
+
+    # Remove strings with 'product=*' attribute but no 'product=default'
+    # This will ensure aapt2 will not throw an error when building these
+    productStrings = tree.xpath("//string[@product]")
+    for ps in productStrings:
+        stringName = ps.get('name')
+        stringsWithSameName = tree.xpath("//string[@name='{0}']".format(stringName))
+
+        # We want to find strings with product='default' or no product attribute at all
+        hasProductDefault = False
+        for string in stringsWithSameName:
+            product = string.get('product')
+            if product is None or product == 'default':
+                hasProductDefault = True
+                break
+
+        # Every occurance of the string has to be removed when no string with the same name and
+        # 'product=default' (or no product attribute) was found
+        if not hasProductDefault:
+            print("{0}: Found string '{1}' with missing 'product=default' attribute"
+                  .format(path, stringName))
+            for string in stringsWithSameName:
+                tree.remove(string)
+                productStrings.remove(string)
 
     header = ''
     comments = tree.xpath('//comment()')
@@ -175,30 +209,42 @@ def clean_file(base_path, project_path, filename):
     fh.close()
 
     # Remove files which don't have any translated strings
-    empty_contents = {
-        '<resources/>',
-        '<resources xmlns:xliff="urn:oasis:names:tc:xliff:document:1.2"/>',
-        ('<resources xmlns:android='
-         '"http://schemas.android.com/apk/res/android"/>'),
-        ('<resources xmlns:android="http://schemas.android.com/apk/res/android"'
-         ' xmlns:xliff="urn:oasis:names:tc:xliff:document:1.2"/>'),
-        ('<resources xmlns:tools="http://schemas.android.com/tools"'
-         ' xmlns:xliff="urn:oasis:names:tc:xliff:document:1.2"/>'),
-        ('<resources xmlns:android="http://schemas.android.com/apk/res/android">\n</resources>'),
-        ('<resources xmlns:android="http://schemas.android.com/apk/res/android"'
-         ' xmlns:xliff="urn:oasis:names:tc:xliff:document:1.2">\n</resources>'),
-        ('<resources xmlns:tools="http://schemas.android.com/tools"'
-         ' xmlns:xliff="urn:oasis:names:tc:xliff:document:1.2">\n</resources>'),
-        ('<resources>\n</resources>')
-    }
-    for line in empty_contents:
-        if line in content:
-            print('Removing ' + path)
-            os.remove(path)
-            break
+    contentList = list(tree)
+    if len(contentList) == 0:
+        print('Removing ' + path)
+        os.remove(path)
+
+
+# For files we can't process due to errors, create a backup
+# and checkout the file to get it back to the previous state
+def reset_file(filepath, repo):
+    backupFile = None
+    parts = filepath.split("/")
+    found = False
+    for s in parts:
+        curPart = s
+        if not found and s.startswith("res"):
+            curPart = s + "_backup"
+            found = True
+        if backupFile is None:
+            backupFile = curPart
+        else:
+            backupFile = backupFile + '/' + curPart
+
+    path, filename = os.path.split(backupFile)
+    if not os.path.exists(path):
+        os.makedirs(path)
+    if os.path.exists(backupFile):
+        i = 1
+        while os.path.exists(backupFile + str(i)):
+            i+=1
+        backupFile = backupFile + str(i)
+    shutil.copy(filepath, backupFile)
+    repo.git.checkout(filepath)
+
 
 def push_as_commit(config_files, base_path, path, name, branch, username):
-    print('Committing %s on branch %s' % (name, branch))
+    print('Committing %s on branch %s: ' % (name, branch), end='')
 
     # Get path
     project_path = path
@@ -220,17 +266,16 @@ def push_as_commit(config_files, base_path, path, name, branch, username):
     try:
         repo.git.commit(m='Automatic AICP translation import')
     except:
-        print('Failed to create commit for %s, probably empty: skipping'
-              % name, file=sys.stderr)
+        print('Failed, probably empty: skipping', file=sys.stderr)
         return
 
     # Push commit
     try:
         repo.git.push('ssh://%s@gerrit.aicp-rom.com:29418/%s' % (username, name),
                       'HEAD:refs/for/%s%%topic=Translations-%s' % (branch, branch))
-        print('Successfully pushed commit for %s' % name)
+        print('Success')
     except:
-        print('Failed to push commit for %s' % name, file=sys.stderr)
+        print('Failed', file=sys.stderr)
 
     _COMMITS_CREATED = True
 
@@ -266,11 +311,12 @@ def submit_gerrit(branch, username):
         '--code-review +2',
         '--submit', js['currentPatchSet']['revision']]
         msg, code = run_subprocess(cmd, True)
+        print('Submitting commit %s: ' % js['url'], end='')
         if code != 0:
             errorText = msg[1].replace('\n\n', '; ').replace('\n', '')
-            print('Submitting commit {0} failed: {1}'.format(js['url'], errorText))
+            print('Failed: %s' % errorText)
         else:
-            print('Success when submitting commit {0}'.format(js['url']))
+            print('Success')
 
         commits += 1
 
@@ -293,16 +339,55 @@ def find_xml(base_path):
             if os.path.splitext(f)[1] == '.xml':
                 yield os.path.join(dp, f)
 
+
+def get_languages(configPath):
+    try:
+        fh = open(configPath, 'r+')
+        data = yaml.safe_load(fh);
+        fh.close();
+
+        languages = []
+        for elem in data['files'][0]['languages_mapping']['android_code']:
+            languages.append(elem);
+        return languages
+    except:
+        return []
+
+
+def available_cpu_count():
+    try:
+        import multiprocessing
+        return multiprocessing.cpu_count()
+    except (ImportError, NotImplementedError):
+        return 4; # Some probably safe default
+
+
+def run_parallel(commands):
+    processes = (subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=sys.stderr)
+        for cmd in commands)
+    slice_size = min(20, available_cpu_count())
+    running_processes = list(islice(processes, slice_size)) # split into slices and start processes
+    while running_processes:
+        for i, process in enumerate(running_processes):
+            # see if the process has finished
+            if process.poll() is not None:
+                out, err = process.communicate()
+                print(out);
+                # get and start next process
+                running_processes[i] = next(processes, None)
+                if running_processes[i] is None:
+                    del running_processes[i]
+                    break
+
 # ############################################################################ #
 
 
 def parse_args():
     parser = argparse.ArgumentParser(
         description="Synchronising AICP translations with Crowdin")
-    parser.add_argument('--username', help='Gerrit username',
-                        required=True)
-    parser.add_argument('--branch', help='AICP branch',
-                        required=True)
+    sync = parser.add_mutually_exclusive_group()
+    parser.add_argument('--username', help='Gerrit username')
+    parser.add_argument('--branch', help='AICP branch', required=True)
     parser.add_argument('-c', '--config', help='Custom yaml config')
     parser.add_argument('--upload-sources', action='store_true',
                         help='Upload sources to AICP Crowdin')
@@ -311,7 +396,7 @@ def parse_args():
     parser.add_argument('--download', action='store_true',
                         help='Download AICP translations from Crowdin')
     parser.add_argument('--local-download', action='store_true',
-                        help='local Download AICP translations from Crowdin')
+                        help='Locally download AICP translations from Crowdin')
     parser.add_argument('--submit', action='store_true',
                         help='Auto-Merge open AICP translations on Gerrit')
     return parser.parse_args()
@@ -379,6 +464,7 @@ def upload_translations_crowdin(branch, config):
                    '--no-import-duplicates', '--import-eq-suggestions',
                    '--auto-approve-imported'])
 
+
 def local_download(base_path, branch, xml, config):
     if config:
         print('\nDownloading translations from Crowdin (custom config)')
@@ -388,9 +474,17 @@ def local_download(base_path, branch, xml, config):
     else:
         print('\nDownloading translations from Crowdin '
               '(AOSP supported languages)')
-        check_run(['java', '-jar', '/usr/local/bin/crowdin-cli.jar',
-                   '--config=%s/config/%s.yaml' % (_DIR, branch),
-                   'download', '--branch=%s' % branch])
+        commands = []
+        config_path = '%s/config/%s.yaml' % (_DIR, branch);
+        languages = get_languages(config_path)
+        for lang in languages:
+            cmd = ['java', '-jar', '/usr/local/bin/crowdin-cli.jar',
+                   '--config=%s' % config_path,
+                   'download', '--branch=%s' % branch,
+                   '-l=%s' % lang]
+            commands.append(cmd)
+        run_parallel(commands)
+
 
 def download_crowdin(base_path, branch, xml, username, config):
     local_download(base_path, branch, xml, config)
@@ -424,7 +518,19 @@ def download_crowdin(base_path, branch, xml, username, config):
             print('WARNING: Cannot determine project root dir of '
                   '[%s], skipping.' % path)
             continue
-        result = path.split('/res')[0].strip('/')
+
+        # Usually the project root is everything before /res
+        # but there are special cases where /res is part of the repo name as well
+        parts = path.split("/res")
+        if len(parts) == 2:
+            result = parts[0]
+        elif len(parts) == 3:
+            result = parts[0] + '/res' + parts[1]
+        else:
+            print('WARNING: Splitting the path not successful for [%s], skipping' % path)
+            continue
+
+        result = result.strip('/')
         if result == path.strip('/'):
             print('WARNING: Cannot determine project root dir of '
                   '[%s], skipping.' % path)
@@ -439,26 +545,44 @@ def download_crowdin(base_path, branch, xml, username, config):
         # project in all_projects and check if it's already in there.
         all_projects.append(result)
 
-        # Search %(branch)/platform_manifest/*_default.xml or
+        # Search AICP/platform_manifest/*.xml or
         # config/%(branch)_extra_packages.xml for the project's name
+        resultPath = None
+        resultProject = None
         for project in items:
             path = project.attributes['path'].value
             if not (result + '/').startswith(path +'/'):
                 continue
-            if result != path:
-                if path in all_projects:
-                    break
-                result = path
-                all_projects.append(result)
+            # We want the longest match, so projects in subfolders of other projects are also
+            # taken into account
+            if resultPath is None or len(path) > len(resultPath):
+                resultPath = path
+                resultProject = project
+
+        # Just in case no project was found
+        if resultPath is None:
+            continue
 
-            br = project.getAttribute('revision') or branch
+        if result != resultPath:
+            if resultPath in all_projects:
+                continue
+            result = resultPath
+            all_projects.append(result)
 
-            push_as_commit(files, base_path, result,
-                           project.getAttribute('name'), br, username)
-            break
+        br = resultProject.getAttribute('revision') or branch
+
+        push_as_commit(files, base_path, result,
+                       resultProject.getAttribute('name'), br, username)
+
+
+def sig_handler(signal_received, frame):
+    print('')
+    print('SIGINT or CTRL-C detected. Exiting gracefully')
+    exit(0)
 
 
 def main():
+    signal(SIGINT, sig_handler)
     args = parse_args()
     default_branch = args.branch
 
@@ -487,14 +611,20 @@ def main():
     if xml_default is None:
         sys.exit(1)
 
+#    xml_aosp = load_xml(x='%s/platform_manifest/aicp-aosp.xml' % base_path)
+#    if xml_aosp is None:
+#        sys.exit(1)
+
     xml_extra = load_xml(x='%s/config/%s_extra_packages.xml' % (_DIR, default_branch))
     if xml_extra is None:
         sys.exit(1)
 
     xml_aicp = load_xml(x='%s/platform_manifest/aicp-default.xml' % base_path)
     if xml_aicp is not None:
+#        xml_files = (xml_default, xml_aosp, xml_aicp, xml_extra)
         xml_files = (xml_default, xml_aicp, xml_extra)
     else:
+#        xml_files = (xml_default, xml_aosp, xml_extra)
         xml_files = (xml_default, xml_extra)
 
     if args.config:
@@ -514,13 +644,12 @@ def main():
     if args.upload_translations:
         upload_translations_crowdin(default_branch, args.config)
 
-    if args.download:
-        download_crowdin(base_path, default_branch, xml_files,
-                         args.username, args.config)
-
     if args.local_download:
         local_download(base_path, default_branch, xml_files, args.config)
 
+    if args.download:
+        download_crowdin(base_path, default_branch, xml_files, args.username, args.config)
+
     if _COMMITS_CREATED:
         print('\nDone!')
         sys.exit(0)