r1
This commit is contained in:
@@ -0,0 +1,210 @@
|
||||
Contents
|
||||
========
|
||||
This directory contains tools for developers working on this repository.
|
||||
|
||||
check-doc.py
|
||||
============
|
||||
|
||||
Check if all command line args are documented. The return value indicates the
|
||||
number of undocumented args.
|
||||
|
||||
clang-format-diff.py
|
||||
===================
|
||||
|
||||
A script to format unified git diffs according to [.clang-format](../../src/.clang-format).
|
||||
|
||||
Requires `clang-format`, installed e.g. via `brew install clang-format` on macOS.
|
||||
|
||||
For instance, to format the last commit with 0 lines of context,
|
||||
the script should be called from the git root folder as follows.
|
||||
|
||||
```
|
||||
git diff -U0 HEAD~1.. | ./contrib/devtools/clang-format-diff.py -p1 -i -v
|
||||
```
|
||||
|
||||
copyright\_header.py
|
||||
====================
|
||||
|
||||
Provides utilities for managing copyright headers of `The PIVX
|
||||
developers` in repository source files. It has three subcommands:
|
||||
|
||||
```
|
||||
$ ./copyright_header.py report <base_directory> [verbose]
|
||||
$ ./copyright_header.py update <base_directory>
|
||||
$ ./copyright_header.py insert <file>
|
||||
```
|
||||
Running these subcommands without arguments displays a usage string.
|
||||
|
||||
copyright\_header.py report \<base\_directory\> [verbose]
|
||||
---------------------------------------------------------
|
||||
|
||||
Produces a report of all copyright header notices found inside the source files
|
||||
of a repository. Useful to quickly visualize the state of the headers.
|
||||
Specifying `verbose` will list the full filenames of files of each category.
|
||||
|
||||
copyright\_header.py update \<base\_directory\> [verbose]
|
||||
---------------------------------------------------------
|
||||
Updates all the copyright headers of `The PIVX developers` which were
|
||||
changed in a year more recent than is listed. For example:
|
||||
```
|
||||
// Copyright (c) <firstYear>-<lastYear> The PIVX developers
|
||||
```
|
||||
will be updated to:
|
||||
```
|
||||
// Copyright (c) <firstYear>-<lastModifiedYear> The PIVX developers
|
||||
```
|
||||
where `<lastModifiedYear>` is obtained from the `git log` history.
|
||||
|
||||
This subcommand also handles copyright headers that have only a single year. In
|
||||
those cases:
|
||||
```
|
||||
// Copyright (c) <year> The PIVX developers
|
||||
```
|
||||
will be updated to:
|
||||
```
|
||||
// Copyright (c) <year>-<lastModifiedYear> The PIVX developers
|
||||
```
|
||||
where the update is appropriate.
|
||||
|
||||
copyright\_header.py insert \<file\>
|
||||
------------------------------------
|
||||
Inserts a copyright header for `The PIVX developers` at the top of the
|
||||
file in either Python or C++ style as determined by the file extension. If the
|
||||
file is a Python file and it has `#!` starting the first line, the header is
|
||||
inserted in the line below it.
|
||||
|
||||
The copyright dates will be set to be `<year_introduced>-<current_year>` where
|
||||
`<year_introduced>` is according to the `git log` history. If
|
||||
`<year_introduced>` is equal to `<current_year>`, it will be set as a single
|
||||
year rather than two hyphenated years.
|
||||
|
||||
If the file already has a copyright for `The PIVX developers`, the
|
||||
script will exit.
|
||||
|
||||
gen-manpages.sh
|
||||
===============
|
||||
|
||||
A small script to automatically create manpages in ../../doc/man by running the release binaries with the -help option.
|
||||
This requires help2man which can be found at: https://www.gnu.org/software/help2man/
|
||||
|
||||
With in-tree builds this tool can be run from any directory within the
|
||||
repostitory. To use this tool with out-of-tree builds set `BUILDDIR`. For
|
||||
example:
|
||||
|
||||
```bash
|
||||
BUILDDIR=$PWD/build contrib/devtools/gen-manpages.sh
|
||||
```
|
||||
|
||||
github-merge.py
|
||||
===============
|
||||
|
||||
A small script to automate merging pull-requests securely and sign them with GPG.
|
||||
|
||||
For example:
|
||||
|
||||
./github-merge.py 3077
|
||||
|
||||
(in any git repository) will help you merge pull request #3077 for the
|
||||
Agrarian-Project/Agrarian repository.
|
||||
|
||||
What it does:
|
||||
* Fetch master and the pull request.
|
||||
* Locally construct a merge commit.
|
||||
* Show the diff that merge results in.
|
||||
* Ask you to verify the resulting source tree (so you can do a make
|
||||
check or whatever).
|
||||
* Ask you whether to GPG sign the merge commit.
|
||||
* Ask you whether to push the result upstream.
|
||||
|
||||
This means that there are no potential race conditions (where a
|
||||
pullreq gets updated while you're reviewing it, but before you click
|
||||
merge), and when using GPG signatures, that even a compromised GitHub
|
||||
couldn't mess with the sources.
|
||||
|
||||
Setup
|
||||
---------
|
||||
Configuring the github-merge tool for the Agrarian repository is done in the following way:
|
||||
|
||||
git config githubmerge.repository Agrarian-Project/Agrarian
|
||||
git config githubmerge.testcmd "make -j4 check" (adapt to whatever you want to use for testing)
|
||||
git config --global user.signingkey mykeyid
|
||||
|
||||
Authentication (optional)
|
||||
--------------------------
|
||||
|
||||
The API request limit for unauthenticated requests is quite low, but the
|
||||
limit for authenticated requests is much higher. If you start running
|
||||
into rate limiting errors it can be useful to set an authentication token
|
||||
so that the script can authenticate requests.
|
||||
|
||||
- First, go to [Personal access tokens](https://github.com/settings/tokens).
|
||||
- Click 'Generate new token'.
|
||||
- Fill in an arbitrary token description. No further privileges are needed.
|
||||
- Click the `Generate token` button at the bottom of the form.
|
||||
- Copy the generated token (should be a hexadecimal string)
|
||||
|
||||
Then do:
|
||||
|
||||
git config --global user.ghtoken "pasted token"
|
||||
|
||||
Create and verify timestamps of merge commits
|
||||
---------------------------------------------
|
||||
To create or verify timestamps on the merge commits, install the OpenTimestamps
|
||||
client via `pip3 install opentimestamps-client`. Then, dowload the gpg wrapper
|
||||
`ots-git-gpg-wrapper.sh` and set it as git's `gpg.program`. See
|
||||
[the ots git integration documentation](https://github.com/opentimestamps/opentimestamps-client/blob/master/doc/git-integration.md#usage)
|
||||
for further details.
|
||||
|
||||
optimize-pngs.py
|
||||
================
|
||||
|
||||
A script to optimize png files in the Agrarian
|
||||
repository (requires pngcrush).
|
||||
|
||||
security-check.py and test-security-check.py
|
||||
============================================
|
||||
|
||||
Perform basic ELF security checks on a series of executables.
|
||||
|
||||
symbol-check.py
|
||||
===============
|
||||
|
||||
A script to check that the (Linux) executables produced by gitian only contain
|
||||
allowed gcc, glibc and libstdc++ version symbols. This makes sure they are
|
||||
still compatible with the minimum supported Linux distribution versions.
|
||||
|
||||
Example usage after a gitian build:
|
||||
|
||||
find ../gitian-builder/build -type f -executable | xargs python3 contrib/devtools/symbol-check.py
|
||||
|
||||
If only supported symbols are used the return value will be 0 and the output will be empty.
|
||||
|
||||
If there are 'unsupported' symbols, the return value will be 1 a list like this will be printed:
|
||||
|
||||
.../64/test_agrarian: symbol memcpy from unsupported version GLIBC_2.14
|
||||
.../64/test_agrarian: symbol __fdelt_chk from unsupported version GLIBC_2.15
|
||||
.../64/test_agrarian: symbol std::out_of_range::~out_of_range() from unsupported version GLIBCXX_3.4.15
|
||||
.../64/test_agrarian: symbol _ZNSt8__detail15_List_nod from unsupported version GLIBCXX_3.4.15
|
||||
|
||||
update-translations.py
|
||||
======================
|
||||
|
||||
Run this script from the root of the repository to update all translations from transifex.
|
||||
It will do the following automatically:
|
||||
|
||||
- fetch all translations
|
||||
- post-process them into valid and committable format
|
||||
- add missing translations to the build system (TODO)
|
||||
|
||||
See doc/translation-process.md for more information.
|
||||
|
||||
circular-dependencies.py
|
||||
========================
|
||||
|
||||
Run this script from the root of the source tree (`src/`) to find circular dependencies in the source code.
|
||||
This looks only at which files include other files, treating the `.cpp` and `.h` file as one unit.
|
||||
|
||||
Example usage:
|
||||
|
||||
cd .../src
|
||||
../contrib/devtools/circular-dependencies.py {*,*/*,*/*/*}.{h,cpp}
|
||||
@@ -0,0 +1,47 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2015-2018 The Bitcoin Core developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
|
||||
'''
|
||||
This checks if all command line args are documented.
|
||||
Return value is 0 to indicate no error.
|
||||
|
||||
Author: @MarcoFalke
|
||||
'''
|
||||
|
||||
from subprocess import check_output
|
||||
import re
|
||||
import sys
|
||||
|
||||
FOLDER_GREP = 'src'
|
||||
FOLDER_TEST = 'src/test/'
|
||||
CMD_ROOT_DIR = '`git rev-parse --show-toplevel`/{}'.format(FOLDER_GREP)
|
||||
CMD_GREP_ARGS = r"egrep -r -I '(map(Multi)?Args(\.count\(|\[)|Get(Bool)?Arg\()\"\-[^\"]+?\"' {} | grep -v '{}'".format(CMD_ROOT_DIR, FOLDER_TEST)
|
||||
CMD_GREP_DOCS = r"egrep -r -I 'HelpMessageOpt\(\"\-[^\"=]+?(=|\")' {}".format(CMD_ROOT_DIR)
|
||||
REGEX_ARG = re.compile(r'(?:map(?:Multi)?Args(?:\.count\(|\[)|Get(?:Bool)?Arg\()\"(\-[^\"]+?)\"')
|
||||
REGEX_DOC = re.compile(r'HelpMessageOpt\(\"(\-[^\"=]+?)(?:=|\")')
|
||||
# list unsupported, deprecated and duplicate args as they need no documentation
|
||||
SET_DOC_OPTIONAL = set(['-rpcssl', '-benchmark', '-h', '-help', '-socks', '-tor', '-debugnet', '-whitelistalwaysrelay', '-prematurewitness', '-walletprematurewitness', '-promiscuousmempoolflags', '-blockminsize', '-sendfreetransactions', '-checklevel', '-liquidityprovider', '-anonymizeagrarianamount'])
|
||||
|
||||
def main():
|
||||
used = check_output(CMD_GREP_ARGS, shell=True, universal_newlines=True)
|
||||
docd = check_output(CMD_GREP_DOCS, shell=True, universal_newlines=True)
|
||||
|
||||
args_used = set(re.findall(re.compile(REGEX_ARG), used))
|
||||
args_docd = set(re.findall(re.compile(REGEX_DOC), docd)).union(SET_DOC_OPTIONAL)
|
||||
args_need_doc = args_used.difference(args_docd)
|
||||
args_unknown = args_docd.difference(args_used)
|
||||
|
||||
print("Args used : {}".format(len(args_used)))
|
||||
print("Args documented : {}".format(len(args_docd)))
|
||||
print("Args undocumented: {}".format(len(args_need_doc)))
|
||||
print(args_need_doc)
|
||||
print("Args unknown : {}".format(len(args_unknown)))
|
||||
print(args_unknown)
|
||||
|
||||
sys.exit(len(args_need_doc))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,88 @@
|
||||
#!/usr/bin/env python3
|
||||
|
||||
import sys
|
||||
import re
|
||||
|
||||
MAPPING = {
|
||||
'core_read.cpp': 'core_io.cpp',
|
||||
'core_write.cpp': 'core_io.cpp',
|
||||
}
|
||||
|
||||
# Directories with header-based modules, where the assumption that .cpp files
|
||||
# define functions and variables declared in corresponding .h files is
|
||||
# incorrect.
|
||||
HEADER_MODULE_PATHS = [
|
||||
'interfaces/'
|
||||
]
|
||||
|
||||
def module_name(path):
|
||||
if path in MAPPING:
|
||||
path = MAPPING[path]
|
||||
if any(path.startswith(dirpath) for dirpath in HEADER_MODULE_PATHS):
|
||||
return path
|
||||
if path.endswith(".h"):
|
||||
return path[:-2]
|
||||
if path.endswith(".c"):
|
||||
return path[:-2]
|
||||
if path.endswith(".cpp"):
|
||||
return path[:-4]
|
||||
return None
|
||||
|
||||
files = dict()
|
||||
deps = dict()
|
||||
|
||||
RE = re.compile("^#include \"(.*)\"")
|
||||
|
||||
# Iterate over files, and create list of modules
|
||||
for arg in sys.argv[1:]:
|
||||
module = module_name(arg)
|
||||
if module is None:
|
||||
print("Ignoring file %s (does not constitute module)\n" % arg)
|
||||
else:
|
||||
files[arg] = module
|
||||
deps[module] = set()
|
||||
|
||||
# Iterate again, and build list of direct dependencies for each module
|
||||
# TODO: implement support for multiple include directories
|
||||
for arg in sorted(files.keys()):
|
||||
module = files[arg]
|
||||
with open(arg, 'r', encoding="utf8") as f:
|
||||
for line in f:
|
||||
match = RE.match(line)
|
||||
if match:
|
||||
include = match.group(1)
|
||||
included_module = module_name(include)
|
||||
if included_module is not None and included_module in deps and included_module != module:
|
||||
deps[module].add(included_module)
|
||||
|
||||
# Loop to find the shortest (remaining) circular dependency
|
||||
have_cycle = False
|
||||
while True:
|
||||
shortest_cycle = None
|
||||
for module in sorted(deps.keys()):
|
||||
# Build the transitive closure of dependencies of module
|
||||
closure = dict()
|
||||
for dep in deps[module]:
|
||||
closure[dep] = []
|
||||
while True:
|
||||
old_size = len(closure)
|
||||
old_closure_keys = sorted(closure.keys())
|
||||
for src in old_closure_keys:
|
||||
for dep in deps[src]:
|
||||
if dep not in closure:
|
||||
closure[dep] = closure[src] + [src]
|
||||
if len(closure) == old_size:
|
||||
break
|
||||
# If module is in its own transitive closure, it's a circular dependency; check if it is the shortest
|
||||
if module in closure and (shortest_cycle is None or len(closure[module]) + 1 < len(shortest_cycle)):
|
||||
shortest_cycle = [module] + closure[module]
|
||||
if shortest_cycle is None:
|
||||
break
|
||||
# We have the shortest circular dependency; report it
|
||||
module = shortest_cycle[0]
|
||||
print("Circular dependency: %s" % (" -> ".join(shortest_cycle + [module])))
|
||||
# And then break the dependency to avoid repeating in other cycles
|
||||
deps[shortest_cycle[-1]] = deps[shortest_cycle[-1]] - set([module])
|
||||
have_cycle = True
|
||||
|
||||
sys.exit(1 if have_cycle else 0)
|
||||
@@ -0,0 +1,166 @@
|
||||
#!/usr/bin/env python3
|
||||
#
|
||||
#===- clang-format-diff.py - ClangFormat Diff Reformatter ----*- python -*--===#
|
||||
#
|
||||
# The LLVM Compiler Infrastructure
|
||||
#
|
||||
# This file is distributed under the University of Illinois Open Source
|
||||
# License.
|
||||
#
|
||||
# ============================================================
|
||||
#
|
||||
# University of Illinois/NCSA
|
||||
# Open Source License
|
||||
#
|
||||
# Copyright (c) 2007-2015 University of Illinois at Urbana-Champaign.
|
||||
# All rights reserved.
|
||||
#
|
||||
# Developed by:
|
||||
#
|
||||
# LLVM Team
|
||||
#
|
||||
# University of Illinois at Urbana-Champaign
|
||||
#
|
||||
# http://llvm.org
|
||||
#
|
||||
# Permission is hereby granted, free of charge, to any person obtaining a copy of
|
||||
# this software and associated documentation files (the "Software"), to deal with
|
||||
# the Software without restriction, including without limitation the rights to
|
||||
# use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies
|
||||
# of the Software, and to permit persons to whom the Software is furnished to do
|
||||
# so, subject to the following conditions:
|
||||
#
|
||||
# * Redistributions of source code must retain the above copyright notice,
|
||||
# this list of conditions and the following disclaimers.
|
||||
#
|
||||
# * Redistributions in binary form must reproduce the above copyright notice,
|
||||
# this list of conditions and the following disclaimers in the
|
||||
# documentation and/or other materials provided with the distribution.
|
||||
#
|
||||
# * Neither the names of the LLVM Team, University of Illinois at
|
||||
# Urbana-Champaign, nor the names of its contributors may be used to
|
||||
# endorse or promote products derived from this Software without specific
|
||||
# prior written permission.
|
||||
#
|
||||
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
||||
# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS
|
||||
# FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
||||
# CONTRIBUTORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
||||
# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
||||
# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS WITH THE
|
||||
# SOFTWARE.
|
||||
#
|
||||
# ============================================================
|
||||
#
|
||||
#===------------------------------------------------------------------------===#
|
||||
|
||||
r"""
|
||||
ClangFormat Diff Reformatter
|
||||
============================
|
||||
|
||||
This script reads input from a unified diff and reformats all the changed
|
||||
lines. This is useful to reformat all the lines touched by a specific patch.
|
||||
Example usage for git/svn users:
|
||||
|
||||
git diff -U0 HEAD^ | clang-format-diff.py -p1 -i
|
||||
svn diff --diff-cmd=diff -x-U0 | clang-format-diff.py -i
|
||||
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import difflib
|
||||
import io
|
||||
import re
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
|
||||
# Change this to the full path if clang-format is not on the path.
|
||||
binary = 'clang-format'
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=
|
||||
'Reformat changed lines in diff. Without -i '
|
||||
'option just output the diff that would be '
|
||||
'introduced.')
|
||||
parser.add_argument('-i', action='store_true', default=False,
|
||||
help='apply edits to files instead of displaying a diff')
|
||||
parser.add_argument('-p', metavar='NUM', default=0,
|
||||
help='strip the smallest prefix containing P slashes')
|
||||
parser.add_argument('-regex', metavar='PATTERN', default=None,
|
||||
help='custom pattern selecting file paths to reformat '
|
||||
'(case sensitive, overrides -iregex)')
|
||||
parser.add_argument('-iregex', metavar='PATTERN', default=
|
||||
r'.*\.(cpp|cc|c\+\+|cxx|c|cl|h|hpp|m|mm|inc|js|ts|proto'
|
||||
r'|protodevel|java)',
|
||||
help='custom pattern selecting file paths to reformat '
|
||||
'(case insensitive, overridden by -regex)')
|
||||
parser.add_argument('-sort-includes', action='store_true', default=False,
|
||||
help='let clang-format sort include blocks')
|
||||
parser.add_argument('-v', '--verbose', action='store_true',
|
||||
help='be more verbose, ineffective without -i')
|
||||
args = parser.parse_args()
|
||||
|
||||
# Extract changed lines for each file.
|
||||
filename = None
|
||||
lines_by_file = {}
|
||||
for line in sys.stdin:
|
||||
match = re.search('^\+\+\+\ (.*?/){%s}(\S*)' % args.p, line)
|
||||
if match:
|
||||
filename = match.group(2)
|
||||
if filename is None:
|
||||
continue
|
||||
|
||||
if args.regex is not None:
|
||||
if not re.match('^%s$' % args.regex, filename):
|
||||
continue
|
||||
else:
|
||||
if not re.match('^%s$' % args.iregex, filename, re.IGNORECASE):
|
||||
continue
|
||||
|
||||
match = re.search('^@@.*\+(\d+)(,(\d+))?', line)
|
||||
if match:
|
||||
start_line = int(match.group(1))
|
||||
line_count = 1
|
||||
if match.group(3):
|
||||
line_count = int(match.group(3))
|
||||
if line_count == 0:
|
||||
continue
|
||||
end_line = start_line + line_count - 1
|
||||
lines_by_file.setdefault(filename, []).extend(
|
||||
['-lines', str(start_line) + ':' + str(end_line)])
|
||||
|
||||
# Reformat files containing changes in place.
|
||||
for filename, lines in lines_by_file.items():
|
||||
if args.i and args.verbose:
|
||||
print('Formatting {}'.format(filename))
|
||||
command = [binary, filename]
|
||||
if args.i:
|
||||
command.append('-i')
|
||||
if args.sort_includes:
|
||||
command.append('-sort-includes')
|
||||
command.extend(lines)
|
||||
command.extend(['-style=file', '-fallback-style=none'])
|
||||
p = subprocess.Popen(command,
|
||||
stdout=subprocess.PIPE,
|
||||
stderr=None,
|
||||
stdin=subprocess.PIPE,
|
||||
universal_newlines=True)
|
||||
stdout, stderr = p.communicate()
|
||||
if p.returncode != 0:
|
||||
sys.exit(p.returncode)
|
||||
|
||||
if not args.i:
|
||||
with open(filename, encoding="utf8") as f:
|
||||
code = f.readlines()
|
||||
formatted_code = io.StringIO(stdout).readlines()
|
||||
diff = difflib.unified_diff(code, formatted_code,
|
||||
filename, filename,
|
||||
'(before formatting)', '(after formatting)')
|
||||
diff_string = ''.join(diff)
|
||||
if len(diff_string) > 0:
|
||||
sys.stdout.write(diff_string)
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,46 @@
|
||||
#!/bin/sh
|
||||
# Copyright (c) 2017 The Bitcoin Core developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
|
||||
# This simple script checks for commits beginning with: scripted-diff:
|
||||
# If found, looks for a script between the lines -BEGIN VERIFY SCRIPT- and
|
||||
# -END VERIFY SCRIPT-. If no ending is found, it reads until the end of the
|
||||
# commit message.
|
||||
|
||||
# The resulting script should exactly transform the previous commit into the current
|
||||
# one. Any remaining diff signals an error.
|
||||
|
||||
if test "x$1" = "x"; then
|
||||
echo "Usage: $0 <commit>..."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
RET=0
|
||||
PREV_BRANCH=`git name-rev --name-only HEAD`
|
||||
PREV_HEAD=`git rev-parse HEAD`
|
||||
for i in `git rev-list --reverse $1`; do
|
||||
if git rev-list -n 1 --pretty="%s" $i | grep -q "^scripted-diff:"; then
|
||||
git checkout --quiet $i^ || exit
|
||||
SCRIPT="`git rev-list --format=%b -n1 $i | sed '/^-BEGIN VERIFY SCRIPT-$/,/^-END VERIFY SCRIPT-$/{//!b};d'`"
|
||||
if test "x$SCRIPT" = "x"; then
|
||||
echo "Error: missing script for: $i"
|
||||
echo "Failed"
|
||||
RET=1
|
||||
else
|
||||
echo "Running script for: $i"
|
||||
echo "$SCRIPT"
|
||||
eval "$SCRIPT"
|
||||
git --no-pager diff --exit-code $i && echo "OK" || (echo "Failed"; false) || RET=1
|
||||
fi
|
||||
git reset --quiet --hard HEAD
|
||||
else
|
||||
if git rev-list "--format=%b" -n1 $i | grep -q '^-\(BEGIN\|END\)[ a-zA-Z]*-$'; then
|
||||
echo "Error: script block marker but no scripted-diff in title"
|
||||
echo "Failed"
|
||||
RET=1
|
||||
fi
|
||||
fi
|
||||
done
|
||||
git checkout --quiet $PREV_BRANCH 2>/dev/null || git checkout --quiet $PREV_HEAD
|
||||
exit $RET
|
||||
@@ -0,0 +1,620 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2016-2018 The Bitcoin Core developers
|
||||
# Copyright (c) 2018-2019 The PIVX developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
|
||||
import re
|
||||
import fnmatch
|
||||
import sys
|
||||
import subprocess
|
||||
import datetime
|
||||
import os
|
||||
|
||||
################################################################################
|
||||
# file filtering
|
||||
################################################################################
|
||||
|
||||
EXCLUDE = [
|
||||
# auto generated:
|
||||
'src/qt/agrarianstrings.cpp',
|
||||
'src/chainparamsseeds.h',
|
||||
# other external copyrights:
|
||||
'src/tinyformat.h',
|
||||
'src/crypto/scrypt.cpp',
|
||||
'test/functional/test_framework/bignum.py',
|
||||
# python init:
|
||||
'*__init__.py',
|
||||
]
|
||||
EXCLUDE_COMPILED = re.compile('|'.join([fnmatch.translate(m) for m in EXCLUDE]))
|
||||
|
||||
EXCLUDE_DIRS = [
|
||||
# git subtrees
|
||||
"src/crypto/ctaes/",
|
||||
"src/leveldb/",
|
||||
"src/secp256k1/",
|
||||
"src/univalue/",
|
||||
]
|
||||
|
||||
INCLUDE = ['*.h', '*.cpp', '*.cc', '*.c', '*.py']
|
||||
INCLUDE_COMPILED = re.compile('|'.join([fnmatch.translate(m) for m in INCLUDE]))
|
||||
|
||||
def applies_to_file(filename):
|
||||
for excluded_dir in EXCLUDE_DIRS:
|
||||
if filename.startswith(excluded_dir):
|
||||
return False
|
||||
return ((EXCLUDE_COMPILED.match(filename) is None) and
|
||||
(INCLUDE_COMPILED.match(filename) is not None))
|
||||
|
||||
################################################################################
|
||||
# obtain list of files in repo according to INCLUDE and EXCLUDE
|
||||
################################################################################
|
||||
|
||||
GIT_LS_CMD = 'git ls-files --full-name'.split(' ')
|
||||
GIT_TOPLEVEL_CMD = 'git rev-parse --show-toplevel'.split(' ')
|
||||
|
||||
def call_git_ls(base_directory):
|
||||
out = subprocess.check_output([*GIT_LS_CMD, base_directory])
|
||||
return [f for f in out.decode("utf-8").split('\n') if f != '']
|
||||
|
||||
def call_git_toplevel():
|
||||
"Returns the absolute path to the project root"
|
||||
return subprocess.check_output(GIT_TOPLEVEL_CMD).strip().decode("utf-8")
|
||||
|
||||
def get_filenames_to_examine(base_directory):
|
||||
"Returns an array of absolute paths to any project files in the base_directory that pass the include/exclude filters"
|
||||
root = call_git_toplevel()
|
||||
filenames = call_git_ls(base_directory)
|
||||
return sorted([os.path.join(root, filename) for filename in filenames if
|
||||
applies_to_file(filename)])
|
||||
|
||||
################################################################################
|
||||
# define and compile regexes for the patterns we are looking for
|
||||
################################################################################
|
||||
|
||||
|
||||
COPYRIGHT_WITH_C = 'Copyright \(c\)'
|
||||
COPYRIGHT_WITHOUT_C = 'Copyright'
|
||||
ANY_COPYRIGHT_STYLE = '(%s|%s)' % (COPYRIGHT_WITH_C, COPYRIGHT_WITHOUT_C)
|
||||
|
||||
YEAR = "20[0-9][0-9]"
|
||||
YEAR_RANGE = '(%s)(-%s)?' % (YEAR, YEAR)
|
||||
YEAR_LIST = '(%s)(, %s)+' % (YEAR, YEAR)
|
||||
ANY_YEAR_STYLE = '(%s|%s)' % (YEAR_RANGE, YEAR_LIST)
|
||||
ANY_COPYRIGHT_STYLE_OR_YEAR_STYLE = ("%s %s" % (ANY_COPYRIGHT_STYLE,
|
||||
ANY_YEAR_STYLE))
|
||||
|
||||
ANY_COPYRIGHT_COMPILED = re.compile(ANY_COPYRIGHT_STYLE_OR_YEAR_STYLE)
|
||||
|
||||
def compile_copyright_regex(copyright_style, year_style, name):
|
||||
return re.compile('%s %s,? %s' % (copyright_style, year_style, name))
|
||||
|
||||
EXPECTED_HOLDER_NAMES = [
|
||||
"Satoshi Nakamoto\n",
|
||||
"The Bitcoin Core developers\n",
|
||||
"The Bitcoin Core developers \n",
|
||||
"Bitcoin Core Developers\n",
|
||||
"the Bitcoin Core developers\n",
|
||||
"The Bitcoin developers\n",
|
||||
"The LevelDB Authors\. All rights reserved\.\n",
|
||||
"BitPay Inc\.\n",
|
||||
"BitPay, Inc\.\n",
|
||||
"University of Illinois at Urbana-Champaign\.\n",
|
||||
"MarcoFalke\n",
|
||||
"Pieter Wuille\n",
|
||||
"Pieter Wuille +\*\n",
|
||||
"Pieter Wuille, Gregory Maxwell +\*\n",
|
||||
"Pieter Wuille, Andrew Poelstra +\*\n",
|
||||
"Ian Miers, Christina Garman and Matthew Green\n",
|
||||
"Andrew Poelstra +\*\n",
|
||||
"Wladimir J. van der Laan\n",
|
||||
"Jeff Garzik\n",
|
||||
"Diederik Huys, Pieter Wuille +\*\n",
|
||||
"Thomas Daede, Cory Fields +\*\n",
|
||||
"Jan-Klaas Kollhof\n",
|
||||
"Sam Rushing\n",
|
||||
"ArtForz -- public domain half-a-node\n",
|
||||
" Projet RNRT SAPHIR\n",
|
||||
"The Zcash developers\n",
|
||||
"The Dash developers\n",
|
||||
"The Dash Developers\n",
|
||||
"The Dash Core developers\n",
|
||||
"The Agrarian developers\n",
|
||||
"The PPCoin developers\n",
|
||||
]
|
||||
|
||||
DOMINANT_STYLE_COMPILED = {}
|
||||
YEAR_LIST_STYLE_COMPILED = {}
|
||||
WITHOUT_C_STYLE_COMPILED = {}
|
||||
|
||||
for holder_name in EXPECTED_HOLDER_NAMES:
|
||||
DOMINANT_STYLE_COMPILED[holder_name] = (
|
||||
compile_copyright_regex(COPYRIGHT_WITH_C, YEAR_RANGE, holder_name))
|
||||
YEAR_LIST_STYLE_COMPILED[holder_name] = (
|
||||
compile_copyright_regex(COPYRIGHT_WITH_C, YEAR_LIST, holder_name))
|
||||
WITHOUT_C_STYLE_COMPILED[holder_name] = (
|
||||
compile_copyright_regex(COPYRIGHT_WITHOUT_C, ANY_YEAR_STYLE,
|
||||
holder_name))
|
||||
|
||||
################################################################################
|
||||
# search file contents for copyright message of particular category
|
||||
################################################################################
|
||||
|
||||
def get_count_of_copyrights_of_any_style_any_holder(contents):
|
||||
return len(ANY_COPYRIGHT_COMPILED.findall(contents))
|
||||
|
||||
def file_has_dominant_style_copyright_for_holder(contents, holder_name):
|
||||
match = DOMINANT_STYLE_COMPILED[holder_name].search(contents)
|
||||
return match is not None
|
||||
|
||||
def file_has_year_list_style_copyright_for_holder(contents, holder_name):
|
||||
match = YEAR_LIST_STYLE_COMPILED[holder_name].search(contents)
|
||||
return match is not None
|
||||
|
||||
def file_has_without_c_style_copyright_for_holder(contents, holder_name):
|
||||
match = WITHOUT_C_STYLE_COMPILED[holder_name].search(contents)
|
||||
return match is not None
|
||||
|
||||
################################################################################
|
||||
# get file info
|
||||
################################################################################
|
||||
|
||||
def read_file(filename):
|
||||
return open(filename, 'r', encoding="utf8").read()
|
||||
|
||||
def gather_file_info(filename):
|
||||
info = {}
|
||||
info['filename'] = filename
|
||||
c = read_file(filename)
|
||||
info['contents'] = c
|
||||
|
||||
info['all_copyrights'] = get_count_of_copyrights_of_any_style_any_holder(c)
|
||||
|
||||
info['classified_copyrights'] = 0
|
||||
info['dominant_style'] = {}
|
||||
info['year_list_style'] = {}
|
||||
info['without_c_style'] = {}
|
||||
for holder_name in EXPECTED_HOLDER_NAMES:
|
||||
has_dominant_style = (
|
||||
file_has_dominant_style_copyright_for_holder(c, holder_name))
|
||||
has_year_list_style = (
|
||||
file_has_year_list_style_copyright_for_holder(c, holder_name))
|
||||
has_without_c_style = (
|
||||
file_has_without_c_style_copyright_for_holder(c, holder_name))
|
||||
info['dominant_style'][holder_name] = has_dominant_style
|
||||
info['year_list_style'][holder_name] = has_year_list_style
|
||||
info['without_c_style'][holder_name] = has_without_c_style
|
||||
if has_dominant_style or has_year_list_style or has_without_c_style:
|
||||
info['classified_copyrights'] = info['classified_copyrights'] + 1
|
||||
return info
|
||||
|
||||
################################################################################
|
||||
# report execution
|
||||
################################################################################
|
||||
|
||||
SEPARATOR = '-'.join(['' for _ in range(80)])
|
||||
|
||||
def print_filenames(filenames, verbose):
|
||||
if not verbose:
|
||||
return
|
||||
for filename in filenames:
|
||||
print("\t%s" % filename)
|
||||
|
||||
def print_report(file_infos, verbose):
|
||||
print(SEPARATOR)
|
||||
examined = [i['filename'] for i in file_infos]
|
||||
print("%d files examined according to INCLUDE and EXCLUDE fnmatch rules" %
|
||||
len(examined))
|
||||
print_filenames(examined, verbose)
|
||||
|
||||
print(SEPARATOR)
|
||||
print('')
|
||||
zero_copyrights = [i['filename'] for i in file_infos if
|
||||
i['all_copyrights'] == 0]
|
||||
print("%4d with zero copyrights" % len(zero_copyrights))
|
||||
print_filenames(zero_copyrights, verbose)
|
||||
one_copyright = [i['filename'] for i in file_infos if
|
||||
i['all_copyrights'] == 1]
|
||||
print("%4d with one copyright" % len(one_copyright))
|
||||
print_filenames(one_copyright, verbose)
|
||||
two_copyrights = [i['filename'] for i in file_infos if
|
||||
i['all_copyrights'] == 2]
|
||||
print("%4d with two copyrights" % len(two_copyrights))
|
||||
print_filenames(two_copyrights, verbose)
|
||||
three_copyrights = [i['filename'] for i in file_infos if
|
||||
i['all_copyrights'] == 3]
|
||||
print("%4d with three copyrights" % len(three_copyrights))
|
||||
print_filenames(three_copyrights, verbose)
|
||||
four_or_more_copyrights = [i['filename'] for i in file_infos if
|
||||
i['all_copyrights'] >= 4]
|
||||
print("%4d with four or more copyrights" % len(four_or_more_copyrights))
|
||||
print_filenames(four_or_more_copyrights, verbose)
|
||||
print('')
|
||||
print(SEPARATOR)
|
||||
print('Copyrights with dominant style:\ne.g. "Copyright (c)" and '
|
||||
'"<year>" or "<startYear>-<endYear>":\n')
|
||||
for holder_name in EXPECTED_HOLDER_NAMES:
|
||||
dominant_style = [i['filename'] for i in file_infos if
|
||||
i['dominant_style'][holder_name]]
|
||||
if len(dominant_style) > 0:
|
||||
print("%4d with '%s'" % (len(dominant_style),
|
||||
holder_name.replace('\n', '\\n')))
|
||||
print_filenames(dominant_style, verbose)
|
||||
print('')
|
||||
print(SEPARATOR)
|
||||
print('Copyrights with year list style:\ne.g. "Copyright (c)" and '
|
||||
'"<year1>, <year2>, ...":\n')
|
||||
for holder_name in EXPECTED_HOLDER_NAMES:
|
||||
year_list_style = [i['filename'] for i in file_infos if
|
||||
i['year_list_style'][holder_name]]
|
||||
if len(year_list_style) > 0:
|
||||
print("%4d with '%s'" % (len(year_list_style),
|
||||
holder_name.replace('\n', '\\n')))
|
||||
print_filenames(year_list_style, verbose)
|
||||
print('')
|
||||
print(SEPARATOR)
|
||||
print('Copyrights with no "(c)" style:\ne.g. "Copyright" and "<year>" or '
|
||||
'"<startYear>-<endYear>":\n')
|
||||
for holder_name in EXPECTED_HOLDER_NAMES:
|
||||
without_c_style = [i['filename'] for i in file_infos if
|
||||
i['without_c_style'][holder_name]]
|
||||
if len(without_c_style) > 0:
|
||||
print("%4d with '%s'" % (len(without_c_style),
|
||||
holder_name.replace('\n', '\\n')))
|
||||
print_filenames(without_c_style, verbose)
|
||||
|
||||
print('')
|
||||
print(SEPARATOR)
|
||||
|
||||
unclassified_copyrights = [i['filename'] for i in file_infos if
|
||||
i['classified_copyrights'] < i['all_copyrights']]
|
||||
print("%d with unexpected copyright holder names" %
|
||||
len(unclassified_copyrights))
|
||||
print_filenames(unclassified_copyrights, verbose)
|
||||
print(SEPARATOR)
|
||||
|
||||
def exec_report(base_directory, verbose):
|
||||
filenames = get_filenames_to_examine(base_directory)
|
||||
file_infos = [gather_file_info(f) for f in filenames]
|
||||
print_report(file_infos, verbose)
|
||||
|
||||
################################################################################
|
||||
# report cmd
|
||||
################################################################################
|
||||
|
||||
REPORT_USAGE = """
|
||||
Produces a report of all copyright header notices found inside the source files
|
||||
of a repository.
|
||||
|
||||
Usage:
|
||||
$ ./copyright_header.py report <base_directory> [verbose]
|
||||
|
||||
Arguments:
|
||||
<base_directory> - The base directory of a bitcoin source code repository.
|
||||
[verbose] - Includes a list of every file of each subcategory in the report.
|
||||
"""
|
||||
|
||||
def report_cmd(argv):
|
||||
if len(argv) == 2:
|
||||
sys.exit(REPORT_USAGE)
|
||||
|
||||
base_directory = argv[2]
|
||||
if not os.path.exists(base_directory):
|
||||
sys.exit("*** bad <base_directory>: %s" % base_directory)
|
||||
|
||||
if len(argv) == 3:
|
||||
verbose = False
|
||||
elif argv[3] == 'verbose':
|
||||
verbose = True
|
||||
else:
|
||||
sys.exit("*** unknown argument: %s" % argv[2])
|
||||
|
||||
exec_report(base_directory, verbose)
|
||||
|
||||
################################################################################
|
||||
# query git for year of last change
|
||||
################################################################################
|
||||
|
||||
GIT_LOG_CMD = "git log --pretty=format:%%ai %s"
|
||||
|
||||
def call_git_log(filename):
|
||||
out = subprocess.check_output((GIT_LOG_CMD % filename).split(' '))
|
||||
return out.decode("utf-8").split('\n')
|
||||
|
||||
def get_git_change_years(filename):
|
||||
git_log_lines = call_git_log(filename)
|
||||
if len(git_log_lines) == 0:
|
||||
return [datetime.date.today().year]
|
||||
# timestamp is in ISO 8601 format. e.g. "2016-09-05 14:25:32 -0600"
|
||||
return [line.split(' ')[0].split('-')[0] for line in git_log_lines]
|
||||
|
||||
def get_most_recent_git_change_year(filename):
|
||||
return max(get_git_change_years(filename))
|
||||
|
||||
################################################################################
|
||||
# read and write to file
|
||||
################################################################################
|
||||
|
||||
def read_file_lines(filename):
|
||||
f = open(filename, 'r', encoding="utf8")
|
||||
file_lines = f.readlines()
|
||||
f.close()
|
||||
return file_lines
|
||||
|
||||
def write_file_lines(filename, file_lines):
|
||||
f = open(filename, 'w', encoding="utf8")
|
||||
f.write(''.join(file_lines))
|
||||
f.close()
|
||||
|
||||
################################################################################
|
||||
# update header years execution
|
||||
################################################################################
|
||||
|
||||
COPYRIGHT = 'Copyright \(c\)'
|
||||
YEAR = "20[0-9][0-9]"
|
||||
YEAR_RANGE = '(%s)(-%s)?' % (YEAR, YEAR)
|
||||
HOLDER = 'The Agrarian developers'
|
||||
UPDATEABLE_LINE_COMPILED = re.compile(' '.join([COPYRIGHT, YEAR_RANGE, HOLDER]))
|
||||
|
||||
def get_updatable_copyright_line(file_lines):
|
||||
index = 0
|
||||
for line in file_lines:
|
||||
if UPDATEABLE_LINE_COMPILED.search(line) is not None:
|
||||
return index, line
|
||||
index = index + 1
|
||||
return None, None
|
||||
|
||||
def parse_year_range(year_range):
|
||||
year_split = year_range.split('-')
|
||||
start_year = year_split[0]
|
||||
if len(year_split) == 1:
|
||||
return start_year, start_year
|
||||
return start_year, year_split[1]
|
||||
|
||||
def year_range_to_str(start_year, end_year):
|
||||
if start_year == end_year:
|
||||
return start_year
|
||||
return "%s-%s" % (start_year, end_year)
|
||||
|
||||
def create_updated_copyright_line(line, last_git_change_year):
|
||||
copyright_splitter = 'Copyright (c) '
|
||||
copyright_split = line.split(copyright_splitter)
|
||||
# Preserve characters on line that are ahead of the start of the copyright
|
||||
# notice - they are part of the comment block and vary from file-to-file.
|
||||
before_copyright = copyright_split[0]
|
||||
after_copyright = copyright_split[1]
|
||||
|
||||
space_split = after_copyright.split(' ')
|
||||
year_range = space_split[0]
|
||||
start_year, end_year = parse_year_range(year_range)
|
||||
if end_year == last_git_change_year:
|
||||
return line
|
||||
return (before_copyright + copyright_splitter +
|
||||
year_range_to_str(start_year, last_git_change_year) + ' ' +
|
||||
' '.join(space_split[1:]))
|
||||
|
||||
def update_updatable_copyright(filename):
|
||||
file_lines = read_file_lines(filename)
|
||||
index, line = get_updatable_copyright_line(file_lines)
|
||||
if not line:
|
||||
print_file_action_message(filename, "No updatable copyright.")
|
||||
return
|
||||
last_git_change_year = get_most_recent_git_change_year(filename)
|
||||
new_line = create_updated_copyright_line(line, last_git_change_year)
|
||||
if line == new_line:
|
||||
print_file_action_message(filename, "Copyright up-to-date.")
|
||||
return
|
||||
file_lines[index] = new_line
|
||||
write_file_lines(filename, file_lines)
|
||||
print_file_action_message(filename,
|
||||
"Copyright updated! -> %s" % last_git_change_year)
|
||||
|
||||
def exec_update_header_year(base_directory):
|
||||
for filename in get_filenames_to_examine(base_directory):
|
||||
update_updatable_copyright(filename)
|
||||
|
||||
################################################################################
|
||||
# update cmd
|
||||
################################################################################
|
||||
|
||||
UPDATE_USAGE = """
|
||||
Updates all the copyright headers of "The PIVX developers" which were
|
||||
changed in a year more recent than is listed. For example:
|
||||
|
||||
// Copyright (c) <firstYear>-<lastYear> The PIVX developers
|
||||
|
||||
will be updated to:
|
||||
|
||||
// Copyright (c) <firstYear>-<lastModifiedYear> The PIVX developers
|
||||
|
||||
where <lastModifiedYear> is obtained from the 'git log' history.
|
||||
|
||||
This subcommand also handles copyright headers that have only a single year. In those cases:
|
||||
|
||||
// Copyright (c) <year> The PIVX developers
|
||||
|
||||
will be updated to:
|
||||
|
||||
// Copyright (c) <year>-<lastModifiedYear> The PIVX developers
|
||||
|
||||
where the update is appropriate.
|
||||
|
||||
Usage:
|
||||
$ ./copyright_header.py update <base_directory>
|
||||
|
||||
Arguments:
|
||||
<base_directory> - The base directory of a agrarian source code repository.
|
||||
"""
|
||||
|
||||
def print_file_action_message(filename, action):
|
||||
print("%-52s %s" % (filename, action))
|
||||
|
||||
def update_cmd(argv):
|
||||
if len(argv) != 3:
|
||||
sys.exit(UPDATE_USAGE)
|
||||
|
||||
base_directory = argv[2]
|
||||
if not os.path.exists(base_directory):
|
||||
sys.exit("*** bad base_directory: %s" % base_directory)
|
||||
exec_update_header_year(base_directory)
|
||||
|
||||
################################################################################
|
||||
# inserted copyright header format
|
||||
################################################################################
|
||||
|
||||
def get_header_lines(header, start_year, end_year):
|
||||
lines = header.split('\n')[1:-1]
|
||||
lines[0] = lines[0] % year_range_to_str(start_year, end_year)
|
||||
return [line + '\n' for line in lines]
|
||||
|
||||
CPP_HEADER = '''
|
||||
// Copyright (c) %s The PIVX developers
|
||||
// Distributed under the MIT software license, see the accompanying
|
||||
// file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
'''
|
||||
|
||||
def get_cpp_header_lines_to_insert(start_year, end_year):
|
||||
return reversed(get_header_lines(CPP_HEADER, start_year, end_year))
|
||||
|
||||
PYTHON_HEADER = '''
|
||||
# Copyright (c) %s The PIVX developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
'''
|
||||
|
||||
def get_python_header_lines_to_insert(start_year, end_year):
|
||||
return reversed(get_header_lines(PYTHON_HEADER, start_year, end_year))
|
||||
|
||||
################################################################################
|
||||
# query git for year of last change
|
||||
################################################################################
|
||||
|
||||
def get_git_change_year_range(filename):
|
||||
years = get_git_change_years(filename)
|
||||
return min(years), max(years)
|
||||
|
||||
################################################################################
|
||||
# check for existing core copyright
|
||||
################################################################################
|
||||
|
||||
def file_already_has_core_copyright(file_lines):
|
||||
index, _ = get_updatable_copyright_line(file_lines)
|
||||
return index is not None
|
||||
|
||||
################################################################################
|
||||
# insert header execution
|
||||
################################################################################
|
||||
|
||||
def file_has_hashbang(file_lines):
|
||||
if len(file_lines) < 1:
|
||||
return False
|
||||
if len(file_lines[0]) <= 2:
|
||||
return False
|
||||
return file_lines[0][:2] == '#!'
|
||||
|
||||
def insert_python_header(filename, file_lines, start_year, end_year):
|
||||
if file_has_hashbang(file_lines):
|
||||
insert_idx = 1
|
||||
else:
|
||||
insert_idx = 0
|
||||
header_lines = get_python_header_lines_to_insert(start_year, end_year)
|
||||
for line in header_lines:
|
||||
file_lines.insert(insert_idx, line)
|
||||
write_file_lines(filename, file_lines)
|
||||
|
||||
def insert_cpp_header(filename, file_lines, start_year, end_year):
|
||||
header_lines = get_cpp_header_lines_to_insert(start_year, end_year)
|
||||
for line in header_lines:
|
||||
file_lines.insert(0, line)
|
||||
write_file_lines(filename, file_lines)
|
||||
|
||||
def exec_insert_header(filename, style):
|
||||
file_lines = read_file_lines(filename)
|
||||
if file_already_has_core_copyright(file_lines):
|
||||
sys.exit('*** %s already has a copyright by The PIVX developers'
|
||||
% (filename))
|
||||
start_year, end_year = get_git_change_year_range(filename)
|
||||
if style == 'python':
|
||||
insert_python_header(filename, file_lines, start_year, end_year)
|
||||
else:
|
||||
insert_cpp_header(filename, file_lines, start_year, end_year)
|
||||
|
||||
################################################################################
|
||||
# insert cmd
|
||||
################################################################################
|
||||
|
||||
INSERT_USAGE = """
|
||||
Inserts a copyright header for "The PIVX developers" at the top of the
|
||||
file in either Python or C++ style as determined by the file extension. If the
|
||||
file is a Python file and it has a '#!' starting the first line, the header is
|
||||
inserted in the line below it.
|
||||
|
||||
The copyright dates will be set to be:
|
||||
|
||||
"<year_introduced>-<current_year>"
|
||||
|
||||
where <year_introduced> is according to the 'git log' history. If
|
||||
<year_introduced> is equal to <current_year>, the date will be set to be:
|
||||
|
||||
"<current_year>"
|
||||
|
||||
If the file already has a copyright for "The PIVX developers", the
|
||||
script will exit.
|
||||
|
||||
Usage:
|
||||
$ ./copyright_header.py insert <file>
|
||||
|
||||
Arguments:
|
||||
<file> - A source file in the bitcoin repository.
|
||||
"""
|
||||
|
||||
def insert_cmd(argv):
|
||||
if len(argv) != 3:
|
||||
sys.exit(INSERT_USAGE)
|
||||
|
||||
filename = argv[2]
|
||||
if not os.path.isfile(filename):
|
||||
sys.exit("*** bad filename: %s" % filename)
|
||||
_, extension = os.path.splitext(filename)
|
||||
if extension not in ['.h', '.cpp', '.cc', '.c', '.py']:
|
||||
sys.exit("*** cannot insert for file extension %s" % extension)
|
||||
|
||||
if extension == '.py':
|
||||
style = 'python'
|
||||
else:
|
||||
style = 'cpp'
|
||||
exec_insert_header(filename, style)
|
||||
|
||||
################################################################################
|
||||
# UI
|
||||
################################################################################
|
||||
|
||||
USAGE = """
|
||||
copyright_header.py - utilities for managing copyright headers of 'The Bitcoin
|
||||
Core developers' in repository source files.
|
||||
|
||||
Usage:
|
||||
$ ./copyright_header <subcommand>
|
||||
|
||||
Subcommands:
|
||||
report
|
||||
update
|
||||
insert
|
||||
|
||||
To see subcommand usage, run them without arguments.
|
||||
"""
|
||||
|
||||
SUBCOMMANDS = ['report', 'update', 'insert']
|
||||
|
||||
if __name__ == "__main__":
|
||||
if len(sys.argv) == 1:
|
||||
sys.exit(USAGE)
|
||||
subcommand = sys.argv[1]
|
||||
if subcommand not in SUBCOMMANDS:
|
||||
sys.exit(USAGE)
|
||||
if subcommand == 'report':
|
||||
report_cmd(sys.argv)
|
||||
elif subcommand == 'update':
|
||||
update_cmd(sys.argv)
|
||||
elif subcommand == 'insert':
|
||||
insert_cmd(sys.argv)
|
||||
@@ -0,0 +1,32 @@
|
||||
#!/usr/bin/env bash
|
||||
|
||||
export LC_ALL=C
|
||||
TOPDIR=${TOPDIR:-$(git rev-parse --show-toplevel)}
|
||||
BUILDDIR=${BUILDDIR:-$TOPDIR}
|
||||
|
||||
BINDIR=${BINDIR:-$BUILDDIR/src}
|
||||
MANDIR=${MANDIR:-$TOPDIR/doc/man}
|
||||
|
||||
BITCOIND=${BITCOIND:-$BINDIR/agrariand}
|
||||
BITCOINCLI=${BITCOINCLI:-$BINDIR/agrarian-cli}
|
||||
BITCOINTX=${BITCOINTX:-$BINDIR/agrarian-tx}
|
||||
BITCOINQT=${BITCOINQT:-$BINDIR/qt/agrarian-qt}
|
||||
|
||||
[ ! -x $BITCOIND ] && echo "$BITCOIND not found or not executable." && exit 1
|
||||
|
||||
# The autodetected version git tag can screw up manpage output a little bit
|
||||
BTCVER=($($BITCOINCLI --version | head -n1 | awk -F'[ -]' '{ print $6, $7 }'))
|
||||
|
||||
# Create a footer file with copyright content.
|
||||
# This gets autodetected fine for bitcoind if --version-string is not set,
|
||||
# but has different outcomes for bitcoin-qt and bitcoin-cli.
|
||||
echo "[COPYRIGHT]" > footer.h2m
|
||||
$BITCOIND --version | sed -n '1!p' >> footer.h2m
|
||||
|
||||
for cmd in $BITCOIND $BITCOINCLI $BITCOINTX $BITCOINQT; do
|
||||
cmdname="${cmd##*/}"
|
||||
help2man -N --version-string=${BTCVER[0]} --include=footer.h2m -o ${MANDIR}/${cmdname}.1 ${cmd}
|
||||
sed -i "s/\\\-${BTCVER[1]}//g" ${MANDIR}/${cmdname}.1
|
||||
done
|
||||
|
||||
rm -f footer.h2m
|
||||
@@ -0,0 +1,95 @@
|
||||
#!/bin/sh
|
||||
# Copyright (c) 2015 The Bitcoin Core developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
|
||||
export LC_ALL=C
|
||||
DIR="$1"
|
||||
COMMIT="$2"
|
||||
if [ -z "$COMMIT" ]; then
|
||||
COMMIT=HEAD
|
||||
fi
|
||||
|
||||
# Taken from git-subtree (Copyright (C) 2009 Avery Pennarun <apenwarr@gmail.com>)
|
||||
find_latest_squash()
|
||||
{
|
||||
dir="$1"
|
||||
sq=
|
||||
main=
|
||||
sub=
|
||||
git log --grep="^git-subtree-dir: $dir/*\$" \
|
||||
--pretty=format:'START %H%n%s%n%n%b%nEND%n' "$COMMIT" |
|
||||
while read a b _; do
|
||||
case "$a" in
|
||||
START) sq="$b" ;;
|
||||
git-subtree-mainline:) main="$b" ;;
|
||||
git-subtree-split:) sub="$b" ;;
|
||||
END)
|
||||
if [ -n "$sub" ]; then
|
||||
if [ -n "$main" ]; then
|
||||
# a rejoin commit?
|
||||
# Pretend its sub was a squash.
|
||||
sq="$sub"
|
||||
fi
|
||||
echo "$sq" "$sub"
|
||||
break
|
||||
fi
|
||||
sq=
|
||||
main=
|
||||
sub=
|
||||
;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
# find latest subtree update
|
||||
latest_squash="$(find_latest_squash "$DIR")"
|
||||
if [ -z "$latest_squash" ]; then
|
||||
echo "ERROR: $DIR is not a subtree" >&2
|
||||
exit 2
|
||||
fi
|
||||
set $latest_squash
|
||||
old=$1
|
||||
rev=$2
|
||||
|
||||
# get the tree in the current commit
|
||||
tree_actual=$(git ls-tree -d "$COMMIT" "$DIR" | head -n 1)
|
||||
if [ -z "$tree_actual" ]; then
|
||||
echo "FAIL: subtree directory $DIR not found in $COMMIT" >&2
|
||||
exit 1
|
||||
fi
|
||||
set $tree_actual
|
||||
tree_actual_type=$2
|
||||
tree_actual_tree=$3
|
||||
echo "$DIR in $COMMIT currently refers to $tree_actual_type $tree_actual_tree"
|
||||
if [ "d$tree_actual_type" != "dtree" ]; then
|
||||
echo "FAIL: subtree directory $DIR is not a tree in $COMMIT" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# get the tree at the time of the last subtree update
|
||||
tree_commit=$(git show -s --format="%T" $old)
|
||||
echo "$DIR in $COMMIT was last updated in commit $old (tree $tree_commit)"
|
||||
|
||||
# ... and compare the actual tree with it
|
||||
if [ "$tree_actual_tree" != "$tree_commit" ]; then
|
||||
git diff $tree_commit $tree_actual_tree >&2
|
||||
echo "FAIL: subtree directory was touched without subtree merge" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# get the tree in the subtree commit referred to
|
||||
if [ "d$(git cat-file -t $rev 2>/dev/null)" != dcommit ]; then
|
||||
echo "subtree commit $rev unavailable: cannot compare" >&2
|
||||
exit
|
||||
fi
|
||||
tree_subtree=$(git show -s --format="%T" $rev)
|
||||
echo "$DIR in $COMMIT was last updated to upstream commit $rev (tree $tree_subtree)"
|
||||
|
||||
# ... and compare the actual tree with it
|
||||
if [ "$tree_actual_tree" != "$tree_subtree" ]; then
|
||||
echo "FAIL: subtree update commit differs from upstream tree!" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "GOOD"
|
||||
@@ -0,0 +1,384 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2016-2017 Bitcoin Core Developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
|
||||
# This script will locally construct a merge commit for a pull request on a
|
||||
# github repository, inspect it, sign it and optionally push it.
|
||||
|
||||
# The following temporary branches are created/overwritten and deleted:
|
||||
# * pull/$PULL/base (the current master we're merging onto)
|
||||
# * pull/$PULL/head (the current state of the remote pull request)
|
||||
# * pull/$PULL/merge (github's merge)
|
||||
# * pull/$PULL/local-merge (our merge)
|
||||
|
||||
# In case of a clean merge that is accepted by the user, the local branch with
|
||||
# name $BRANCH is overwritten with the merged result, and optionally pushed.
|
||||
import os
|
||||
from sys import stdin,stdout,stderr
|
||||
import argparse
|
||||
import hashlib
|
||||
import subprocess
|
||||
import sys
|
||||
import json
|
||||
import codecs
|
||||
from urllib.request import Request, urlopen
|
||||
from urllib.error import HTTPError
|
||||
|
||||
# External tools (can be overridden using environment)
|
||||
GIT = os.getenv('GIT','git')
|
||||
BASH = os.getenv('BASH','bash')
|
||||
|
||||
# OS specific configuration for terminal attributes
|
||||
ATTR_RESET = ''
|
||||
ATTR_PR = ''
|
||||
COMMIT_FORMAT = '%h %s (%an)%d'
|
||||
if os.name == 'posix': # if posix, assume we can use basic terminal escapes
|
||||
ATTR_RESET = '\033[0m'
|
||||
ATTR_PR = '\033[1;36m'
|
||||
COMMIT_FORMAT = '%C(bold blue)%h%Creset %s %C(cyan)(%an)%Creset%C(green)%d%Creset'
|
||||
|
||||
def git_config_get(option, default=None):
|
||||
'''
|
||||
Get named configuration option from git repository.
|
||||
'''
|
||||
try:
|
||||
return subprocess.check_output([GIT,'config','--get',option]).rstrip().decode('utf-8')
|
||||
except subprocess.CalledProcessError:
|
||||
return default
|
||||
|
||||
def get_response(req_url, ghtoken):
|
||||
req = Request(req_url)
|
||||
if ghtoken is not None:
|
||||
req.add_header('Authorization', 'token ' + ghtoken)
|
||||
return urlopen(req)
|
||||
|
||||
def retrieve_json(req_url, ghtoken, use_pagination=False):
|
||||
'''
|
||||
Retrieve json from github.
|
||||
Return None if an error happens.
|
||||
'''
|
||||
try:
|
||||
reader = codecs.getreader('utf-8')
|
||||
if not use_pagination:
|
||||
return json.load(reader(get_response(req_url, ghtoken)))
|
||||
|
||||
obj = []
|
||||
page_num = 1
|
||||
while True:
|
||||
req_url_page = '{}?page={}'.format(req_url, page_num)
|
||||
result = get_response(req_url_page, ghtoken)
|
||||
obj.extend(json.load(reader(result)))
|
||||
|
||||
link = result.headers.get('link', None)
|
||||
if link is not None:
|
||||
link_next = [l for l in link.split(',') if 'rel="next"' in l]
|
||||
if len(link_next) > 0:
|
||||
page_num = int(link_next[0][link_next[0].find("page=")+5:link_next[0].find(">")])
|
||||
continue
|
||||
break
|
||||
return obj
|
||||
except HTTPError as e:
|
||||
error_message = e.read()
|
||||
print('Warning: unable to retrieve pull information from github: %s' % e)
|
||||
print('Detailed error: %s' % error_message)
|
||||
return None
|
||||
except Exception as e:
|
||||
print('Warning: unable to retrieve pull information from github: %s' % e)
|
||||
return None
|
||||
|
||||
def retrieve_pr_info(repo,pull,ghtoken):
|
||||
req_url = "https://api.github.com/repos/"+repo+"/pulls/"+pull
|
||||
return retrieve_json(req_url,ghtoken)
|
||||
|
||||
def retrieve_pr_comments(repo,pull,ghtoken):
|
||||
req_url = "https://api.github.com/repos/"+repo+"/issues/"+pull+"/comments"
|
||||
return retrieve_json(req_url,ghtoken,use_pagination=True)
|
||||
|
||||
def retrieve_pr_reviews(repo,pull,ghtoken):
|
||||
req_url = "https://api.github.com/repos/"+repo+"/pulls/"+pull+"/reviews"
|
||||
return retrieve_json(req_url,ghtoken,use_pagination=True)
|
||||
|
||||
def ask_prompt(text):
|
||||
print(text,end=" ",file=stderr)
|
||||
stderr.flush()
|
||||
reply = stdin.readline().rstrip()
|
||||
print("",file=stderr)
|
||||
return reply
|
||||
|
||||
def get_symlink_files():
|
||||
files = sorted(subprocess.check_output([GIT, 'ls-tree', '--full-tree', '-r', 'HEAD']).splitlines())
|
||||
ret = []
|
||||
for f in files:
|
||||
if (int(f.decode('utf-8').split(" ")[0], 8) & 0o170000) == 0o120000:
|
||||
ret.append(f.decode('utf-8').split("\t")[1])
|
||||
return ret
|
||||
|
||||
def tree_sha512sum(commit='HEAD'):
|
||||
# request metadata for entire tree, recursively
|
||||
files = []
|
||||
blob_by_name = {}
|
||||
for line in subprocess.check_output([GIT, 'ls-tree', '--full-tree', '-r', commit]).splitlines():
|
||||
name_sep = line.index(b'\t')
|
||||
metadata = line[:name_sep].split() # perms, 'blob', blobid
|
||||
assert(metadata[1] == b'blob')
|
||||
name = line[name_sep+1:]
|
||||
files.append(name)
|
||||
blob_by_name[name] = metadata[2]
|
||||
|
||||
files.sort()
|
||||
# open connection to git-cat-file in batch mode to request data for all blobs
|
||||
# this is much faster than launching it per file
|
||||
p = subprocess.Popen([GIT, 'cat-file', '--batch'], stdout=subprocess.PIPE, stdin=subprocess.PIPE)
|
||||
overall = hashlib.sha512()
|
||||
for f in files:
|
||||
blob = blob_by_name[f]
|
||||
# request blob
|
||||
p.stdin.write(blob + b'\n')
|
||||
p.stdin.flush()
|
||||
# read header: blob, "blob", size
|
||||
reply = p.stdout.readline().split()
|
||||
assert(reply[0] == blob and reply[1] == b'blob')
|
||||
size = int(reply[2])
|
||||
# hash the blob data
|
||||
intern = hashlib.sha512()
|
||||
ptr = 0
|
||||
while ptr < size:
|
||||
bs = min(65536, size - ptr)
|
||||
piece = p.stdout.read(bs)
|
||||
if len(piece) == bs:
|
||||
intern.update(piece)
|
||||
else:
|
||||
raise IOError('Premature EOF reading git cat-file output')
|
||||
ptr += bs
|
||||
dig = intern.hexdigest()
|
||||
assert(p.stdout.read(1) == b'\n') # ignore LF that follows blob data
|
||||
# update overall hash with file hash
|
||||
overall.update(dig.encode("utf-8"))
|
||||
overall.update(" ".encode("utf-8"))
|
||||
overall.update(f)
|
||||
overall.update("\n".encode("utf-8"))
|
||||
p.stdin.close()
|
||||
if p.wait():
|
||||
raise IOError('Non-zero return value executing git cat-file')
|
||||
return overall.hexdigest()
|
||||
|
||||
def get_acks_from_comments(head_commit, comments):
|
||||
assert len(head_commit) == 6
|
||||
ack_str ='\n\nACKs for commit {}:\n'.format(head_commit)
|
||||
for c in comments:
|
||||
review = [l for l in c['body'].split('\r\n') if 'ACK' in l and head_commit in l]
|
||||
if review:
|
||||
ack_str += ' {}:\n'.format(c['user']['login'])
|
||||
ack_str += ' {}\n'.format(review[0])
|
||||
return ack_str
|
||||
|
||||
def print_merge_details(pull, title, branch, base_branch, head_branch):
|
||||
print('%s#%s%s %s %sinto %s%s' % (ATTR_RESET+ATTR_PR,pull,ATTR_RESET,title,ATTR_RESET+ATTR_PR,branch,ATTR_RESET))
|
||||
subprocess.check_call([GIT,'log','--graph','--topo-order','--pretty=format:'+COMMIT_FORMAT,base_branch+'..'+head_branch])
|
||||
|
||||
def parse_arguments():
|
||||
epilog = '''
|
||||
In addition, you can set the following git configuration variables:
|
||||
githubmerge.repository (mandatory),
|
||||
user.signingkey (mandatory),
|
||||
user.ghtoken (default: none).
|
||||
githubmerge.host (default: git@github.com),
|
||||
githubmerge.branch (no default),
|
||||
githubmerge.testcmd (default: none).
|
||||
'''
|
||||
parser = argparse.ArgumentParser(description='Utility to merge, sign and push github pull requests',
|
||||
epilog=epilog)
|
||||
parser.add_argument('pull', metavar='PULL', type=int, nargs=1,
|
||||
help='Pull request ID to merge')
|
||||
parser.add_argument('branch', metavar='BRANCH', type=str, nargs='?',
|
||||
default=None, help='Branch to merge against (default: githubmerge.branch setting, or base branch for pull, or \'master\')')
|
||||
return parser.parse_args()
|
||||
|
||||
def main():
|
||||
# Extract settings from git repo
|
||||
repo = git_config_get('githubmerge.repository')
|
||||
host = git_config_get('githubmerge.host','git@github.com')
|
||||
opt_branch = git_config_get('githubmerge.branch',None)
|
||||
testcmd = git_config_get('githubmerge.testcmd')
|
||||
ghtoken = git_config_get('user.ghtoken')
|
||||
signingkey = git_config_get('user.signingkey')
|
||||
if repo is None:
|
||||
print("ERROR: No repository configured. Use this command to set:", file=stderr)
|
||||
print("git config githubmerge.repository <owner>/<repo>", file=stderr)
|
||||
sys.exit(1)
|
||||
if signingkey is None:
|
||||
print("ERROR: No GPG signing key set. Set one using:",file=stderr)
|
||||
print("git config --global user.signingkey <key>",file=stderr)
|
||||
sys.exit(1)
|
||||
|
||||
if host.startswith(('https:','http:')):
|
||||
host_repo = host+"/"+repo+".git"
|
||||
else:
|
||||
host_repo = host+":"+repo
|
||||
|
||||
# Extract settings from command line
|
||||
args = parse_arguments()
|
||||
pull = str(args.pull[0])
|
||||
|
||||
# Receive pull information from github
|
||||
info = retrieve_pr_info(repo,pull,ghtoken)
|
||||
if info is None:
|
||||
sys.exit(1)
|
||||
comments = retrieve_pr_comments(repo,pull,ghtoken) + retrieve_pr_reviews(repo,pull,ghtoken)
|
||||
if comments is None:
|
||||
sys.exit(1)
|
||||
title = info['title'].strip()
|
||||
body = info['body'].strip()
|
||||
# precedence order for destination branch argument:
|
||||
# - command line argument
|
||||
# - githubmerge.branch setting
|
||||
# - base branch for pull (as retrieved from github)
|
||||
# - 'master'
|
||||
branch = args.branch or opt_branch or info['base']['ref'] or 'master'
|
||||
|
||||
# Initialize source branches
|
||||
head_branch = 'pull/'+pull+'/head'
|
||||
base_branch = 'pull/'+pull+'/base'
|
||||
merge_branch = 'pull/'+pull+'/merge'
|
||||
local_merge_branch = 'pull/'+pull+'/local-merge'
|
||||
|
||||
devnull = open(os.devnull, 'w', encoding="utf8")
|
||||
try:
|
||||
subprocess.check_call([GIT,'checkout','-q',branch])
|
||||
except subprocess.CalledProcessError:
|
||||
print("ERROR: Cannot check out branch %s." % (branch), file=stderr)
|
||||
sys.exit(3)
|
||||
try:
|
||||
subprocess.check_call([GIT,'fetch','-q',host_repo,'+refs/pull/'+pull+'/*:refs/heads/pull/'+pull+'/*',
|
||||
'+refs/heads/'+branch+':refs/heads/'+base_branch])
|
||||
except subprocess.CalledProcessError:
|
||||
print("ERROR: Cannot find pull request #%s or branch %s on %s." % (pull,branch,host_repo), file=stderr)
|
||||
sys.exit(3)
|
||||
try:
|
||||
subprocess.check_call([GIT,'log','-q','-1','refs/heads/'+head_branch], stdout=devnull, stderr=stdout)
|
||||
except subprocess.CalledProcessError:
|
||||
print("ERROR: Cannot find head of pull request #%s on %s." % (pull,host_repo), file=stderr)
|
||||
sys.exit(3)
|
||||
try:
|
||||
subprocess.check_call([GIT,'log','-q','-1','refs/heads/'+merge_branch], stdout=devnull, stderr=stdout)
|
||||
except subprocess.CalledProcessError:
|
||||
print("ERROR: Cannot find merge of pull request #%s on %s." % (pull,host_repo), file=stderr)
|
||||
sys.exit(3)
|
||||
subprocess.check_call([GIT,'checkout','-q',base_branch])
|
||||
subprocess.call([GIT,'branch','-q','-D',local_merge_branch], stderr=devnull)
|
||||
subprocess.check_call([GIT,'checkout','-q','-b',local_merge_branch])
|
||||
|
||||
try:
|
||||
# Go up to the repository's root.
|
||||
toplevel = subprocess.check_output([GIT,'rev-parse','--show-toplevel']).strip()
|
||||
os.chdir(toplevel)
|
||||
# Create unsigned merge commit.
|
||||
if title:
|
||||
firstline = 'Merge #%s: %s' % (pull,title)
|
||||
else:
|
||||
firstline = 'Merge #%s' % (pull,)
|
||||
message = firstline + '\n\n'
|
||||
message += subprocess.check_output([GIT,'log','--no-merges','--topo-order','--pretty=format:%h %s (%an)',base_branch+'..'+head_branch]).decode('utf-8')
|
||||
message += '\n\nPull request description:\n\n ' + body.replace('\n', '\n ') + '\n'
|
||||
message += get_acks_from_comments(head_commit=subprocess.check_output([GIT,'log','-1','--pretty=format:%H',head_branch]).decode('utf-8')[:6], comments=comments)
|
||||
try:
|
||||
subprocess.check_call([GIT,'merge','-q','--commit','--no-edit','--no-ff','--no-gpg-sign','-m',message.encode('utf-8'),head_branch])
|
||||
except subprocess.CalledProcessError:
|
||||
print("ERROR: Cannot be merged cleanly.",file=stderr)
|
||||
subprocess.check_call([GIT,'merge','--abort'])
|
||||
sys.exit(4)
|
||||
logmsg = subprocess.check_output([GIT,'log','--pretty=format:%s','-n','1']).decode('utf-8')
|
||||
if logmsg.rstrip() != firstline.rstrip():
|
||||
print("ERROR: Creating merge failed (already merged?).",file=stderr)
|
||||
sys.exit(4)
|
||||
|
||||
symlink_files = get_symlink_files()
|
||||
for f in symlink_files:
|
||||
print("ERROR: File %s was a symlink" % f)
|
||||
if len(symlink_files) > 0:
|
||||
sys.exit(4)
|
||||
|
||||
# Put tree SHA512 into the message
|
||||
try:
|
||||
first_sha512 = tree_sha512sum()
|
||||
message += '\n\nTree-SHA512: ' + first_sha512
|
||||
except subprocess.CalledProcessError:
|
||||
print("ERROR: Unable to compute tree hash")
|
||||
sys.exit(4)
|
||||
try:
|
||||
subprocess.check_call([GIT,'commit','--amend','--no-gpg-sign','-m',message.encode('utf-8')])
|
||||
except subprocess.CalledProcessError:
|
||||
print("ERROR: Cannot update message.", file=stderr)
|
||||
sys.exit(4)
|
||||
|
||||
print_merge_details(pull, title, branch, base_branch, head_branch)
|
||||
print()
|
||||
|
||||
# Run test command if configured.
|
||||
if testcmd:
|
||||
if subprocess.call(testcmd,shell=True):
|
||||
print("ERROR: Running %s failed." % testcmd,file=stderr)
|
||||
sys.exit(5)
|
||||
|
||||
# Show the created merge.
|
||||
diff = subprocess.check_output([GIT,'diff',merge_branch+'..'+local_merge_branch])
|
||||
subprocess.check_call([GIT,'diff',base_branch+'..'+local_merge_branch])
|
||||
if diff:
|
||||
print("WARNING: merge differs from github!",file=stderr)
|
||||
reply = ask_prompt("Type 'ignore' to continue.")
|
||||
if reply.lower() == 'ignore':
|
||||
print("Difference with github ignored.",file=stderr)
|
||||
else:
|
||||
sys.exit(6)
|
||||
else:
|
||||
# Verify the result manually.
|
||||
print("Dropping you on a shell so you can try building/testing the merged source.",file=stderr)
|
||||
print("Run 'git diff HEAD~' to show the changes being merged.",file=stderr)
|
||||
print("Type 'exit' when done.",file=stderr)
|
||||
if os.path.isfile('/etc/debian_version'): # Show pull number on Debian default prompt
|
||||
os.putenv('debian_chroot',pull)
|
||||
subprocess.call([BASH,'-i'])
|
||||
|
||||
second_sha512 = tree_sha512sum()
|
||||
if first_sha512 != second_sha512:
|
||||
print("ERROR: Tree hash changed unexpectedly",file=stderr)
|
||||
sys.exit(8)
|
||||
|
||||
# Sign the merge commit.
|
||||
print_merge_details(pull, title, branch, base_branch, head_branch)
|
||||
while True:
|
||||
reply = ask_prompt("Type 's' to sign off on the above merge, or 'x' to reject and exit.").lower()
|
||||
if reply == 's':
|
||||
try:
|
||||
subprocess.check_call([GIT,'commit','-q','--gpg-sign','--amend','--no-edit'])
|
||||
break
|
||||
except subprocess.CalledProcessError:
|
||||
print("Error while signing, asking again.",file=stderr)
|
||||
elif reply == 'x':
|
||||
print("Not signing off on merge, exiting.",file=stderr)
|
||||
sys.exit(1)
|
||||
|
||||
# Put the result in branch.
|
||||
subprocess.check_call([GIT,'checkout','-q',branch])
|
||||
subprocess.check_call([GIT,'reset','-q','--hard',local_merge_branch])
|
||||
finally:
|
||||
# Clean up temporary branches.
|
||||
subprocess.call([GIT,'checkout','-q',branch])
|
||||
subprocess.call([GIT,'branch','-q','-D',head_branch],stderr=devnull)
|
||||
subprocess.call([GIT,'branch','-q','-D',base_branch],stderr=devnull)
|
||||
subprocess.call([GIT,'branch','-q','-D',merge_branch],stderr=devnull)
|
||||
subprocess.call([GIT,'branch','-q','-D',local_merge_branch],stderr=devnull)
|
||||
|
||||
# Push the result.
|
||||
while True:
|
||||
reply = ask_prompt("Type 'push' to push the result to %s, branch %s, or 'x' to exit without pushing." % (host_repo,branch)).lower()
|
||||
if reply == 'push':
|
||||
subprocess.check_call([GIT,'push',host_repo,'refs/heads/'+branch])
|
||||
break
|
||||
elif reply == 'x':
|
||||
sys.exit(1)
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
|
||||
@@ -0,0 +1,113 @@
|
||||
#!/usr/bin/env bash
|
||||
#
|
||||
# Copyright (c) 2017 The Bitcoin Core developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
#
|
||||
# Check for new lines in diff that introduce trailing whitespace.
|
||||
|
||||
# We can't run this check unless we know the commit range for the PR.
|
||||
|
||||
export LC_ALL=C
|
||||
while getopts "?" opt; do
|
||||
case $opt in
|
||||
?)
|
||||
echo "Usage: .lint-whitespace.sh [N]"
|
||||
echo " TRAVIS_COMMIT_RANGE='<commit range>' .lint-whitespace.sh"
|
||||
echo " .lint-whitespace.sh -?"
|
||||
echo "Checks unstaged changes, the previous N commits, or a commit range."
|
||||
echo "TRAVIS_COMMIT_RANGE='47ba2c3...ee50c9e' .lint-whitespace.sh"
|
||||
exit 0
|
||||
;;
|
||||
esac
|
||||
done
|
||||
|
||||
if [ -z "${TRAVIS_COMMIT_RANGE}" ]; then
|
||||
if [ "$1" ]; then
|
||||
TRAVIS_COMMIT_RANGE="HEAD~$1...HEAD"
|
||||
else
|
||||
TRAVIS_COMMIT_RANGE="HEAD"
|
||||
fi
|
||||
fi
|
||||
|
||||
showdiff() {
|
||||
if ! git diff -U0 "${TRAVIS_COMMIT_RANGE}" -- "." ":(exclude)depends/patches/" ":(exclude)src/leveldb/" ":(exclude)src/secp256k1/" ":(exclude)src/univalue/" ":(exclude)doc/release-notes/"; then
|
||||
echo "Failed to get a diff"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
showcodediff() {
|
||||
if ! git diff -U0 "${TRAVIS_COMMIT_RANGE}" -- *.cpp *.h *.md *.py *.sh ":(exclude)src/leveldb/" ":(exclude)src/secp256k1/" ":(exclude)src/univalue/" ":(exclude)doc/release-notes/"; then
|
||||
echo "Failed to get a diff"
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
RET=0
|
||||
|
||||
# Check if trailing whitespace was found in the diff.
|
||||
if showdiff | grep -E -q '^\+.*\s+$'; then
|
||||
echo "This diff appears to have added new lines with trailing whitespace."
|
||||
echo "The following changes were suspected:"
|
||||
FILENAME=""
|
||||
SEEN=0
|
||||
SEENLN=0
|
||||
while read -r line; do
|
||||
if [[ "$line" =~ ^diff ]]; then
|
||||
FILENAME="$line"
|
||||
SEEN=0
|
||||
elif [[ "$line" =~ ^@@ ]]; then
|
||||
LINENUMBER="$line"
|
||||
SEENLN=0
|
||||
else
|
||||
if [ "$SEEN" -eq 0 ]; then
|
||||
# The first time a file is seen with trailing whitespace, we print the
|
||||
# filename (preceded by a newline).
|
||||
echo
|
||||
echo "$FILENAME"
|
||||
SEEN=1
|
||||
fi
|
||||
if [ "$SEENLN" -eq 0 ]; then
|
||||
echo "$LINENUMBER"
|
||||
SEENLN=1
|
||||
fi
|
||||
echo "$line"
|
||||
fi
|
||||
done < <(showdiff | grep -E '^(diff --git |@@|\+.*\s+$)')
|
||||
RET=1
|
||||
fi
|
||||
|
||||
# Check if tab characters were found in the diff.
|
||||
if showcodediff | perl -nle '$MATCH++ if m{^\+.*\t}; END{exit 1 unless $MATCH>0}' > /dev/null; then
|
||||
echo "This diff appears to have added new lines with tab characters instead of spaces."
|
||||
echo "The following changes were suspected:"
|
||||
FILENAME=""
|
||||
SEEN=0
|
||||
SEENLN=0
|
||||
while read -r line; do
|
||||
if [[ "$line" =~ ^diff ]]; then
|
||||
FILENAME="$line"
|
||||
SEEN=0
|
||||
elif [[ "$line" =~ ^@@ ]]; then
|
||||
LINENUMBER="$line"
|
||||
SEENLN=0
|
||||
else
|
||||
if [ "$SEEN" -eq 0 ]; then
|
||||
# The first time a file is seen with a tab character, we print the
|
||||
# filename (preceded by a newline).
|
||||
echo
|
||||
echo "$FILENAME"
|
||||
SEEN=1
|
||||
fi
|
||||
if [ "$SEENLN" -eq 0 ]; then
|
||||
echo "$LINENUMBER"
|
||||
SEENLN=1
|
||||
fi
|
||||
echo "$line"
|
||||
fi
|
||||
done < <(showcodediff | perl -nle 'print if m{^(diff --git |@@|\+.*\t)}')
|
||||
RET=1
|
||||
fi
|
||||
|
||||
exit $RET
|
||||
@@ -0,0 +1,102 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2017-2018 The PIVX developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
|
||||
import os, sys
|
||||
from subprocess import check_output
|
||||
|
||||
def countRelevantCommas(line):
|
||||
openParensPosStack = []
|
||||
openParensPos = 0
|
||||
charCounter = 0
|
||||
numRelevantCommas = 0
|
||||
firstOpenParensIndex = line.find("(")
|
||||
|
||||
for char in line:
|
||||
if char == '(':
|
||||
openParensPosStack.append(charCounter)
|
||||
|
||||
if char == ')':
|
||||
openParensPosStack.pop()
|
||||
|
||||
if char == "," and openParensPosStack[-1] == firstOpenParensIndex:
|
||||
numRelevantCommas += 1
|
||||
charCounter += 1
|
||||
|
||||
return numRelevantCommas
|
||||
|
||||
if __name__ == "__main__":
|
||||
out = check_output("git rev-parse --show-toplevel", shell=True, universal_newlines=True)
|
||||
srcDir = out.rstrip() + "/src/"
|
||||
|
||||
filelist = [os.path.join(dp, f) for dp, dn, filenames in os.walk(srcDir) for f in filenames if os.path.splitext(f)[1] == '.cpp' or os.path.splitext(f)[1] == '.h' ]
|
||||
incorrectInstanceCounter = 0
|
||||
|
||||
for file in filelist:
|
||||
f = open(file,"r", encoding="utf-8")
|
||||
data = f.read()
|
||||
rows = data.split("\n")
|
||||
count = 0
|
||||
full_data = []
|
||||
lineCounter = 1
|
||||
|
||||
tempLine = ""
|
||||
tempCount = 0
|
||||
|
||||
for row in rows:
|
||||
# Collapse multiple lines into one
|
||||
tempLine += row
|
||||
|
||||
# Line contains LogPrint or LogPrintf
|
||||
if tempLine.find("LogPrint") != -1:
|
||||
if tempLine.count("(") == tempLine.count(")"):
|
||||
havePercents = tempLine.count('%') > 0
|
||||
|
||||
if havePercents:
|
||||
# This line of code has a format specifier that requires checking number of associated arguments
|
||||
# Determine the number of arguments provided, see if that matches the number of format specifiers
|
||||
# Count the number of commas after the format specifier string. Check to see if it matches the number of format specifiers.
|
||||
# Assumes quotes are not escaped in the specifier string and there are no percent signs when specifying the debug level.
|
||||
|
||||
# First, determine the position of the comma after the format specifier section, named commaAfterEndSpecifierStringIndex
|
||||
firstSpecifierIndex = tempLine.find('%')
|
||||
startSpecifierStringIndex = tempLine.rfind('"',firstSpecifierIndex)
|
||||
endSpecifierStringIndex = tempLine.find('"',firstSpecifierIndex)
|
||||
commaAfterEndSpecifierStringIndex = tempLine.find(',',endSpecifierStringIndex)
|
||||
|
||||
# Count the number of commas after the specifier string
|
||||
line = "(" + tempLine[commaAfterEndSpecifierStringIndex:-1]
|
||||
numCommas = countRelevantCommas(line)
|
||||
|
||||
# Determine number of extra percents after specifier string
|
||||
numExtraPercents = tempLine.count('%', commaAfterEndSpecifierStringIndex)
|
||||
|
||||
# Subtract extra from total count. This is the number of expected specifiers
|
||||
# ignore %%
|
||||
numPercents = tempLine.count('%') - numExtraPercents - 2*tempLine.count('%%')
|
||||
|
||||
if numPercents != numCommas:
|
||||
print("Incorrect number of arguments for LogPrint(f) statement found.")
|
||||
print(str(file) + ":" + str(lineCounter - tempCount))
|
||||
print("Line = " + tempLine)
|
||||
print("numRelevantCommas = " + str(numCommas) + ", numRelevantPercents = " + str(numPercents))
|
||||
print("")
|
||||
|
||||
incorrectInstanceCounter += 1
|
||||
|
||||
# Done with this multiline, clear tempLine
|
||||
tempLine = ""
|
||||
tempCount = 0
|
||||
else:
|
||||
tempCount += 1
|
||||
else:
|
||||
# No LogPrint, clear tempLine
|
||||
tempLine = ""
|
||||
tempCount = 0
|
||||
|
||||
lineCounter += 1
|
||||
|
||||
print("# of incorrect instances: " + str(incorrectInstanceCounter))
|
||||
|
||||
sys.exit(incorrectInstanceCounter)
|
||||
@@ -0,0 +1,76 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2014-2018 The Bitcoin Core developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
'''
|
||||
Run this script every time you change one of the png files. Using pngcrush, it will optimize the png files, remove various color profiles, remove ancillary chunks (alla) and text chunks (text).
|
||||
#pngcrush -brute -ow -rem gAMA -rem cHRM -rem iCCP -rem sRGB -rem alla -rem text
|
||||
'''
|
||||
import os
|
||||
import sys
|
||||
import subprocess
|
||||
import hashlib
|
||||
from PIL import Image # pip3 install Pillow
|
||||
|
||||
def file_hash(filename):
|
||||
'''Return hash of raw file contents'''
|
||||
with open(filename, 'rb') as f:
|
||||
return hashlib.sha256(f.read()).hexdigest()
|
||||
|
||||
def content_hash(filename):
|
||||
'''Return hash of RGBA contents of image'''
|
||||
i = Image.open(filename)
|
||||
i = i.convert('RGBA')
|
||||
data = i.tobytes()
|
||||
return hashlib.sha256(data).hexdigest()
|
||||
|
||||
pngcrush = 'pngcrush'
|
||||
git = 'git'
|
||||
folders = ["src/qt/res/movies", "src/qt/res/icons", "share/pixmaps"]
|
||||
basePath = subprocess.check_output([git, 'rev-parse', '--show-toplevel'], universal_newlines=True, encoding='utf8').rstrip('\n')
|
||||
totalSaveBytes = 0
|
||||
noHashChange = True
|
||||
|
||||
outputArray = []
|
||||
for folder in folders:
|
||||
absFolder=os.path.join(basePath, folder)
|
||||
for file in os.listdir(absFolder):
|
||||
extension = os.path.splitext(file)[1]
|
||||
if extension.lower() == '.png':
|
||||
print("optimizing {}...".format(file), end =' ')
|
||||
file_path = os.path.join(absFolder, file)
|
||||
fileMetaMap = {'file' : file, 'osize': os.path.getsize(file_path), 'sha256Old' : file_hash(file_path)}
|
||||
fileMetaMap['contentHashPre'] = content_hash(file_path)
|
||||
|
||||
try:
|
||||
subprocess.call([pngcrush, "-brute", "-ow", "-rem", "gAMA", "-rem", "cHRM", "-rem", "iCCP", "-rem", "sRGB", "-rem", "alla", "-rem", "text", file_path],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
except:
|
||||
print("pngcrush is not installed, aborting...")
|
||||
sys.exit(0)
|
||||
|
||||
#verify
|
||||
if "Not a PNG file" in subprocess.check_output([pngcrush, "-n", "-v", file_path], stderr=subprocess.STDOUT, universal_newlines=True, encoding='utf8'):
|
||||
print("PNG file "+file+" is corrupted after crushing, check out pngcursh version")
|
||||
sys.exit(1)
|
||||
|
||||
fileMetaMap['sha256New'] = file_hash(file_path)
|
||||
fileMetaMap['contentHashPost'] = content_hash(file_path)
|
||||
|
||||
if fileMetaMap['contentHashPre'] != fileMetaMap['contentHashPost']:
|
||||
print("Image contents of PNG file {} before and after crushing don't match".format(file))
|
||||
sys.exit(1)
|
||||
|
||||
fileMetaMap['psize'] = os.path.getsize(file_path)
|
||||
outputArray.append(fileMetaMap)
|
||||
print("done")
|
||||
|
||||
print("summary:\n+++++++++++++++++")
|
||||
for fileDict in outputArray:
|
||||
oldHash = fileDict['sha256Old']
|
||||
newHash = fileDict['sha256New']
|
||||
totalSaveBytes += fileDict['osize'] - fileDict['psize']
|
||||
noHashChange = noHashChange and (oldHash == newHash)
|
||||
print(fileDict['file']+"\n size diff from: "+str(fileDict['osize'])+" to: "+str(fileDict['psize'])+"\n old sha256: "+oldHash+"\n new sha256: "+newHash+"\n")
|
||||
|
||||
print("completed. Checksum stable: "+str(noHashChange)+". Total reduction: "+str(totalSaveBytes)+" bytes")
|
||||
@@ -0,0 +1,215 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2015-2018 The Bitcoin Core developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
'''
|
||||
Perform basic ELF security checks on a series of executables.
|
||||
Exit status will be 0 if successful, and the program will be silent.
|
||||
Otherwise the exit status will be 1 and it will log which executables failed which checks.
|
||||
Needs `readelf` (for ELF) and `objdump` (for PE).
|
||||
'''
|
||||
import subprocess
|
||||
import sys
|
||||
import os
|
||||
|
||||
READELF_CMD = os.getenv('READELF', '/usr/bin/readelf')
|
||||
OBJDUMP_CMD = os.getenv('OBJDUMP', '/usr/bin/objdump')
|
||||
NONFATAL = {} # checks which are non-fatal for now but only generate a warning
|
||||
|
||||
def check_ELF_PIE(executable):
|
||||
'''
|
||||
Check for position independent executable (PIE), allowing for address space randomization.
|
||||
'''
|
||||
p = subprocess.Popen([READELF_CMD, '-h', '-W', executable], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE, universal_newlines=True)
|
||||
(stdout, stderr) = p.communicate()
|
||||
if p.returncode:
|
||||
raise IOError('Error opening file')
|
||||
|
||||
ok = False
|
||||
for line in stdout.splitlines():
|
||||
line = line.split()
|
||||
if len(line)>=2 and line[0] == 'Type:' and line[1] == 'DYN':
|
||||
ok = True
|
||||
return ok
|
||||
|
||||
def get_ELF_program_headers(executable):
|
||||
'''Return type and flags for ELF program headers'''
|
||||
p = subprocess.Popen([READELF_CMD, '-l', '-W', executable], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE, universal_newlines=True)
|
||||
(stdout, stderr) = p.communicate()
|
||||
if p.returncode:
|
||||
raise IOError('Error opening file')
|
||||
in_headers = False
|
||||
count = 0
|
||||
headers = []
|
||||
for line in stdout.splitlines():
|
||||
if line.startswith('Program Headers:'):
|
||||
in_headers = True
|
||||
if line == '':
|
||||
in_headers = False
|
||||
if in_headers:
|
||||
if count == 1: # header line
|
||||
ofs_typ = line.find('Type')
|
||||
ofs_offset = line.find('Offset')
|
||||
ofs_flags = line.find('Flg')
|
||||
ofs_align = line.find('Align')
|
||||
if ofs_typ == -1 or ofs_offset == -1 or ofs_flags == -1 or ofs_align == -1:
|
||||
raise ValueError('Cannot parse elfread -lW output')
|
||||
elif count > 1:
|
||||
typ = line[ofs_typ:ofs_offset].rstrip()
|
||||
flags = line[ofs_flags:ofs_align].rstrip()
|
||||
headers.append((typ, flags))
|
||||
count += 1
|
||||
return headers
|
||||
|
||||
def check_ELF_NX(executable):
|
||||
'''
|
||||
Check that no sections are writable and executable (including the stack)
|
||||
'''
|
||||
have_wx = False
|
||||
have_gnu_stack = False
|
||||
for (typ, flags) in get_ELF_program_headers(executable):
|
||||
if typ == 'GNU_STACK':
|
||||
have_gnu_stack = True
|
||||
if 'W' in flags and 'E' in flags: # section is both writable and executable
|
||||
have_wx = True
|
||||
return have_gnu_stack and not have_wx
|
||||
|
||||
def check_ELF_RELRO(executable):
|
||||
'''
|
||||
Check for read-only relocations.
|
||||
GNU_RELRO program header must exist
|
||||
Dynamic section must have BIND_NOW flag
|
||||
'''
|
||||
have_gnu_relro = False
|
||||
for (typ, flags) in get_ELF_program_headers(executable):
|
||||
# Note: not checking flags == 'R': here as linkers set the permission differently
|
||||
# This does not affect security: the permission flags of the GNU_RELRO program header are ignored, the PT_LOAD header determines the effective permissions.
|
||||
# However, the dynamic linker need to write to this area so these are RW.
|
||||
# Glibc itself takes care of mprotecting this area R after relocations are finished.
|
||||
# See also https://marc.info/?l=binutils&m=1498883354122353
|
||||
if typ == 'GNU_RELRO':
|
||||
have_gnu_relro = True
|
||||
|
||||
have_bindnow = False
|
||||
p = subprocess.Popen([READELF_CMD, '-d', '-W', executable], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE, universal_newlines=True)
|
||||
(stdout, stderr) = p.communicate()
|
||||
if p.returncode:
|
||||
raise IOError('Error opening file')
|
||||
for line in stdout.splitlines():
|
||||
tokens = line.split()
|
||||
if len(tokens)>1 and tokens[1] == '(BIND_NOW)' or (len(tokens)>2 and tokens[1] == '(FLAGS)' and 'BIND_NOW' in tokens[2:]):
|
||||
have_bindnow = True
|
||||
return have_gnu_relro and have_bindnow
|
||||
|
||||
def check_ELF_Canary(executable):
|
||||
'''
|
||||
Check for use of stack canary
|
||||
'''
|
||||
p = subprocess.Popen([READELF_CMD, '--dyn-syms', '-W', executable], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE, universal_newlines=True)
|
||||
(stdout, stderr) = p.communicate()
|
||||
if p.returncode:
|
||||
raise IOError('Error opening file')
|
||||
ok = False
|
||||
for line in stdout.splitlines():
|
||||
if '__stack_chk_fail' in line:
|
||||
ok = True
|
||||
return ok
|
||||
|
||||
def get_PE_dll_characteristics(executable):
|
||||
'''
|
||||
Get PE DllCharacteristics bits.
|
||||
Returns a tuple (arch,bits) where arch is 'i386:x86-64' or 'i386'
|
||||
and bits is the DllCharacteristics value.
|
||||
'''
|
||||
p = subprocess.Popen([OBJDUMP_CMD, '-x', executable], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE, universal_newlines=True)
|
||||
(stdout, stderr) = p.communicate()
|
||||
if p.returncode:
|
||||
raise IOError('Error opening file')
|
||||
arch = ''
|
||||
bits = 0
|
||||
for line in stdout.splitlines():
|
||||
tokens = line.split()
|
||||
if len(tokens)>=2 and tokens[0] == 'architecture:':
|
||||
arch = tokens[1].rstrip(',')
|
||||
if len(tokens)>=2 and tokens[0] == 'DllCharacteristics':
|
||||
bits = int(tokens[1],16)
|
||||
return (arch,bits)
|
||||
|
||||
IMAGE_DLL_CHARACTERISTICS_HIGH_ENTROPY_VA = 0x0020
|
||||
IMAGE_DLL_CHARACTERISTICS_DYNAMIC_BASE = 0x0040
|
||||
IMAGE_DLL_CHARACTERISTICS_NX_COMPAT = 0x0100
|
||||
|
||||
def check_PE_DYNAMIC_BASE(executable):
|
||||
'''PIE: DllCharacteristics bit 0x40 signifies dynamicbase (ASLR)'''
|
||||
(arch,bits) = get_PE_dll_characteristics(executable)
|
||||
reqbits = IMAGE_DLL_CHARACTERISTICS_DYNAMIC_BASE
|
||||
return (bits & reqbits) == reqbits
|
||||
|
||||
# On 64 bit, must support high-entropy 64-bit address space layout randomization in addition to DYNAMIC_BASE
|
||||
# to have secure ASLR.
|
||||
def check_PE_HIGH_ENTROPY_VA(executable):
|
||||
'''PIE: DllCharacteristics bit 0x20 signifies high-entropy ASLR'''
|
||||
(arch,bits) = get_PE_dll_characteristics(executable)
|
||||
if arch == 'i386:x86-64':
|
||||
reqbits = IMAGE_DLL_CHARACTERISTICS_HIGH_ENTROPY_VA
|
||||
else: # Unnecessary on 32-bit
|
||||
assert(arch == 'i386')
|
||||
reqbits = 0
|
||||
return (bits & reqbits) == reqbits
|
||||
|
||||
def check_PE_NX(executable):
|
||||
'''NX: DllCharacteristics bit 0x100 signifies nxcompat (DEP)'''
|
||||
(arch,bits) = get_PE_dll_characteristics(executable)
|
||||
return (bits & IMAGE_DLL_CHARACTERISTICS_NX_COMPAT) == IMAGE_DLL_CHARACTERISTICS_NX_COMPAT
|
||||
|
||||
CHECKS = {
|
||||
'ELF': [
|
||||
('PIE', check_ELF_PIE),
|
||||
('NX', check_ELF_NX),
|
||||
('RELRO', check_ELF_RELRO),
|
||||
('Canary', check_ELF_Canary)
|
||||
],
|
||||
'PE': [
|
||||
('DYNAMIC_BASE', check_PE_DYNAMIC_BASE),
|
||||
('HIGH_ENTROPY_VA', check_PE_HIGH_ENTROPY_VA),
|
||||
('NX', check_PE_NX)
|
||||
]
|
||||
}
|
||||
|
||||
def identify_executable(executable):
|
||||
with open(filename, 'rb') as f:
|
||||
magic = f.read(4)
|
||||
if magic.startswith(b'MZ'):
|
||||
return 'PE'
|
||||
elif magic.startswith(b'\x7fELF'):
|
||||
return 'ELF'
|
||||
return None
|
||||
|
||||
if __name__ == '__main__':
|
||||
retval = 0
|
||||
for filename in sys.argv[1:]:
|
||||
try:
|
||||
etype = identify_executable(filename)
|
||||
if etype is None:
|
||||
print('%s: unknown format' % filename)
|
||||
retval = 1
|
||||
continue
|
||||
|
||||
failed = []
|
||||
warning = []
|
||||
for (name, func) in CHECKS[etype]:
|
||||
if not func(filename):
|
||||
if name in NONFATAL:
|
||||
warning.append(name)
|
||||
else:
|
||||
failed.append(name)
|
||||
if failed:
|
||||
print('%s: failed %s' % (filename, ' '.join(failed)))
|
||||
retval = 1
|
||||
if warning:
|
||||
print('%s: warning %s' % (filename, ' '.join(warning)))
|
||||
except IOError:
|
||||
print('%s: cannot open' % filename)
|
||||
retval = 1
|
||||
sys.exit(retval)
|
||||
|
||||
@@ -0,0 +1,10 @@
|
||||
#!/bin/sh
|
||||
set -e
|
||||
if [ $# -ne 3 ];
|
||||
then echo "usage: $0 <input> <stripped-binary> <debug-binary>"
|
||||
fi
|
||||
|
||||
@OBJCOPY@ --enable-deterministic-archives -p --only-keep-debug $1 $3
|
||||
@OBJCOPY@ --enable-deterministic-archives -p --strip-debug $1 $2
|
||||
@STRIP@ --enable-deterministic-archives -p -s $2
|
||||
@OBJCOPY@ --enable-deterministic-archives -p --add-gnu-debuglink=$3 $2
|
||||
@@ -0,0 +1,177 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2014 Wladimir J. van der Laan
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
'''
|
||||
A script to check that the (Linux) executables produced by gitian only contain
|
||||
allowed gcc, glibc and libstdc++ version symbols. This makes sure they are
|
||||
still compatible with the minimum supported Linux distribution versions.
|
||||
|
||||
Example usage:
|
||||
|
||||
find ../gitian-builder/build -type f -executable | xargs python3 contrib/devtools/symbol-check.py
|
||||
'''
|
||||
import subprocess
|
||||
import re
|
||||
import sys
|
||||
import os
|
||||
|
||||
# Debian 6.0.9 (Squeeze) has:
|
||||
#
|
||||
# - g++ version 4.4.5 (https://packages.debian.org/search?suite=default§ion=all&arch=any&searchon=names&keywords=g%2B%2B)
|
||||
# - libc version 2.11.3 (https://packages.debian.org/search?suite=default§ion=all&arch=any&searchon=names&keywords=libc6)
|
||||
# - libstdc++ version 4.4.5 (https://packages.debian.org/search?suite=default§ion=all&arch=any&searchon=names&keywords=libstdc%2B%2B6)
|
||||
#
|
||||
# Ubuntu 10.04.4 (Lucid Lynx) has:
|
||||
#
|
||||
# - g++ version 4.4.3 (http://packages.ubuntu.com/search?keywords=g%2B%2B&searchon=names&suite=lucid§ion=all)
|
||||
# - libc version 2.11.1 (http://packages.ubuntu.com/search?keywords=libc6&searchon=names&suite=lucid§ion=all)
|
||||
# - libstdc++ version 4.4.3 (http://packages.ubuntu.com/search?suite=lucid§ion=all&arch=any&keywords=libstdc%2B%2B&searchon=names)
|
||||
#
|
||||
# Taking the minimum of these as our target.
|
||||
#
|
||||
# According to GNU ABI document (http://gcc.gnu.org/onlinedocs/libstdc++/manual/abi.html) this corresponds to:
|
||||
# GCC 4.4.0: GCC_4.4.0
|
||||
# GCC 4.4.2: GLIBCXX_3.4.13, CXXABI_1.3.3
|
||||
# (glibc) GLIBC_2_11
|
||||
#
|
||||
MAX_VERSIONS = {
|
||||
'GCC': (4,4,0),
|
||||
'CXXABI': (1,3,3),
|
||||
'GLIBCXX': (3,4,13),
|
||||
'GLIBC': (2,11),
|
||||
'LIBATOMIC': (1,0)
|
||||
}
|
||||
# See here for a description of _IO_stdin_used:
|
||||
# https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=634261#109
|
||||
|
||||
# Ignore symbols that are exported as part of every executable
|
||||
IGNORE_EXPORTS = {
|
||||
'_edata', '_end', '__end__', '_init', '__bss_start', '__bss_start__', '_bss_end__', '__bss_end__', '_fini', '_IO_stdin_used', 'stdin', 'stdout', 'stderr'
|
||||
}
|
||||
READELF_CMD = os.getenv('READELF', '/usr/bin/readelf')
|
||||
CPPFILT_CMD = os.getenv('CPPFILT', '/usr/bin/c++filt')
|
||||
# Allowed NEEDED libraries
|
||||
ALLOWED_LIBRARIES = {
|
||||
# bitcoind and bitcoin-qt
|
||||
'libgcc_s.so.1', # GCC base support
|
||||
'libc.so.6', # C library
|
||||
'libpthread.so.0', # threading
|
||||
'libanl.so.1', # DNS resolve
|
||||
'libm.so.6', # math library
|
||||
'librt.so.1', # real-time (clock)
|
||||
'libatomic.so.1',
|
||||
'ld-linux-x86-64.so.2', # 64-bit dynamic linker
|
||||
'ld-linux.so.2', # 32-bit dynamic linker
|
||||
'ld-linux-aarch64.so.1', # 64-bit ARM dynamic linker
|
||||
'ld-linux-armhf.so.3', # 32-bit ARM dynamic linker
|
||||
'ld-linux-riscv64-lp64d.so.1', # 64-bit RISC-V dynamic linker
|
||||
# bitcoin-qt only
|
||||
'libX11-xcb.so.1', # part of X11
|
||||
'libX11.so.6', # part of X11
|
||||
'libxcb.so.1', # part of X11
|
||||
'libfontconfig.so.1', # font support
|
||||
'libfreetype.so.6', # font parsing
|
||||
'libdl.so.2' # programming interface to dynamic linker
|
||||
}
|
||||
ARCH_MIN_GLIBC_VER = {
|
||||
'80386': (2,1),
|
||||
'X86-64': (2,2,5),
|
||||
'ARM': (2,4),
|
||||
'AArch64':(2,17),
|
||||
'RISC-V': (2,27)
|
||||
}
|
||||
class CPPFilt(object):
|
||||
'''
|
||||
Demangle C++ symbol names.
|
||||
|
||||
Use a pipe to the 'c++filt' command.
|
||||
'''
|
||||
def __init__(self):
|
||||
self.proc = subprocess.Popen(CPPFILT_CMD, stdin=subprocess.PIPE, stdout=subprocess.PIPE, universal_newlines=True)
|
||||
|
||||
def __call__(self, mangled):
|
||||
self.proc.stdin.write(mangled + '\n')
|
||||
self.proc.stdin.flush()
|
||||
return self.proc.stdout.readline().rstrip()
|
||||
|
||||
def close(self):
|
||||
self.proc.stdin.close()
|
||||
self.proc.stdout.close()
|
||||
self.proc.wait()
|
||||
|
||||
def read_symbols(executable, imports=True):
|
||||
'''
|
||||
Parse an ELF executable and return a list of (symbol,version) tuples
|
||||
for dynamic, imported symbols.
|
||||
'''
|
||||
p = subprocess.Popen([READELF_CMD, '--dyn-syms', '-W', '-h', executable], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE, universal_newlines=True)
|
||||
(stdout, stderr) = p.communicate()
|
||||
if p.returncode:
|
||||
raise IOError('Could not read symbols for %s: %s' % (executable, stderr.strip()))
|
||||
syms = []
|
||||
for line in stdout.splitlines():
|
||||
line = line.split()
|
||||
if 'Machine:' in line:
|
||||
arch = line[-1]
|
||||
if len(line)>7 and re.match('[0-9]+:$', line[0]):
|
||||
(sym, _, version) = line[7].partition('@')
|
||||
is_import = line[6] == 'UND'
|
||||
if version.startswith('@'):
|
||||
version = version[1:]
|
||||
if is_import == imports:
|
||||
syms.append((sym, version, arch))
|
||||
return syms
|
||||
|
||||
def check_version(max_versions, version, arch):
|
||||
if '_' in version:
|
||||
(lib, _, ver) = version.rpartition('_')
|
||||
else:
|
||||
lib = version
|
||||
ver = '0'
|
||||
ver = tuple([int(x) for x in ver.split('.')])
|
||||
if not lib in max_versions:
|
||||
return False
|
||||
return ver <= max_versions[lib] or lib == 'GLIBC' and ver <= ARCH_MIN_GLIBC_VER[arch]
|
||||
|
||||
def read_libraries(filename):
|
||||
p = subprocess.Popen([READELF_CMD, '-d', '-W', filename], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE, universal_newlines=True)
|
||||
(stdout, stderr) = p.communicate()
|
||||
if p.returncode:
|
||||
raise IOError('Error opening file')
|
||||
libraries = []
|
||||
for line in stdout.splitlines():
|
||||
tokens = line.split()
|
||||
if len(tokens)>2 and tokens[1] == '(NEEDED)':
|
||||
match = re.match('^Shared library: \[(.*)\]$', ' '.join(tokens[2:]))
|
||||
if match:
|
||||
libraries.append(match.group(1))
|
||||
else:
|
||||
raise ValueError('Unparseable (NEEDED) specification')
|
||||
return libraries
|
||||
|
||||
if __name__ == '__main__':
|
||||
cppfilt = CPPFilt()
|
||||
retval = 0
|
||||
for filename in sys.argv[1:]:
|
||||
# Check imported symbols
|
||||
for sym,version,arch in read_symbols(filename, True):
|
||||
if version and not check_version(MAX_VERSIONS, version, arch):
|
||||
print('%s: symbol %s from unsupported version %s' % (filename, cppfilt(sym), version))
|
||||
retval = 1
|
||||
# Check exported symbols
|
||||
if arch != 'RISC-V':
|
||||
for sym,version,arch in read_symbols(filename, False):
|
||||
if sym in IGNORE_EXPORTS:
|
||||
continue
|
||||
print('%s: export of symbol %s not allowed' % (filename, cppfilt(sym)))
|
||||
retval = 1
|
||||
# Check dependency libraries
|
||||
for library_name in read_libraries(filename):
|
||||
if library_name not in ALLOWED_LIBRARIES:
|
||||
print('%s: NEEDED library %s is not allowed' % (filename, library_name))
|
||||
retval = 1
|
||||
|
||||
sys.exit(retval)
|
||||
|
||||
|
||||
@@ -0,0 +1,71 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2015-2017 The Bitcoin Core developers
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
'''
|
||||
Test script for security-check.py
|
||||
'''
|
||||
import subprocess
|
||||
import unittest
|
||||
|
||||
def write_testcode(filename):
|
||||
with open(filename, 'w', encoding="utf8") as f:
|
||||
f.write('''
|
||||
#include <stdio.h>
|
||||
int main()
|
||||
{
|
||||
printf("the quick brown fox jumps over the lazy god\\n");
|
||||
return 0;
|
||||
}
|
||||
''')
|
||||
|
||||
def call_security_check(cc, source, executable, options):
|
||||
subprocess.check_call([cc,source,'-o',executable] + options)
|
||||
p = subprocess.Popen(['./security-check.py',executable], stdout=subprocess.PIPE, stderr=subprocess.PIPE, stdin=subprocess.PIPE, universal_newlines=True)
|
||||
(stdout, stderr) = p.communicate()
|
||||
return (p.returncode, stdout.rstrip())
|
||||
|
||||
class TestSecurityChecks(unittest.TestCase):
|
||||
def test_ELF(self):
|
||||
source = 'test1.c'
|
||||
executable = 'test1'
|
||||
cc = 'gcc'
|
||||
write_testcode(source)
|
||||
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-zexecstack','-fno-stack-protector','-Wl,-znorelro']),
|
||||
(1, executable+': failed PIE NX RELRO Canary'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-znoexecstack','-fno-stack-protector','-Wl,-znorelro']),
|
||||
(1, executable+': failed PIE RELRO Canary'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-znoexecstack','-fstack-protector-all','-Wl,-znorelro']),
|
||||
(1, executable+': failed PIE RELRO'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-znoexecstack','-fstack-protector-all','-Wl,-znorelro','-pie','-fPIE']),
|
||||
(1, executable+': failed RELRO'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,-znoexecstack','-fstack-protector-all','-Wl,-zrelro','-Wl,-z,now','-pie','-fPIE']),
|
||||
(0, ''))
|
||||
|
||||
def test_32bit_PE(self):
|
||||
source = 'test1.c'
|
||||
executable = 'test1.exe'
|
||||
cc = 'i686-w64-mingw32-gcc'
|
||||
write_testcode(source)
|
||||
|
||||
self.assertEqual(call_security_check(cc, source, executable, []),
|
||||
(1, executable+': failed DYNAMIC_BASE NX'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--nxcompat']),
|
||||
(1, executable+': failed DYNAMIC_BASE'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--nxcompat','-Wl,--dynamicbase']),
|
||||
(0, ''))
|
||||
def test_64bit_PE(self):
|
||||
source = 'test1.c'
|
||||
executable = 'test1.exe'
|
||||
cc = 'x86_64-w64-mingw32-gcc'
|
||||
write_testcode(source)
|
||||
|
||||
self.assertEqual(call_security_check(cc, source, executable, []), (1, executable+': failed DYNAMIC_BASE NX\n'+executable+': warning HIGH_ENTROPY_VA'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--nxcompat']), (1, executable+': failed DYNAMIC_BASE\n'+executable+': warning HIGH_ENTROPY_VA'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--nxcompat','-Wl,--dynamicbase']), (0, executable+': warning HIGH_ENTROPY_VA'))
|
||||
self.assertEqual(call_security_check(cc, source, executable, ['-Wl,--nxcompat','-Wl,--dynamicbase','-Wl,--high-entropy-va']), (0, ''))
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
|
||||
@@ -0,0 +1,210 @@
|
||||
#!/usr/bin/env python3
|
||||
# Copyright (c) 2014 Wladimir J. van der Laan
|
||||
# Distributed under the MIT software license, see the accompanying
|
||||
# file COPYING or http://www.opensource.org/licenses/mit-license.php.
|
||||
'''
|
||||
Run this script from the root of the repository to update all translations from
|
||||
transifex.
|
||||
It will do the following automatically:
|
||||
|
||||
- fetch all translations using the tx tool
|
||||
- post-process them into valid and committable format
|
||||
- remove invalid control characters
|
||||
- remove location tags (makes diffs less noisy)
|
||||
|
||||
TODO:
|
||||
- auto-add new translations to the build system according to the translation process
|
||||
'''
|
||||
import subprocess
|
||||
import re
|
||||
import sys
|
||||
import os
|
||||
import io
|
||||
import xml.etree.ElementTree as ET
|
||||
|
||||
# Name of transifex tool
|
||||
TX = 'tx'
|
||||
# Name of source language file
|
||||
SOURCE_LANG = 'agrarian_en.ts'
|
||||
# Directory with locale files
|
||||
LOCALE_DIR = 'src/qt/locale'
|
||||
# Minimum number of messages for translation to be considered at all
|
||||
MIN_NUM_MESSAGES = 10
|
||||
|
||||
def check_at_repository_root():
|
||||
if not os.path.exists('.git'):
|
||||
print('No .git directory found')
|
||||
print('Execute this script at the root of the repository', file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
def fetch_all_translations():
|
||||
if subprocess.call([TX, 'pull', '-f', '-a']):
|
||||
print('Error while fetching translations', file=sys.stderr)
|
||||
sys.exit(1)
|
||||
|
||||
def find_format_specifiers(s):
|
||||
'''Find all format specifiers in a string.'''
|
||||
pos = 0
|
||||
specifiers = []
|
||||
while True:
|
||||
percent = s.find('%', pos)
|
||||
if percent < 0:
|
||||
break
|
||||
try:
|
||||
specifiers.append(s[percent+1])
|
||||
except:
|
||||
print('Failed to get specifier')
|
||||
pos = percent+2
|
||||
return specifiers
|
||||
|
||||
def split_format_specifiers(specifiers):
|
||||
'''Split format specifiers between numeric (Qt) and others (strprintf)'''
|
||||
numeric = []
|
||||
other = []
|
||||
for s in specifiers:
|
||||
if s in {'1','2','3','4','5','6','7','8','9'}:
|
||||
numeric.append(s)
|
||||
else:
|
||||
other.append(s)
|
||||
|
||||
# If both numeric format specifiers and "others" are used, assume we're dealing
|
||||
# with a Qt-formatted message. In the case of Qt formatting (see https://doc.qt.io/qt-5/qstring.html#arg)
|
||||
# only numeric formats are replaced at all. This means "(percentage: %1%)" is valid, without needing
|
||||
# any kind of escaping that would be necessary for strprintf. Without this, this function
|
||||
# would wrongly detect '%)' as a printf format specifier.
|
||||
if numeric:
|
||||
other = []
|
||||
|
||||
# numeric (Qt) can be present in any order, others (strprintf) must be in specified order
|
||||
return set(numeric),other
|
||||
|
||||
def sanitize_string(s):
|
||||
'''Sanitize string for printing'''
|
||||
return s.replace('\n',' ')
|
||||
|
||||
def check_format_specifiers(source, translation, errors, numerus):
|
||||
source_f = split_format_specifiers(find_format_specifiers(source))
|
||||
# assert that no source messages contain both Qt and strprintf format specifiers
|
||||
# if this fails, go change the source as this is hacky and confusing!
|
||||
assert(not(source_f[0] and source_f[1]))
|
||||
try:
|
||||
translation_f = split_format_specifiers(find_format_specifiers(translation))
|
||||
except IndexError:
|
||||
errors.append("Parse error in translation for '%s': '%s'" % (sanitize_string(source), sanitize_string(translation)))
|
||||
return False
|
||||
else:
|
||||
if source_f != translation_f:
|
||||
if numerus and source_f == (set(), ['n']) and translation_f == (set(), []) and translation.find('%') == -1:
|
||||
# Allow numerus translations to omit %n specifier (usually when it only has one possible value)
|
||||
return True
|
||||
errors.append("Mismatch between '%s' and '%s'" % (sanitize_string(source), sanitize_string(translation)))
|
||||
return False
|
||||
return True
|
||||
|
||||
def all_ts_files(suffix=''):
|
||||
for filename in os.listdir(LOCALE_DIR):
|
||||
# process only language files, and do not process source language
|
||||
if not filename.endswith('.ts'+suffix) or filename == SOURCE_LANG+suffix:
|
||||
continue
|
||||
if suffix: # remove provided suffix
|
||||
filename = filename[0:-len(suffix)]
|
||||
filepath = os.path.join(LOCALE_DIR, filename)
|
||||
yield(filename, filepath)
|
||||
|
||||
FIX_RE = re.compile(b'[\x00-\x09\x0b\x0c\x0e-\x1f]')
|
||||
def remove_invalid_characters(s):
|
||||
'''Remove invalid characters from translation string'''
|
||||
return FIX_RE.sub(b'', s)
|
||||
|
||||
# Override cdata escape function to make our output match Qt's (optional, just for cleaner diffs for
|
||||
# comparison, disable by default)
|
||||
_orig_escape_cdata = None
|
||||
def escape_cdata(text):
|
||||
text = _orig_escape_cdata(text)
|
||||
text = text.replace("'", ''')
|
||||
text = text.replace('"', '"')
|
||||
return text
|
||||
|
||||
def postprocess_translations(reduce_diff_hacks=False):
|
||||
print('Checking and postprocessing...')
|
||||
|
||||
if reduce_diff_hacks:
|
||||
global _orig_escape_cdata
|
||||
_orig_escape_cdata = ET._escape_cdata
|
||||
ET._escape_cdata = escape_cdata
|
||||
|
||||
for (filename,filepath) in all_ts_files():
|
||||
os.rename(filepath, filepath+'.orig')
|
||||
|
||||
have_errors = False
|
||||
for (filename,filepath) in all_ts_files('.orig'):
|
||||
# pre-fixups to cope with transifex output
|
||||
parser = ET.XMLParser(encoding='utf-8') # need to override encoding because 'utf8' is not understood only 'utf-8'
|
||||
with open(filepath + '.orig', 'rb') as f:
|
||||
data = f.read()
|
||||
# remove control characters; this must be done over the entire file otherwise the XML parser will fail
|
||||
data = remove_invalid_characters(data)
|
||||
tree = ET.parse(io.BytesIO(data), parser=parser)
|
||||
|
||||
# iterate over all messages in file
|
||||
root = tree.getroot()
|
||||
for context in root.findall('context'):
|
||||
for message in context.findall('message'):
|
||||
numerus = message.get('numerus') == 'yes'
|
||||
source = message.find('source').text
|
||||
translation_node = message.find('translation')
|
||||
# pick all numerusforms
|
||||
if numerus:
|
||||
translations = [i.text for i in translation_node.findall('numerusform')]
|
||||
else:
|
||||
translations = [translation_node.text]
|
||||
|
||||
for translation in translations:
|
||||
if translation is None:
|
||||
continue
|
||||
errors = []
|
||||
valid = check_format_specifiers(source, translation, errors, numerus)
|
||||
|
||||
for error in errors:
|
||||
print('%s: %s' % (filename, error))
|
||||
|
||||
if not valid: # set type to unfinished and clear string if invalid
|
||||
translation_node.clear()
|
||||
translation_node.set('type', 'unfinished')
|
||||
have_errors = True
|
||||
|
||||
# Remove location tags
|
||||
for location in message.findall('location'):
|
||||
message.remove(location)
|
||||
|
||||
# Remove entire message if it is an unfinished translation
|
||||
if translation_node.get('type') == 'unfinished':
|
||||
context.remove(message)
|
||||
|
||||
# check if document is (virtually) empty, and remove it if so
|
||||
num_messages = 0
|
||||
for context in root.findall('context'):
|
||||
for message in context.findall('message'):
|
||||
num_messages += 1
|
||||
if num_messages < MIN_NUM_MESSAGES:
|
||||
print('Removing %s, as it contains only %i messages' % (filepath, num_messages))
|
||||
continue
|
||||
|
||||
# write fixed-up tree
|
||||
# if diff reduction requested, replace some XML to 'sanitize' to qt formatting
|
||||
if reduce_diff_hacks:
|
||||
out = io.BytesIO()
|
||||
tree.write(out, encoding='utf-8')
|
||||
out = out.getvalue()
|
||||
out = out.replace(b' />', b'/>')
|
||||
with open(filepath, 'wb') as f:
|
||||
f.write(out)
|
||||
else:
|
||||
tree.write(filepath, encoding='utf-8')
|
||||
return have_errors
|
||||
|
||||
if __name__ == '__main__':
|
||||
check_at_repository_root()
|
||||
fetch_all_translations()
|
||||
postprocess_translations()
|
||||
|
||||
Reference in New Issue
Block a user