[cpython] Allison Kaptur - Bytes in the Machine: Inside the CPython interpreter - PyCon 2015 (ref)
[cpython] Lecture 8 - User-defined classes and objects (ref)

2. Utility

[py3clean, clean] How to clean *.pyc, *.pyo and __pycache__

py3clean -v .

[git. gitignore] How to setup the gitignore to untrack the dummy files (view) (blog) (ref)

3. Pylint

[pylint, init] How to let the the pylint focus on your project, .pylintrc

# Content of .pylintrc

[MASTER]

extension-pkg-whitelist=PyQt5

[pylint, test module]

#pylint: disable=line-too-long, attribute-defined-outside-init, missing-module-docstring, missing-class-docstring, missing-function-docstring, invalid-name, unused-argument

[pylint, vscode, plugin] vscode-python-docstring

Vscode: Ctrl + Shift + P => Add Py docstring

[pylint, docstring] What is docstring (ref)

# one liine

def sum(x, y):

"""Returns arg1 value add to arg2 value."""

return a+b

print sum.__doc__

[pylint, disable] How to locally disable pylint message?

Key:

# pylint: disable=attribute-defined-outside-init

Example

class MyClass():

#pylint: disable=attribute-defined-outside-init

def my_method(self):

self.x = 1234

4. Pytest

[pytest] Official site (ref), (doc)
[pytest] MUST READ (ref)
[pytest, tutorial] Good Integration Practices (ref)
[packaging, example] How to write a module based package (ref)(view)(blog)
[pytest, cmd, module] How to pytest a specific tests within a module

Command

# 一定要 test_ 開頭的 py 檔名

pytest test_xxx.py -s # -s, show the stdout

Test Module

# File: test_xxx.py

class TestClass(): # 一定要 Test 起始的類別名稱

def test_method(self): # 一定要 Test起始的類別名稱

assert(1)

def test_fun():

assert(1)

[pytest, cmd, function] How to pytest a specific test function

Command

pytest test_xxx.py::test_function -s # -s, show the stdout

Test Module

# File: test_xxx.py

class TestClass():

def test_method(self):

assert(1)

def test_fun():

assert(1)

[pytest, cmd, method] How to pytest a specific test method

Command

pytest test_xxx.py::TestClass::test_method -s # -s, show the stdout

Test Module

# File: test_xxx.py

class TestClass():

def test_method(self):

assert(1)

def test_fun():

assert(1)

[pytest, module] 如何撰寫 Python 模組 (包含自動測試) (view) (blog)
[pytest, naming rule] How to let your testing module be discovered

Test Module

# File: test_xxx.py # 一定要 test_ 開頭的 python 檔名

class TestClass(): # 一定要 Test 開頭的 class name

def test_method(self): # <------ ok 一定要 test 開頭的 method name

assert(1)

@classmethod

def test_classmethod(cls): # x classmethod 不行!

pass

@staticmethod

def test_staticmethod(): # <----- ok ( test 開頭的 static method name)

assert(1)

def test_fun(): # <-----ok ( test 開頭的 function name)

assert(1)

[pytest, layout]

root

├── src

│ └── mynamespace

│ │

│ └── mypkg

│ ├── __init__.py

│ └── PropertyWidget_App.py

│

├── tests

│ ├── __init__.py

│ └── test_pkg1

│ ├── __init__.py

│ ├── test_PropertyWidget_App.py

│ └── utility.py

└── setup.py

[pytest, setup] Setup

Command

root$ pip install -e .

setup.py

----

from setuptools import setup, find_namespace_packages

setup(

name="mypackage2",

# 因為 native namespace package 沒有 __init__.py, 所以 find_packages

# 要改成 find_namespace_packages:

# 尋找放在 src 底下的 namespace packages

packages=find_namespace_packages('src'),

package_dir={'': 'src'} # root 從目前目錄換成 ./src 目錄下 (ref)

)

[pytest, test] 如何存取放在 test 裡的公用 module, utility.py (view)
[pytest, test] 如何存取目標待測 module (view)
[pytest, vscode] 使用 VSCode 進行 pytest 自動測試與除錯你的自動測試程式 (view)
[pytest, package] How to install your dev package for test (view) (ref)
[pytest, searching] How pytest search the module? (ref)

root

├── src

│ └── mypkg

│ ├── __init__.py

│ └── test_foo.py

│

├── tests

│ ├── __init__.py

│ └── test_pkg1

│ ├── __init__.py

│ └── test_view.py

└── setup.py

如果目錄中有 __init__.py, 那麼 pytest 會一直朔源往上找到一個沒有含有 __init__.py 的目錄名稱, 當作 basedir. 然後根據這條路. 往下建立完整 package name.

例如: module test_view.py 的搜尋 => tests.test_pkg1.test_view

Code

from tests.test_pkg1.test_view import TESTABC

[pytest, config] Where is the config file for pytest

Config File

Location: the root of your repository

Name: pytest.ini

Doc: https://docs.pytest.org/en/latest/reference.html#id97

Command line override

-o

pytest -o console_output_style=classic -o cache_dir=/tmp/mycache

[pytest, qt] The Pytest-QT (ref)

自動測試所有由 test 開頭的 functions

Command

python3 -m pytest test_module.py

Code

from PyQt5.QtWidgets import QMainWindow, QApplication

class t_ViewEditWidget_App(QMainWindow)

def test_gui_auto(qtbot):

app = QApplication(sys.argv) # 注意: 沒有處理 QApplication instance 只能有一個的情況

ex = t_ViewEditWidget_App()

qtbot.add_widget(ex)

ex.__init_UI__()

qtbot.keyClicks(ex.widget_view.comb_edit, '2') # simulate user key '2'

app.exec_()

[pytest, qt] 自動測試點擊輸入 UI 元件 (view) (ref)
[pytest, class] How to group multiple tests in a class (view) (ref) (blog)
[pytest, menu] How to test by menu

Quick

ok, act, msg = self.window.get_menu_action('Load: Google Sheet')

if not ok:

return msg

act.trigger()

Code

@staticmethod

def get_menu_action_static(menu, str_name):

if menu is None:

return 0, None, 'Menu is None.'

for act in menu.actions():

if act.menu():

ok, act, msg = t_ViewWidget.get_menu_action_static(menu, str_name)

if not ok:

return 0, None, msg

if str_name in act.text():

return 1, act, 'ok'

return 0, None, 'Not found'

def test_xxx():

ok, act, msg = self.window.get_menu_action('Load: Google Sheet')

if not ok:

return msg

act.trigger()

[pytest, button, mouse] How to simulate user mouse click (ref)

from PyQt5.QtCore import Qt

qtbot.mouseClick(dict_test['bt_add'], Qt.LeftButton)

[pytest, edit, char] How to simulate user key in character

qtbot.keyClicks(window.widget_target.__line_edit__, '1')

[pytest, key, enter] How to key the Enter

qtbot.keyPress(obj_value, Qt.Key_Enter)

[method, inject] How to inject a method to object (ref)

method_names = [

"keyPress",

"keyClick",

"keyClicks",

"keyEvent",

"keyPress",

"keyRelease",

"keyToAscii",

"mouseClick",

"mouseDClick",

"mouseMove",

"mousePress",

"mouseRelease",

]

for method_name in method_names:

method = create_qtest_proxy_method(method_name)

if method is not None:

setattr(cls, method_name, method)

[method, override] 透過覆蓋的方式, 直接避免了使用者介入. How to dynamically override a instance's method (ref)

def test_override_normal_dlg(self, qtbot, monkeypatch):

lst_item_rd = TestVector.get_item_rd_list()

tree = Model.create_tree(lst_item_rd)

lst_leaf = tree.leaves()

node_1st = lst_leaf[0]

monkeypatch.setattr(PropertyDlg, "ask", classmethod(lambda *args: QDialog.Accepted))

result = PropertyDlg.ask('Test Dlg', tree, node_1st, None)

if result == QDialog.Accepted:

print('ok')

# 把 QApplication::exit() method 內容, 換成 lst_exit_calls.append(1)

def test_override_exit_button(qtbot, monkeypatch):

lst_exit_calls = []

monkeypatch.setattr(QApplication, "exit", lambda: lst_exit_calls.append(1))

button = get_app_exit_button()

qtbot.click(button)

assert lst_exit_calls == [1]

5. Transfer the script to exe

[pyinstaller] Pyinstaller (ref)

# install

pip install pyinstaller

# run

pyinstaller hello.py

Issue: pyinstaller creating EXE RuntimeError: maximum recursion depth exceeded while calling a Python object

<sol>

# in xxx.spec

import sys

sys.setrecursionlimit(5000)

# run

pyinstaller xxx.spec

Reference

By Aviral (ref)

Issue : Cannot find existing PyQt5 plugin directories

<sol>

pip install PyQt5

[python, _cxFreeze] Create a Python Standalone Application using cx_Freeze (view) (blog)

6. Tox

[tox, template] 最簡單的測試模板 (download)
[tox, install] How to install tox (ref)

pip install tox

pip show tox # verification

[tox, tox.ini] A basic setup file to tox (blog)

# content of: tox.ini , put in same dir as setup.py

[tox]

envlist = py36

[testenv]

# install pytest in the virtualenv where commands will be executed

deps =

#pytest # for 一般無 GUI 的 package

pytest-qt # for QT GUI

# pytest-qt 要有 DISPLAY, 否則執行 tox 會立即掛掉 tox: InvocationError (ref)

# 解法:

# 1. 安裝 pytest-xvfb (建議作法, 因為完全沒有 GUI 的環境也可執行)

# 2. 直接在 tox.ini 加入下面這行, 直接把DISPLAY 傳給 pytest-qt (一定要有GUI的環境)

passenv = DISPLAY XAUTHORITY

commands =

# NOTE: you can run any command line tool here - not just tests

pytest

[pytest, override] How to override a method in class (ref) (blog)

from functools import wraps # This convenience func preserves name and docstring

def add_method(cls):

def decorator(func):

@wraps(func)

def wrapper(self, *args, **kwargs):

return func(*args, **kwargs)

setattr(cls, func.__name__, wrapper)

# Note we are not binding func, but wrapper which accepts self but does exactly the same as func

return func # returning func means func can still be used normally

return decorator

class A():

def __foo__(self):

print('original')

@add_method(A)

def __foo__():

print('hello world!')

a = A()

a.__foo__()

7. Environment

[AI] 最簡單的方式安裝 AI 開發環境 (view)
[core] Python official site (ref)
[doc] Language Document (ref)
[install, anaconda] Anaconda (ref), (ref2)
[install, opencv] How to install Anaconda with OpenCV (view)

jupyter notebook --NotebookApp.iopub_data_rate_limit=1.0e10

[install, jupyter, python kernel] How to install the python execution backend (view)
[install, scipy] How to install Scipy (view)
[install, ide, activepython] liClips (ref), ActivePython (ref)
[install, ide, spyder] spyder (ref) (installation) (download)
[ide, code, hotkey] remove whole line (ref)

Ctrl + Shift + K

[ide, code, hotkey] Shift left/right
[conda, install] How to install conda (ref)

wget https://repo.anaconda.com/archive/Anaconda3-5.3.0-Linux-x86_64.sh

bash Anaconda-latest-Linux-x86_64.sh

conda install tensorflow-gpu

[conda, upgrade] How to upgrade conda?

conda update --all

[conda, env, create] How to create an environment

conda create --name venv_demo python=3

[conda, env, clone] How to clone an envirnonment (ref)

conda create --name myclone_new --clone myenv_old

[conda, env, remove] How to remove an environment

conda env remove -n venv_name

[conda, env, rename] How to rename an envorinment (ref)

conda create --name new_name --clone old_name

conda remove --name old_name --all

[conda, env, list] How to list all envoronment

conda info --envs

[conda, env, activate] How to activate an environment

source activate venv_demo

[conda, env, deactivate] How to deactivate an environment (ref)

conda deactivate

[conda, package, install] How to install package using conda

conda install pandas

[vscode-insiders]
[vscode, install] How to install vscode

sudo apt update

sudo apt install software-properties-common apt-transport-https wget

wget -q https://packages.microsoft.com/keys/microsoft.asc -O- | sudo apt-key add -

sudo add-apt-repository "deb [arch=amd64] https://packages.microsoft.com/repos/vscode stable main"

sudo apt update

sudo apt install code

[vscode, uninstall] How to uninstall vscode

sudo apt-get purge code

rm -fr .vscode && rm -fr .config/Code

[vscode, goto] How to launch vscode at specified line

code -g hello_world.py:122

[vscode, package, setting sync] How to sync every setting for your VSCode?

Extension: Settings Sync
Login: [Extensions] -> [Settings Sync] -> Logiin with Github
Download: F1: Sync download & Sync Upload
Auto Setup: Shift + Alt + D

[vscode, upgrade] How to upgrade vscode

wget https://vscode-update.azurewebsites.net/latest/linux-deb-x64/stable -O /tmp/code_latest_amd64.deb

sudo dpkg -i /tmp/code_latest_amd64.deb

[vscode, env] How to debug your python code with vscode in a specified conda environment

Step 1: Launch vscode

Step 2: Click the running environment at the left bottom corner

Step 3: Select the favor environment to debug

[vscode, short, method] How to show class and method list

Ctrl + Shfit +O AND :

[vscode, package] Popular extension for Python

Trailing Spaces
Anaconda extension
Indent-Rainbow
Hightlight-icemode, ice-liu
GitLens, Eric Amodio

[vscode, debug] How to write a vscode launch.json for debug your python code

{

"name": "Python: Current File (Debug Mode)",

"type": "python",

"request": "launch",

"program": "${file}",

"console": "integratedTerminal",

"args": ["-mode", "1", "-ecg", "xxxjson", "-log", "output.log", "-debug", "output.debug", "-o", "result.json"]

}

[vscode, keymap] How to setup the vscode keymap as Visual Studio

Step 1: [Code] -> [Preferences] -> [Key Maps]

Step 2: Install [Visual Studio Key Maps]

[vscode, keymap, sublime]

Remove line: Ctrl + X
Move cursor to left: Ctrl + Left Arrow

[pip, install] How to install the latest version of package

pip install {package-name} -U

[pip, package, list] How to list the out the packages installed in your environment

pip freeze

[nvidia, cuda, install] How to install nvidia cuda package (ref)

Installation Instructions:

`sudo dpkg -i cuda-repo-ubuntu1604-9-2-local_9.2.148-1_amd64.deb`
`sudo apt-key add /var/cuda-repo-<version>/7fa2af80.pub`
`sudo apt-get update`
`sudo apt-get install cuda`

Other installation options are available in the form of meta-packages. For example, to install all the library packages, replace "cuda" with the "cuda-libraries-9-2" meta package. For more information on all the available meta packages click here.

[cython, concept] Cython is a compiler which compiles Python-like code files to C code. Still, ‘’Cython is not a Python to C translator’‘. That is, it doesn’t take your full program and “turns it into C” – rather, the result makes full use of the Python runtime environment. (Ref)
[cython, pyx] cython 可以直接編譯成 binary，用 C 或 python 都可以直接使用
[cython, py] python 可以直接編譯成 binary, 用 python 調用 (用 C 調用?)
[cython, opt] How to use Python 3 syntax to compile the code?

cython -3

[cython] cython 是用來加速關鍵片段，把熱區用 cython 重寫，產生 python module ，讓傳統 python 呼叫，https://en.m.wikipedia.org/wiki/Cython
[cython] How to use Cython to transfer Python to C code (view)
[cython, so] 直接使用編譯出來的 python modle (view)

Q: 如果要被轉換的 module_a.py 裡面有呼叫 module_b.py, 會不會有問題?

不會!

Setup.py 會把 module_a 轉成 .so, 然後再呼叫 module_b.py

Q: 如果目錄中有 .py 和 .so, 系統會選那一個執行? => 選 .so

[cython, c] How to use Cython modoule from C (view)
[cython, pyx] How to invoke pyx code from C (view)
[cython, python] How to invoke python code from C using cython (view)

build.sh

#!/bin/bash

# ************* Build *************

cython *.py

cython *.pyx

src_dir=./

for fullfilename in `ls $src_dir*.c`; do

echo $fullfilename

filename=$(basename -- "$fullfilename")

extension="${filename##*.}"

filename="${filename%.*}"

#echo $filename

#echo $extension

#echo $filename

gcc $filename.c -o $filename.o -shared -pthread -fPIC -fwrapv -O2 -Wall -fno-strict-aliasing -I/usr/include/python3.5

done

gcc main.c -ldl bridge.o -I/usr/include/python3.5 -L/usr/lib/python3.5/config-3.5m-x86_64-linux-gnu -lpython3.5

LD_LIBRARY_PATH=.:$LD_LIBRARY_PATH

PYTHONPATH=.:$PYTHONPATH

export PYTHONPATH

export LD_LIBRARY_PATH

# ************* Run *************

./a.out

[cython, python] 直接在 Cython 中, 混合 Python code (view)
[cython, app] How to create a standalone Python code using cython (view) (ref 使用範例: 包含 msvc project), 不錯的解說 (ref), 但沒提到 numpy 找不到的問題

目前: 如果 python 有 import numpy 就會找不到, The generated C-code always looks up the modules in the original python site-packages dir (ref)

[cython, android] Cython on Android (ref)
[ipython, check] How to know if your code is executed in Jupyter notebook? (ref)

#code

def is_interactive():

import __main__ as main

return not hasattr(main, '__file__')

#usage

print('is_interactive = ', is_interactive())

#result

is_interactive = True

[platform] How to check the running platform such as ios or Android (ref)

if sys.platform == 'linux': # Android or Ubuntu

print('linux')

if sys.platform == 'ios': # ios

print('filter: ios')

# method 2

import platform

platform.system() != 'Linux':

import platform

if global_core.bRelease == True and platform.system() != 'Linux':

...

else:

...

[core] Python official site (link)
[ide, all in one] Anaconda (link)
[ide] liClips (link), ActivePython (link)
[ide] spyder (ref) (installation) (download)
matplotlib (view) (installation) (plotfile_demo)
[iso 8601 date format] csv plot (view)

8. Package

8.1. Conda

[conda, list] List all installed packages

conda list

[conda, install] Install a package

conda install package-name

[conda, remove] remove a package

conda remove package-name

8.2. pip

[package, version] How to get the package version (ref)

print (np.__version__)

[conda, pip] Install pip in conda environment

conda install pip

[pip vs. pip3] (ref)

pip3 always operates on the Python3 environment only, as pip2 does with Python2. pip operates on whichever environment is appropriate to the context. For example if you are in a Python3 venv, pip will operate on the Python3 environment.

[pip, list] How to list packages that pip maintain

pip list

[pip, install] How to install package

pip install package-name

[pip, install, dev] How to install the package in develop mode

pip install -e .

[pip, uninstall] How to uninstall a package

pip uninstall package-name

[pip, show] How to show a package detail information

pip show package-name

[xxxxxx, pip3, upgrade] How to upgrade the pip (ref) xxxxxxx

# Step 1: Upgrade pip

pip3 install --upgrade pip

# Step 2: Fix /usr/bin/pip

-- Original --

from pip import main

if __name__ == '__main__':

sys.exit(main())

-- New --

from pip import __main__

if __name__ == '__main__':

sys.exit(__main__._main())

8.3. Import

[module, init] How to use __init__

Quick

__init__.py 只是標明這個目錄是 python module. 沒有其他用途. 把 module 放進 package folder

要使用, 就是在 import 的前面加上 package folder name 而已

Detail

Step 1: create a package folder

mkdir bak

Step 2: Create a make file

touch __init__.py

Step 3: Usage

-- ./local/LOCAL_CFG.py --

class LOCAL_CFG():

CloudService = {

'LAN': {

'test': 'test'

}

-- end of file --

from local.LOCAL_CFG import LOCAL_CFG

lan = LOCAL_CFG.CloudService['LAN']

lan

Result

{'test': 'test'}

[module, path, get, imort] How to ge the module path name (ref)

from pathlib import Path

import sys

path_cur = Path(sys.modules[__name__].__file__)

sys.path.append(path_cur.parents[0].as_posix() + '/package_loc/')

import os

from os.path import dirname

str_module_path = dirname(os.path.abspath(sys.modules[__name__].__file__))

sys.path.append(str_module_path + '/package_loc/')

from os.path import dirname

str_module_path = dirname(os.path.abspath(sys.modules[__name__].__file__))

PACKAGE_PATH = str_module_path +/package_loc/'

sys.path.extend([PACKAGE_PATH,])

[import] How to use import that scope in class

class MyClass():

import wfdb

def fun(cls, data):

signals, fields = self.wfdb.rdsamp(...)

# usage

obj = MyClass()

obj.fun()

[import] How to get module object by giving a path with file name? (ref)

Key:

getattr(instance, str_method)()

Code:

class FOO:

def foo1(self):

print('foo1')

def foo2(self):

print('foo2')

f_obj = FOO()

getattr(f_obj, 'foo1')()

Result:

foo1

8.4. Packaging to library

[packaging, layout] The regular pakcage sourcelayout (ref)

root/

setup.py

src/

mypkg/

__init__.py

app.py

view.py

tests/

__init__.py

foo/

__init__.py

test_view.py

bar/

__init__.py

test_view.py

[packaging, layout] The native namepsace package source layout (PEP420) (import system)(ref)

root/

setup.py

namespace/

# No __init__.py here

src/

mypkg/

__init__.py

app.py

view.py

tests/

__init__.py

foo/

__init__.py

test_view.py

bar/

__init__.py

test_view.py

setup.py

from setuptools import setup, find_namespace_packages

setup(

name="mypackage2",

# 因為 native namespace package 沒有 __init__.py, 所以 find_packages

# 要改成 find_namespace_packages:

# 尋找放在 src 底下的 namespace packages

packages=find_namespace_packages('src'),

package_dir={'': 'src'} # root 從目前目錄換成 ./src 目錄下 (ref)

)

[packaging, setup.py] packages and package_dir (ref) (view) (blog)

packages: specified your python modules can be found in the packages list.

Ex: packages = ["foo"]:

表示要處理的package 叫做 foo. 系統在 foo 目錄下, 預期有 __init__.py. 也就是 foo/__init__.py

Ex: packags = find_packages("src"):

find_packages 會傳回 src 目錄下的每一個 package name list, 在那些 name 目錄下都有一個 __init__.py

package_dir: to tell the Distutils about your convention.

Ex:
package_dir{'':src} # 說明 package root , 從目前目錄. 換成在目錄 src 下找
packages = {'foo', 'foo.bar'}

意思是 package foo 放在在目錄 src/foo/__init__.py
意思是 package foo.bar: 在目錄 src/foo/bar/__inti__.py

Ex:
package_dir{'foo':'lib'} # 說明 package foo 換成在目錄 lib 下找
packages = {'foo', 'foo.bar'}

意思是 package foo 在目錄 lib/__init__.py
意思是 package foo.bar: 在目錄 lib/bar/__inti__.py

[packaging, install] How to write a module based package (ref)(view)

9. Language

[arg] How to get the argument from command line (view)
[chinese in code] How to add utf-8 character in python (ref)
[comment] How to add comment -- ' ' ' enclosed for multi-line comment OR # (ref)
[comment] comment example

"""Computes the spectrogram of the wave.

seg_length: number of samples in each segment

win_flag: boolean, whether to apply hamming window to each segment

returns: Spectrogram

"""

9.1. Comment

[comment, one line] #
[comment, multi-line] 雙引號 x 3 ex: """

10. Enum

[ennum, create] How to create enumerate type (ref)

from enum import Enum, auto

class Loc(Enum):

Left = auto()

Top = auto()

Right = auto()

Bottom = auto()

# Usage

Loc.Left

10.1. Variable

[var, static] How to define a static varialbe in python (ref)
[var, type] How to determine the type of a variable

type(variable)

[var, naming, _] What is the meaning of a underscope symbol, _ (ref)

For storing the value of last expression in interpreter

[global] How to use global variable in function

Code [ref]

gABC = 0

def setupGlobalVariable():

global gABC # declare the global variable

print('in A:: gABC (before) = ', gABC)

gABC = 123

print('in A:: gABC (setup global new value) = ', gABC)

if __name__ == "__main__":

print('in main:: gABC = ', gABC)

setupGlobalVariable()

print('in main:: gABC (updated) = ', gABC)

Result

in main:: gABC = 0

in A:: gABC (before) = 0

in A:: gABC (setup global new value) = 123

in main:: gABC (updated) = 123

[global, files] How to use global variable in multiple files (ref)

== global_demo.py ==

MY_GLOBAL_DATA = 123

== a.py ==

Import glo

[nan] How to check NaN

if math.isnan(my_data[i]):

my_data[i] = 0

[int, max] What is the maximun value for an integer

# maximum value

import sys

sys.maxsize

# minimun value

-sys.maxsize - 1

[list, add] list add (view)

10.2. Scope

[with] What is with keyword in Python -- 讓 with 下面的 code 可以安全地執行, 無論如何都可以受到 with 後面的 object 控制 (ref)

10.3. Module

[function, class, method] How to list all declared functions, classes, methods for a given package/module?

from pathlib import Path

import inspect

import importlib

def test_main():

path_package = Path('.')

lst_file = list(path_package.glob('*.py'))

for file in lst_file:

print('In module: ', file)

str_module = inspect.getmodulename(file)

if str_module is None:

msg = '[Error] There is no module for giving ' + file

return 0, msg

mod = importlib.import_module(str_module, path_package)

lst_fun_module = inspect.getmembers(mod, predicate=inspect.isfunction)

lst_class = inspect.getmembers(mod, predicate=inspect.isclass)

lst_class = get_class_obj_list(lst_class, '__module__', str_module) # skip the imported classes

print_info('Function List (in module:{})'.format(str_module), lst_fun_module, '')

print_info('Class List (in module:{})'.format(str_module), lst_class, '')

for class_obj_pair in lst_class:

lst_method = inspect.getmembers(class_obj_pair[1], predicate=inspect.ismethod)

print_info('Methods (in class:{})'.format(class_obj_pair[0]), lst_method,'\t')

lst_fun_class = inspect.getmembers(class_obj_pair[1], predicate=inspect.isfunction)

print_info('Function List (in class:{})'.format(class_obj_pair[0]), lst_fun_class,'\t')

return 1, 'ok'

def get_class_obj_list(lst_class, str_atribute, str_filter):

lst_class_new = list()

for class_obj_pair in lst_class:

str_attr = getattr(class_obj_pair[1], str_atribute)

if str_attr != str_filter:

continue

lst_class_new.append(class_obj_pair)

return lst_class_new

def print_info(str_title, lst_data, str_tab_level):

print('{}({}):'.format(str_tab_level+str_title, str(len(lst_data))))

for i, data in enumerate(lst_data):

print(str_tab_level + '\t' + str(i) + '. ', data[1],' ', data[0])

10.4. Class

[class, path] How to get path where the class located

def myfun():

from pathlib import Path

import sys

path_cur = Path(sys.modules[__name__].__file__).parents[0]

print('path_cur = ', path_cur)

[class, instance] How to create a class instance by its name? (ref. 1) (ref. 2)

下面的方法, 可以讓你直接輸入類別的名稱 (字串), 就產生物件. 因此, 你就可以寫一個給所有物件都通用的 function, 把重複的程式碼降到低.

Method 1:

Structure

root

├── src

│ └── mynamespace

│ └── mypkg

│ ├── __init__.py

│ └── foo.py

Target Class

class FOO():

def __init__(self):

print('FOO')

Usage

import importlib

str_model = 'mynamespace.mypkg.foo'

str_class = 'FOO'

module = importlib.import_module(str_module)

class_ = getattr(module, str_class)

window = class_()

Result

FOO

Method 2:

Key:

foo = globals()[str_classname]()

Code:

class FOO():

def __init__(self):

print('FOO')

Usage

str_classname = 'FOO'

foo = globals()[str_classname]()

Result

FOO

注意: 不要使用 eval(str_classname), 因為 cracker 會有機會把 str_classname 換成他們的程式, 無意間, 讓你的程式的控制權, 變成 cracker 的

[import] How to use relative method to import a python module? (view)
[class, type] How to check the instance type?

當你在撰寫一堆衍生類會用到的基礎類 method 時, 若希望在這 method 中的流程中, 希望針對特別的衍生類做特別的事. 例如: 衍生類 A 做 A 流程, 衍生類 B 做 B 流程. 但共同流程又放在基礎類中.

你會需要這個知識. 動態的了解某一個 instance, 它的型態為何, 根據型態, 做不同的事情. 讓你的程式碼保持乾淨, 不留下重複的 code.

Key:

isinstance(instance, type)

Code:

class FOO():

def __init__(self):

Pass

foo = FOO()

print(isinstance(foo, FOO))

Result

True

print(' type = ', type(test_unit))

if not isinstance(test_unit, str):

[class, subclass] How to determine whether a class is subclass?

Method 2

Key:

issubclass(type, base-type)

Code:

class BASE():

pass

class FOO(BASE):

def __init__(self):

Pass

foo = FOO()

print(issubclass(type(foo), BASE)

Result

True

[class, static method] How to declar a static method in class (ref)

class MyClass():

@staticmethod

def test_read(data): # 只能存取 class 中的 static variable, read-only

print('data = ', data)

# usage

MyClass.test_read(1234)

# output

Data = 1234

[class, class method] How to declare a class method (ref)

class MyClass():

@classmethod

def test_read(cls, data):

print('data = ', data) # 可以存取 class 中的 class variable, read/write

# usage

MyClass.test_read(1234)

# output

Data = 1234

[class, class method] How to get the class method name?

Key:

inspect.currentframe().f_code.co_name

Code:

class A():

@classmethod

def test_read(cls):

print('class method name = ', inspect.currentframe().f_code.co_name)

Result

class method name =test_read

[class, class variable] How to declare/access a class variable

class MyClass():

class_variable = 123

def fun(cls, data):

print('MyClass.class_variable = ', MyClass.class_variable)

# usage

obj = MyClass()

obj.fun()

# output

MyClass.class_variable = 123

[class, import] 如何 import 一個內部使用的 package

Hint: 盡量包在 function 內

class MyClass():

#import whdb # <--- don't do this, 放在這裡會展開

def fun(self, data):

import wfdb

self.wfdb = wfdb # use this

def fun2(self):

self.wfdb.read(...) # usage

[constructor, super] How to use super (ref)

class Rectangle:

def __init__(self, length, width):

self.length = length

self.width = width

def area(self):

return self.length * self.width

def perimeter(self):

return 2 * self.length + 2 * self.width

# Here we declare that the Square class inherits from the Rectangle class

class Square(Rectangle):

def __init__(self, length):

super().__init__(length, length)

[class, super] How to invoke super method

class Rectangle:

def __init__(self, length, width):

self.length = length

self.width = width

def area(self):

return self.length * self.width

def perimeter(self):

return 2 * self.length + 2 * self.width

# Here we declare that the Square class inherits from the Rectangle class

class Square(Rectangle):

def __init__(self, length):

super().__init__(length, length)

def fun(self):

super().area()

[class, variable, permission] private, protected, public variable

Privated variabne: __variable__
Protected variable: _variable_
Public variable: variable

[class, name] How to get class name

class t_CloudService():

def __init__(self):

self.lst_class = inspect.getmembers(self, predicate=inspect.isclass)

self.class_name = self.lst_class[0][1]

print('class name = ', self.class_name)

t_CloudService()

Output

class name = <class '__main__.t_CloudService'>

[class, list] How to list classes for giving a module name

Key

str_package = Path('.')

str_module = inspect.getmodulename(file)

mod = importlib.import_module(str_module, str_package)

lst_class = inspect.getmembers(mod, predicate=inspect.isclass)

Test File

#-- test_main.py --

from pathlib import Path

class TestMain():

def __init__(self):

pass

Code

from pathlib import Path

import inspect

import importlib

str_package = Path('.')

str_module = 'test_main.py'

file = Path(str_module)

print('In module: ', file)

str_module = inspect.getmodulename(file)

if str_module is None:

msg = '[Error] There is no module for giving ' + file

return 0, msg

mod = importlib.import_module(str_module, str_package)

lst_class = inspect.getmembers(mod, predicate=inspect.isclass)

print_info('Class List (in module:{})'.format(str_module), lst_class, '')

Util

def print_info(str_title, lst_data, str_tab_level):

print('{}({}):'.format(str_tab_level+str_title, str(len(lst_data))))

for i, data in enumerate(lst_data):

print(str_tab_level + '\t' + str(i) + '. ', data[1],' ', data[0])

Result

Class List (in module:test_main)(2):

0. <class 'pathlib.Path'> Path

1. <class 'test_main.TestMain'> TestMain

[class, method, list] How to get all methods for the class (ref_1) (ref_2)

import inspect

class t_CloudService():

def __init__(self):

self.title = 'Test'

def run(self):

lst_fun = inspect.getmembers(self, predicate=inspect.ismethod)

for tub_fun in lst_fun:

if 'test' in tub_fun[0]:

tub_fun[1]()

return 1, 'ok'

def test_fun1(self):

print('test_fun1')

def test_fun2(self):

print('test_fun2')

@classmethod

def test_cls_fun3(cls):

print('test_cls_fun3')

}

t_CloudService().run()

Output

test_fun1

test_fun2

test_cls_fun3

[class, method, name] How to get the current function name or method name (ref)

import inspect

import types

from typing import cast

class myClass():

def del_data(self):

fun_name = cast(types.FrameType, inspect.currentframe()).f_code.co_name

print ('fun_name = ' , fun_name)

Test

obj = myClass()

obj.del_data()

Output

Fun_name = del_data

[class, method, invoke] How to invoke a method by it's name

Key:

getattr(instance, str_method)()

Code:

class FOO:

def foo1(self):

print('foo1')

def foo2(self):

print('foo2')

f_obj = FOO()

getattr(f_obj, 'foo1')()

[class, to_string] Python to_string method

class MyClass():

def __str__(self):

return 'MyClass'

# Usage

my_obj = MyClass()

print(my_obj)

Output

MyClass

10.5. Time

[time, local] How to get the local time

Code

import time

time.localtime()

Output

time.struct_time(tm_year=2020, tm_mon=4, tm_mday=9, tm_hour=0, tm_min=28, tm_sec=43, tm_wday=3, tm_yday=100, tm_isdst=0)

Code

import time

time.localtime().tm_hour

Output

[time, utc] How to get UTC time (ref)

import datetime

import pytz

# define epoch, the beginning of times in the UTC timestamp world

epoch = datetime.datetime(1970,1,1,0,0,0)

now = datetime.datetime.utcnow()

timestamp = (now - epoch).total_seconds()

>>> 1505329554.617216

[datetime, start] How to get the start daytime

Code

from datetime import datetime, date, time

today_beginning = datetime.combine(date.today(), time())

Result

datetime.datetime(2019, 12, 12, 0, 0)

[time, range] How to get a day range from 00:00:00 ~ 23:59:59

def get_day_range(cls, now):

# return range: 00:00:00 ~ 23:59:59

start = datetime.datetime(now.year, now.month, now.day)

nextday = start + relativedelta.relativedelta(days=1)

end = nextday + relativedelta.relativedelta(seconds=-1)

return start, end

[time, range] How to get a month range from yyyy:mm:day=1,00:00:00 ~ yyyy:mm:"end_of_day", 23:59:59

def get_month_range(now):

# return range: yyyy:mm:day=1,00:00:00 ~ yyyy:mm:"end_of_day", 23:59:59

start = datetime.datetime(now.year, now.month, 1)

nextmonth = start + relativedelta.relativedelta(months=1)

end_of_day = nextmonth + relativedelta.relativedelta(days=-1, hours = 23, minutes=59, seconds=59)

return start, end_of_day

[time, range] How to get year range yyyy:mm=1:day=1,00:00:00 ~ yyyy:"end_of_month":"end_of_day", 23:59:59

def get_year_range( now):

# return range: yyyy:mm=1:day=1,00:00:00 ~ yyyy:"end_of_month":"end_of_day", 23:59:59

start = datetime.datetime(now.year, 1, 1)

nextmonth = start + relativedelta.relativedelta(years=1)

end_of_day = nextmonth + relativedelta.relativedelta(days=-1, hours = 23, minutes=59, seconds=59)

return start, end_of_day

[timezone, list] How to list all timezone

Code

import pytz

pytz.utc

Result

['Africa/Abidjan', 'Africa/Accra', 'Africa/Addis_Ababa', 'Africa/Algiers', 'Africa/Asmara', 'Africa/Asmera', 'Africa/Bamako', 'Africa/Bangui', 'Africa/Banjul', 'Africa/Bissau', 'Africa/Blantyre', 'Africa/Brazzaville', 'Africa/Bujumbura', ....

[timezone, list] How to get the timezone

Mac Pro

# require administrator

$ systemsetup -gettimezone

Time Zone: Europe/Copenhagen

On Debian and Ubuntu systems, you can read /etc/timezone:

Linux

$ cat /etc/timezone

Europe/Oslo

On RedHat and direved systems, you'll need to read it from /etc/sysconfig/clock:

BSD

$ grep ZONE /etc/sysconfig/clock

ZONE="Europe/Oslo"

[QTimeEdit, set] How to set the QTimeEdit (ref)

from PyQt5.QtWidgets import *

from PyQt5.Qt import *

hh = 0; mm = 1

time_edit = QTimeEdit()

time_edit.setTime(QTime(hh, mm))

10.6. String

[string, format] lots of string format (ref)
[string, format] How to show a float value with limited digitial

Format

str_num = '{編號1, 格式} {編號2, 格式}'.format(第一筆浮點數, 第二筆浮點數)

Code

print('st-level = {0:.2f} < {1:.2f} (mv)'.format(1.1123, 1.34212312))

Output

st-level = 1.11 < 1.34 (mv)

Format

str_num = '{編號1} {編號2}'.format(參數 1, 參數 2)

Code

print('st-level = {0} < {1} (mv)'.format("1.1123", "1.34212312"))

Output

st-level = 1.11 < 1.34 (mv)

[string, array] How to create a string array (ref)

arr_aux_note = ['' for x in range(size)]

[string, replace] Search and replace a string in file (view)
[string ,remove] How to remove the leading spaces from a string (ref)

msg.lstrip()

[string ,remove] How to remove the latest spaces from a string (ref)

msg.rstrip()

[string, split] How to split the string by {+ or -} characters (ref)

str_voltage = re.split(r'\+|\-', str_aux)[1]

if '+' in str_line:

str_voltage = '+' + str_voltage

else:

str_voltage = '-' + str_voltage

[string, count] Counting b in a (ref)

def count(a, b, m, n):

if ((m == 0 and n == 0) or n == 0):

return 1

if (m == 0):

return 0

if (a[m - 1] == b[n - 1]):

return (count(a, b, m - 1, n - 1) +

count(a, b, m - 1, n))

else:

return count(a, b, m - 1, n)

Usage:

b = ':'

Str_hh_mm_ss = '1:24:00.924'

c = count(str_hh_mm_ss, b, len(str_hh_mm_ss), len(b))

Output

c = 2

[string, parser, SafeConfigParser] How to parsing ini format configure file using SafeConfigParser (view)
[string, parser, configobj] How to use configobj to parse INI-based configuration file -- for vmx format (view)
[string, parser, extract number] How to extract number from a string (view)
[string, parser, string2num] How to conver string 2 number

Code

import numpy as np

example_string = "1 2 -1 -3"

example_string = example_string.strip() # remove lead space

intList = [int(s) for s in example_string.split(' ')]

print('intList = ', intList)

intArray = np.asarray(intList)

print('intArray = ', intArray)

Result

intList = [1, 2, -1, -3]

intArray = [ 1 2 -1 -3]

[string, inverse] How to inverse a string? (ref)

msg = '1234567'

rev_msg = msg[::-1]

Output

'7654321'

[string, regular expression] How to parse a prefix sting follow a number end such as level_1, level_2, etc... (ref)

Code

import re

lst_level = [key for key in item_rd.keys() if re.search('^level_\d', key)]

print(lst_level)

'''

Where

'^' match the start of the string.
\d match any decimal digital

'''

Output

level_1 level_2

[string, regular expression, num] How to parse the nubmer from a string

Key

regex = r'(\d+)'

Message

import re
re.findall(r'(\d+)', 'Beat-by-beat comparison results for record 28')

Output

['28']

[string, re, 1/2] How to parse the factors from a string

Key

regex =r'(\d+)/(\d+)'

Message

import re

line ='QRS sensitivity: 30% (604/2437)'
re.findall(r'(\d+)/(\d+)', line)

Output

[('604', '2437') ]

[string, regular expression, %] How to parse the percentage number from a string (ref) (demo)

# 最簡單的方法解析出字串中的百分比數字

Key

regex = r'(\d*(\.\d+)?%)'

其中

\d*: 表示小數點左邊 0 個或無限多個數字
\.: 表示小數點 .
\d+: 表示小數點右邊 1 個或無限多個數字
?: 百分比 % 左邊可以有 0 或 1 組數字

Code

import re

msg = ' QRS sensitivity: .23% 24.78% 30% (604/2437)'

regex = r'(\d*(\.\d+)?%)'

m = re.findall(regex, msg)

print(m)

print(m[0][0])

Output

# Find 3 matched strings that are all have two groups

[('.23%', '.23'), ('24.78%', '.78'), ('30%', '')]

'.23%'

[string, regular expression, hh:mm:ss] How to parsing the time format string

>>> line = "Duration sensitivity: - (0.000/0.000)"

>>> re.findall(r'(?:\d+):*(?:\d+):*(?:\d+).\d*', line)

['000/0', '000)']

>>> line = 'Duration sensitivity: - (4.000/10:2.023)'

>>> re.findall(r'(?:\d+):*(?:\d+):*(?:\d+)\.*\d*', line)

['000', '10:2.023']

>>> line = "Duration sensitivity: - (0.000/0.000)"

>>> re.findall(r'(?:\d+):*(?:\d+):*(?:\d+)\.*\d*', line)

['000', '000']

>>> line = "Duration sensitivity: 86% (10:33.436/01:12:19.896)"

>>> re.findall(r'(?:\d+):*(?:\d+):*(?:\d+)\.*\d*', line)

['10:33.436', '01:12:19.896']

>>> line = "Duration sensitivity: 86% (03:10:33.436/01:12:19.896)"

['03:10:33.436', '01:12:19.896']

[string, split] How to split a string with multiple separators (ref)

Key:

\s => space

code

>>> a='Beautiful, is; better*than\nugly'

>>> import re

>>> re.split('; |, |\*|\n',a)

['Beautiful', 'is', 'better', 'than', 'ugly']

[string, parser, awk] How to split a string like awk (view)
[string, parser, visualstudio version] How to extract version number from a string (view)
[string, parser, grep] How to implement grep from python (ref)
[string, parser, sed] How is the sed function in Python (ref)
[string, endwith] How to use end with (ref)

str.endswith(suffix[, start[, end]])

[string, suffix] How to remove the suffix (ref)

msg.replace('.json','')

[string, split] How to split a string
[string, join] How to join the string list to sentance

>>> sentence = ['this','is','a','sentence']

>>> '-'.join(sentence)

'this-is-a-sentence'

[string, format] How to format the output (ref)

#Code

strFormatNum = "{:03d}".format(1) # 1 -> 001

print('strFormatNum = ', strFormatNum)

#Result

strFormatNum = 001

#Code

st_ele = "{:2.2f}".format(st_ele) # 1.232323 -> 1.23

#Code

'{:.2%}'.format(0.9995)

Output

'99.95%'

[string, format] How to format a string with padding and aligning (ref)

{:>10}'.format('test')

Output

[string, contains] How to determine a substring in a string?

if 'abc' in mystring:

continue

[string, decode] How to decode a byte literial ascii string to unicode string? (ref)

Key

str_msg = b'ST,GS,+ 0. 0.0tl.T\r\n\n'.decode('ascii', 'replace')

Code

from subprocess import *

std_out = Popen(['echo', 'ST,GS,+ 0. 0.0tl.T\r\n'], stdout=PIPE).communicate()[0]

print('byte literial = ', std_out)

str_msg = std_out.decode('ascii', 'replace') # use U+FFFD to replace that cannot be decoded input

print('str_msg = ', str_msg)

Output

byte literial = b'ST,GS,+ 0. 0.0tl.T\r\n\n'

str_msg = ST,GS,+ 0. 0.0tl.T

Reference

10.7. Exception

[exception, raise] throw exception (ref)

raise ValueError('Error: the mode does not supported.)

[exception, catch] How to catch IndexError (ref)

try:

print('feature = ', myfun())

except Exception as e:

msg = '[exception] file = ' + str(file) + '\nformat(e) = ' + format(e)

print(msg)

[call stack] How to print the call stack when exception (ref)

import traceback

traceback.print_exc()

10.8. File I/O

[print] How to print value -- str() (view)
[print] example (ref)

print('length =', len(wave.ys), 'wave.ys=', wave.ys)

print('array_2d = ' + str(array_2d))

[print, \n] Print without endline (ref)

print('.', end='')

sys.stdout.flush()

[read, input] How to get input from user -- age = input("Your age? ") (ref)

txt = input("Show Result? [Enter to continue] ")

[read, line] How to read line by line from a file

(ref)

filepath = 'Iliad.txt'

with open(filepath) as fp:

for cnt, line in enumerate(fp):

print("Line {}: {}".format(cnt, line))

with open(result_file) as f:

while 1:

line = f.readline()

print(line)

[serialization] How to serialize the data

Code

import pickle

class Serializion():

@classmethod

def save(cls, filename, data):

file = open(filename, 'wb')

pickle.dump(data, file)

file.close()

@classmethod

def load(cls, filename):

file = open(filename, 'rb')

data = pickle.load(file)

return data

@classmethod

def test(cls):

dict_data = dict()

dict_data['V1'] = [[1, 1, 1, 1], [1, 2, 3, 4]]

dict_data['V2'] = [[1, 0, 0, 1], [11, 12, 13, 14]]

print('dict_data = ', dict_data)

Serializion.save('data', dict_data)

dict_data_loaded = Serializion.load('data')

print('dict_data_loaded = ', dict_data_loaded)

if __name__ == '__main__':

Serializion.test()

Result

dict_data = {'V1': [[1, 1, 1, 1], [1, 2, 3, 4]], 'V2': [[1, 0, 0, 1], [11, 12, 13, 14]]}

dict_data_loaded = {'V1': [[1, 1, 1, 1], [1, 2, 3, 4]], 'V2': [[1, 0, 0, 1], [11, 12, 13, 14]]}

[pathlib, path, home] How to get the home directory

home = str(Path.home()) #/home/jing

[pathlib, path, create] How to create a path object (ref)

from pathlib import Path

# add subfolder -- using / operator

home = str(Path.home()) #/home/jing

db_root = Path(home+'/database') #/home/jing/database

New_path = db_root / 'ahadb' #/home/jing/database/ahadb

[folder, create] How to create a folder if it does not exist (ref)

Code

from pathlib import Path

filename = Path(./data/data.dat)

path = filename.parents[0] # get the directory containing filename

path.mkdir(parents=True, exist_ok=True)

Result

./data/mydata

[pathlib, string] How to represent a pathlib object to a posix string (ref)

Purpose:

有時候, 你會想把 pathlib.Path 物件轉成 posix 字串交給舊版的程式處理. 例如使用 python 3.5 時, 某些 file 檔案操作, 吃 pathlib.Path 會有 PosixPath object has no attribute rfind 的問題, 但你又不想為了這種程式碼的相容性問題, 升級整個 autobuild 系統中 docker 裡的 python 版本.

這時候, 請服用 the 'as_posix' method.

# method 1

from pathlib import Path

home = Path.home()

posix_home = home.as_posix()

# method 2

from pathlib import PureWindowsPath

>>> p = PureWindowsPath('c:\\windows')

>>> str(p)

'c:\\windows'

>>> p.as_posix()

'c:/windows'

[pathlib, folder] How to get the folder name from a giving file (ref)

>>> import pathlib

>>> p = pathlib.Path('/path/to/my/file')

>>> p.parents[0]

PosixPath('/path/to/my')

[pathlib, list] How to list files (ref)

from pathlib import Path

db_root = Path('/home/jing')

lst_file = [f for f in db_root.iterdir() if f.is_file()]

print('lst_file = ', lst_file)

[pathlib, list, suffix] How to list files for giving a specified suffix?

from pathlib import Path

db_root = Path('/home/jing')

lst_file = [f for f in db_root.iterdir() if f.suffix == '.txt']

print('lst_file = ', lst_file)

[pathlib, list, nest] How to find a set of file for given the directory and all subdirectories, recursively (ref)

from pathlib import Path

sorted(Path(str_directory).rglob('*.json'))

[pathlib, file, suffix] How to remove all suffix for a given file path (ref)

>>> PurePosixPath('my/library.tar.gz').stem

'library.tar'

>>> PurePosixPath('my/library.tar').stem

'library'

>>> PurePosixPath('my/library').stem

'library'

[patlib, check, exist] How to chec a file or folder exists (ref)

from pathlib import Path

# file

my_file = Path("/path/to/file")

if my_file.is_file():

# file exists

# folder

if my_file.is_dir():

# directory exists

# common

if my_file.exists():

# path exists

[os, check, exist] How to check a file exists (view)

import os.path

if os.path.isfile(strfullfilename):

print('The ' + strfullfilename + ' exists')

# other methods

import os

if file.is_file() and os.stat(file).st_size !=0:

[os, path, current] How to get the current working directory (view)

cwd = os.getcwd()

[write, line] How to write a line (view)
[xml, walk] Walk Through a XML File (view)
[json, read] How to read a json file

import json

from io import StringIO

import numpy as np

with open('20181026105158.json', 'r') as read_file:

dict_data = json.load(read_file)

# list all keys

print('dict_data.keys = ', dict_data.keys())

# list all data

'''

for key in dict_data.keys():

print('\n\nkey = ', key)

print(dict_data[key])

'''

print('sampling rate = ', dict_data['SampleRate'])

#print('I = ', dict_data['I'])

char_samples = dict_data['I']

samples = np.genfromtxt(StringIO(char_samples), delimiter=',')

#for i in range(len(samples)):

# print('samples[i] = ', samples[i])

[json, read, panda] How to read a json file, the panda way

df = pd.read_json(filename, lines=True)

dict_data = df.to_dict('list')

for key in dict_data.keys():

dict_data[key] = dict_data[key][0]

[json, read, ijson] How to read a json file, the ijson way, high speed, iterative, big json (ref)

[json, write, dict2json] How to write a json file

import json

data = {'key1': value1, 'key2': value2}

ret = json.dumps(data)

with open('out.json', 'w') as fp:

fp.write(ret)

[ext, remove] How to remove the file extension

print(os.path.splitext(file)[0])

a = filename.split('.')

filename = a[0]

print('ext = ', filename.split('.')[1])

[filename, get filename] How to get the filename from full path file name

a = filename.split('/')

filename = a[len(a) - 1]

[binary, read/write] How to read/write binary file (view)
[filelist, get] How to get a file list (ref)

def get_record_files(folder):

from os import listdir

from os.path import isfile, join

list_record_files = [f for f in listdir(folder) if isfile(join(folder, f)) and f.endswith('.json')]

list_record_files.sort()

return list_record_files

[file, list, walk, pathlib] How to get file list

from pathlib import Path

lst_file = list(Path('.').glob('*.py'))

for file in lst_file:

print(file)

[file, list, walk, os] How to get file list which name matched patten (view)

def get_rd_lst(db_root, filetype):

import os

import fnmatch

lst = []

for root, dirnames, filenames in os.walk(db_root):

for file in filenames:

if fnmatch.fnmatch(file, '*.'+filetype):

lst.append(file)

return lst

[file, rename] How to rename a file

import os

import shutil

import fnmatch

# 20180921100858-1.csv -> 1.csv

top = os.getcwd()

for root, dirnames, filenames in os.walk(top, topdown=False):

for file in filenames:

if fnmatch.fnmatch(file, '*.csv'):

new_file = file.split('-')[1]

print('new_file = ', new_file)

shutil.move(file, new_file)

print('done.')

[path, current path] How to get the current path

top = os.getcwd()

[list2array] How to conver a string list to string array

import numpy as np

list_file = ['a', 'b']

array_file = np.array(list_file)

[list2file] How to write a list to file

def list2file(list, filename):

f = open(filename,'w')

for file in list:

f.write(file + '\n') # python will convert \n to os.linesep

f.close()

print('Saved. Filename = ', filename)

# Usage

list2file(list)

[csv, array2csv] How to write a numpy array to csv file (view)

import numpy as np

np.savetxt("foo.csv", a, delimiter=",")

[csv, list2csv] How to conver a string list to a csv file

import numpy as np

list_file = ['a', 'b']

array_file = np.array(list_file)

np.savetxt('foo.csv', array_file, delimiter=',', fmt="%s")

[csv, read] How to read csv file (view)

#coding=utf-8

from numpy import genfromtxt

my_data = genfromtxt('my_file.csv', delimiter=',')

print(my_data)

[csv, write] How to write a csv file

-- random.csv --

,A,B,C,D,E,F,G,H,I,J

0,34718,504126,300537,3915,495726,393245,799912,0.13608098919247258,RAtjosiLi3,mLi76GdMfw

1,331436,274446,125015,738075,981729,339870,858206,0.3136006613850645,TWqZuUSKuF,XgJlyA9vhN

2,37632,903171,402204,349530,50752,715056,807419,0.9438284999300468,3C4EDdHJCk,NgbblS8bGL

3,971291,515941,891050,906111,285121,300320,202968,0.1535596225635345,jsLwyMzE6k,DMaIpgYKTH

4,351980,826238,330909,331309,274201,131890,363056,0.642644784666259,jW6LeSBQPq,fS4GBl4qY9

5,391821,919391,586872,320736,116217,879968,739022,0.6427802565359229,5nsC2ZLyHz,anwY3cw0iu

6,306897,90674,808579,859465,145243,123849,601992,0.17999096658733182,1YuOt9Az3p,q7CmaaFiFp

import pandas as pd

import numpy as np

# setting the number of rows for the CSV file

N = 1000000

# creating a pandas dataframe (df) with 8 columns and N rows with random integers between 999 and 999999 and with column names from A to H

df = pd.DataFrame(np.random.randint(999,999999,size=(N, 7)), columns=list('ABCDEFG'))

# creating one column 'H' of float type using the uniform distribution

df['H'] = np.random.rand(N)

# creating two additional columns with random strings

df['I'] = pd.util.testing.rands_array(10, N)

df['J'] = pd.util.testing.rands_array(10, N)

# print the dataframe to see what we have created

print(df)

# export the dataframe to csv using comma delimiting

df.to_csv("random.csv", sep=',')

[csv, read, partial] How to read a csv file row by row -- 最簡單的方法處理大 csv 檔案讀取 (ref)

-- random.csv --

,A,B,C,D,E,F,G,H,I,J

0,34718,504126,300537,3915,495726,393245,799912,0.13608098919247258,RAtjosiLi3,mLi76GdMfw

1,331436,274446,125015,738075,981729,339870,858206,0.3136006613850645,TWqZuUSKuF,XgJlyA9vhN

2,37632,903171,402204,349530,50752,715056,807419,0.9438284999300468,3C4EDdHJCk,NgbblS8bGL

3,971291,515941,891050,906111,285121,300320,202968,0.1535596225635345,jsLwyMzE6k,DMaIpgYKTH

4,351980,826238,330909,331309,274201,131890,363056,0.642644784666259,jW6LeSBQPq,fS4GBl4qY9

5,391821,919391,586872,320736,116217,879968,739022,0.6427802565359229,5nsC2ZLyHz,anwY3cw0iu

6,306897,90674,808579,859465,145243,123849,601992,0.17999096658733182,1YuOt9Az3p,q7CmaaFiFp

import pandas as pd

import numpy as np

# setting the number of rows for the CSV file

N = 1000000

# creating a pandas dataframe (df) with 8 columns and N rows with random integers between 999 and 999999 and with column names from A to H

df = pd.DataFrame(np.random.randint(999,999999,size=(N, 7)), columns=list('ABCDEFG'))

# creating one column 'H' of float type using the uniform distribution

df['H'] = np.random.rand(N)

# creating two additional columns with random strings

df['I'] = pd.util.testing.rands_array(10, N)

df['J'] = pd.util.testing.rands_array(10, N)

# print the dataframe to see what we have created

print(df)

# export the dataframe to csv using comma delimiting

df.to_csv("random.csv", sep=',')

Code

import csv

with open('random.csv') as csvfile:

data = csv.DictReader(csvfile)

for row in data:

print(row)

print(row['A'])

Output

OrderedDict([('', '0'), ('A', '376988'), ('B', '385694'), ('C', '689334'), ('D', '355737'), ('E', '741947'), ('F', '224477'), ('G', '92799'), ('H', '0.164930634850616'), ('I', '5a9HgWzCB0'), ('J', 'BGShbPaWrg')])

376988

[csv, read, partial] How to read a csv file only specified row (ref 不滿意的方法)

[csv, read] How to read a csv file row by row (ref)

import csv

with open(file, newline='') as csvfile:

data = csv.reader(csvfile, delimiter=',')

for row in data:

print('row = ', row)

[csv, read, panda] pd version (ref)

milk_data = pd.read_csv('https://raw.githubusercontent.com/plotly/datasets/master/monthly-milk-production-pounds.csv')

import pandas as pd

import numpy as np

df = pd.read_csv(strFullFileName, header=None)

print('df = ', df)

# data frame to list

mylist = df[0].values.tolist()

[csv, read, panda] read csv, named the column, and print

import pandas as pd

import numpy as np

df = pd.read_csv(strFullFileName, names=['name1', 'name2'])

print('df = ', df['name1'])

[csv, list, panda] How to conver a csv file to list

import pandas as pd

import numpy as np

df = pd.read_csv(strFullFileName, header=None)

print('df = ', df)

# data frame to list

mylist = df[0].values.tolist()

[csv, fullfilename, filename] How to conver a fullfilename file into filename csv file (view)
[csv, csv2dataframe] How to conver csv file to dataframe?

import pandas as pd

df = pd.read_csv(target_path + datacsv_file)

print('df = ', df)

print('df[I] = ', df['key1'])

print('df[v6] = ', df['key2'])

[panda, append row, dict] How to insert a dataframe to dataframe

dict1 = {'lead' : ' I', 'index' : 0}

dict2 = {'lead': 'II', 'index' : 1}

df1 = pd.DataFrame(dict1, index=[0])

df2 = pd.DataFrame(dict2, index=[1])

df12 = df1.append(df2)

df12.to_csv(filename, sep=',')

[panda, row num] How to get the row number from a given data frame

row_num = len(df.index)

if row_num == 0:

print('[Error] row_num == 0')

return False, None

[panda, row] How to get a row data by given index

value_list = []

# 1.

row_num = len(df.index)

if row_num == 0:

print('[Error] row_num == 0')

return False, None

#print('row_num = ', row_num)

for i in range(row_num):

df_row = df.iloc[i]

if df_row['lead'] == strlead:

feature_value = df_row[feature_name]

value_list.append(feature_value)

print('value_list = ', value_list)

print('value_list len = ', len(value_list))

[generator] How to write a generator program (ref) (flow)

一次讀一批 patch 出來

[generator, h5py] How to use h5py for training on large datasets (ref)

問題: 巨大的 training set 無法一次載入進行 training
作法/觀念: 搭配 HDF5 檔案格式和 data generator 方式, 一個 batch 一個 batch 的把資料餵給 model, 可以讓 model 多線程平行載入資料並且一個 batch 一個 batch 的訓練巨大 training set. 記憶體只需要 batch 的量. 非常好用.

[hdf5, panda] Table format (ref)

處理大 json 檔案 (ref)

Method 1

Step 1: Conver the json 2 hdf5 format

Step 2: Processing the big data

from pandas import (

DataFrame, HDFStore

)

import pandas as pd

import numpy as np

# creating a dataframe, here we use numpy to generate random numbers

df = DataFrame(np.random.randn(5,3), columns=['A', 'B', 'C',])

# creating a HDF5 file

store = HDFStore('dataset.h5')

# adding dataframe to the HDF5 file

store.put('d1', df, format='table', data_columns=True)

# access the store

store['d1']

[hdf5, panda] Read/Write hdf (ref)

# pip install tables

def save_to_hdf(dict_ecg, strkey, strFullFileName):

df = pd.DataFrame(dict_ecg)

df.to_hdf(strFullFileName, strkey, mode = 'w')

def load_from_hdf(strFullFileName, strkey):

return pd.read_hdf(strFullFileName, strkey)

def hdf_read_write_demo():

path = '/jing_home/src/MI/'

ecgfile='test.csv'

dict_ecg = get_ecg_data(path+ecgfile)

# dict -> hdf

save_to_hdf(dict_ecg,'training', './test.hdf')

# hdf -> dict

df = load_from_hdf('./test.hdf', 'training')

print('df = ', df)

print('df[I] = ', df['II'])

print('df[v6] = ', df['v6'])

[h5py, append write/batch read] Append write to h5py file (view)
[file, create] If the file does not at local, dowload it (view)
[file, folder] Get the folder name by giving the full file name

def splitall(strFullFilename):

"""Split a path into all of its parts.

From: Python Cookbook, Credit: Trent Mick

"""

allparts = []

while 1:

parts = os.path.split(strFullFilename)

if parts[0] == strFullFilename:

allparts.insert(0, parts[0])

break

elif parts[1] == strFullFilename:

allparts.insert(0, parts[1])

break

else:

strFullFilename = parts[0]

allparts.insert(0, parts[1])

return allparts

FullFilenameList = splitall(strFullFilename)

strParentFolderName= FullFilenameList[len(FullFilenameList)-2]

[file, walk] List files from a specific top folder (ref)

# list *.dat from /ptbdb (including all sub-directories)

top = os.getcwd() + '/ptbdb'

for root, dirnames, filenames in os.walk(top, topdown=True):

for file in filenames:

if fnmatch.fnmatch(file, '*.dat'):

print(file)

strFilename=file #s0504_re.dat

strFullFilename=os.path.join(root, strFilename) # /home/jing/ami/wfdb-python/ptbdb/patient267/s0504_re.dat

#刪除副檔名

strRecordName= os.path.splitext(strFilename)[0] # s0341lre

FullFilenameList = splitall(strFullFilename)

strParentFolderName= FullFilenameList[len(FullFilenameList)-2] # patient267

def splitall(strFullFilename):

"""Split a path into all of its parts.

From: Python Cookbook, Credit: Trent Mick

"""

allparts = []

while 1:

parts = os.path.split(strFullFilename)

if parts[0] == strFullFilename:

allparts.insert(0, parts[0])

break

elif parts[1] == strFullFilename:

allparts.insert(0, parts[1])

break

else:

strFullFilename = parts[0]

allparts.insert(0, parts[1])

return allparts

[file, rename] How to reanme files -- os.rename(file, strNewFileName) (view)
[find, glob] How to get *.dat from a given folder

#code

import glob, os

path = '/jing_home/test/*/'

datfiles = glob.glob(path + '*.dat')

print('datfiles = ', datfiles)

#result

datafiles = ['/jing_home/test/a/1.dat', .....]

[find, fnmatch] How to find a file from a given folder

def find(pattern, path):

result = []

i = 0

for root, dirs, files in os.walk(path):

for name in files:

if fnmatch.fnmatch(name, pattern):

i = i + 1

result.append(os.path.join(root, name))

print(i, os.path.join(root, name))

if i == 2:

return result

# Usage

file_list = find(mi_file+'.*', '/media/jing/D/work')

[find, fnmatch] How to find a set of find from a given folder

def find(file_list, path):

result = []

i = 0

for root, dirs, files in os.walk(path):

for mi_file in file_list:

pattern = mi_file+'.*'

for name in files:

if fnmatch.fnmatch(name, pattern):

i = i + 1

result.append(os.path.join(root, name))

print(i, os.path.join(root, name))

if i == 2:

break

return result

# usage

matched_file_list = find(st_file_list, '/media/jing/D/work/nas_local/chemi_mi_detector/CHIMEI')

print('matched_file = ', matched_file_list)

[folder, create] How to create a new folder if it does not exist (ref)

if not os.path.exists(strResult):

os.makedirs(strResult)

[folder, cur] How to know current folder -- os.getcwd()
[path, parse] Splitting a Path into list of terms, 依照 '_' 分隔完整檔名 (view)
[path, norm] Get the normalized path name -- os.path.normpath(path) (ref)

strWFDBModulePath = os.path.normpath(os.path.join(os.getcwd(), '../', 'wfdb-python'))

print(strWFDBModulePath)

[path, get] How to get the module path name (ref)

print('os.path.dirname(strFullFilename) = ', os.path.dirname(strFullFilename))

Code

# Give an absoluate filename, rename the folder name

def changeFolderName(strFullFilename, i):

srcPath = strFullFilename[:(strFullFilename.rfind('/'))]

dstPath = srcPath + '_' + str(i)

print('srcPath = ', srcPath)

print('dstPath = ', dstPath)

os.rename(srcPath, dstPath)

print(srcPath + ' -> ' + dstPath)

strFullFilename = '/home/jing/abc.xml '

changeFolderName(strFullFilename, 1)

Result

srcPath =/home/jing/abc

dstPath =/home/jing/abc_1

[module, path, get, imort] How to ge the module path name (ref)

from os.path import dirname

path_main_ecg = dirname(os.path.abspath(sys.modules[__name__].__file__))

PACKAGE_PATH_ECG_FEATURE = path_main_ecg +/v2/python/'

sys.path.extend([PACKAGE_PATH_ECG_FEATURE,])

[folder, rename] How to rename a folder by given the file in that (view)
[folder, parsing] Splitting a Path into list of terms (view)
[excel, write] How to create a Excel file by Python (view)
[excel, write, dict2excel] How to write a Excel file from dictionary (view)

import pandas as pd

from pandas import ExcelWriter

from pandas import ExcelFile

''''

-------------------------------

| key1 | key 2 | key 3 |

--------- --------- ---------

| v1 v2 v3 |

| v4 v5 v6

-------------------------------|

'''

def main():

# write to excel

lst_dict_data = []

lst_dict_data.append({'key 1': v1, 'key 2': v2, 'key3': v3)

lst_dict_data.append({'key 1': v4, 'key 2': v4, 'key3': v5)

lst_col = ['key1', 'key2', 'key3']

df = pd.DataFrame(lst_dict_data)

writer = ExcelWriter('Pandas-Example2.xlsx')

df.to_excel(writer, 'Sheet1', lst_col)

writer.save()

[excel, read] How to read an Excel file using panda (ref) (col_order)

import pandas as pd

from pandas import ExcelWriter

from pandas import ExcelFile

# READ

df = pd.read_excel('sample.xlsx', sheet_name='member', dtype = 'str')

#print("Column headings:")

#print(df.columns)

list_person = []

for i in df.index:

dict_person = {}

for key in df.columns:

dict_person[key] = df[key][i]

list_person.append(dict_person)

#print('one person = ', dict_person)

# WRITE

df = pd.DataFrame({'a':[1,3,5,7,4,5,6,4,7,8,9],

'b':[3,5,6,2,4,6,7,8,7,8,9]})

writer = ExcelWriter('Pandas-Example2.xlsx')

df.to_excel(writer,'Sheet1',index=False)

writer.save()

[path, filename] How to get filename for a given path

filename = Path('abc/def/file.csv').name #file.csv

[zipfile, extract] How to extract a passwd encrpyted zip file

Zipping the file

zip -e my.zip ./mac_pro/GLOBAL_CFG.py

Passwd:

Code

# extract *.py from my.zip

from zipfile import ZipFile

with ZipFile('my.zip', 'r') as zipObj:

lst_files = zipObj.namelist()

for file in lst_files:

if file.endswith('py'):

zipObj.extract(file, '.', passwd.encode('utf-8'))

Output

mac_pro/GLOBAL_CFG.py

10.9. Network

[network, connect] How to check if there is a connection (ref)

import socket

REMOTE_SERVER = "www.google.com"

def is_connected(hostname):

try:

# see if we can resolve the host name -- tells us if there is

# a DNS listening

host = socket.gethostbyname(hostname)

# connect to the host -- tells us if the host is actually

# reachable

s = socket.create_connection((host, 80), 2)

s.close()

return True

except:

pass

return False

%timeit is_connected(REMOTE_SERVER)

> 10 loops, best of 3: 42.2 ms per loop

[mac, address] How to get mac address (ref)

str_mac = ''.join(re.findall('..', '%012x' % uuid.getnode()))

10.10. Device I/O

[serial] How to access the serial port using Python

import serial

ser = serial.Serial('/dev/ttyUSB0') # open serial port

print('ser.name = ', ser.name) # check which port was really used

# ser.write(b'hello') # write a string

ser.close() # close port

with serial.Serial('/dev/ttyUSB0', 9600, timeout=1) as ser:

while 1:

line = ser.readline() # read a '\n' terminated line

print(line)

[rs232, check] How to check the connection for a RS232 to usb device? (blog)

import serial

def is_connected(self):

Port = '/dev/ttyUSB0' # 'COM1' (for Windows)

ok, port, msg = (1, '/dev/ttyUSB0', 'ok')

if not ok:

return 0, msg

ser = serial

try:

ser = serial.Serial(port, 9600, timeout=10)

while ser.read():

print('serial open test')

break

print('done!. serial closed')

ser.close()

return 1, 'ok'

except serial.serialutil.SerialException:

print('exception: device not found')

return 0, 'Error_Device_Not_Found'

Category	Cate-Level1	Item	Note
port	check	usb	lsusb
port	check	kernel	dmsg \| tail
port	check	device	/dev/ttyUSB0
port	check	stty	sudo stty -F /dev/sttyUSB0 -a
user	setup	group	sudo usermod -a -G dialout $USER

[usb, printer] How to print chinese text to escpos print using pyusb (view)

big-5

[usb, printer] How to print chinese text to escpos print using python-escpos (view)

10.11. Function

[function] function usage (view)
[function, main] How to write main functoin (ref)

#!/usr/bin/python

import sys

def main():

# print command line arguments

"""

sys.argv[0] # program

sys.argv[1] # arg1

sys.argv[2] # arg2

"""

for arg in sys.argv[1:]:

print ('arg =', arg)

if __name__ == "__main__":

main()

[fun, arg, list] How to pass a list to function (view)

[main, arg, parse] How to use argparse? (view)

import argparse

# 設定參數

parser = argparse.ArgumentParser()

parser.add_argument('-mode', '--mode', type=int, default = 0, help="The running mode")

parser.add_argument(-file, '--file_input', help="An input file")

parser.add_argument('-log', '--file_log', help="The output file") parser.add_argument('-debug', '--file_debug', help="debug information")

# 讀取參數

args = parser.parse_args()

print("args.mode =", args.mode)

print("args.file_ecg = ", args.file_input)

print("args.file_log = ", args.file_log)

print("args.file_debug = ", args.file_debug)

Visual Studio Code Debug

{

"name": "Python: Current File (Integrated Terminal)",

"type": "python",

"request": "launch",

"program": "${file}",

"console": "integratedTerminal",

"args": ["-mode", "0", "-file", "a.json", "-log", "a.log", "-debug", "a.debug"]

}

[import, path] Import the packages located at the relative folder

import sys, os

path = os.path.normpath(os.path.join(os.getcwd(), '../', 'wfdb-python'))

sys.path.append(path)

import wfdb

[function, library] 把 function 整理在一個地方

import sys

sys.path.insert(1, r'./mylib) # add to pythonpath

from my_util_file import my_function

[function, name] How to get the current function name or method name (ref)

import inspect

import types

from typing import cast

class myClass():

def del_data(self):

str_name=cast(types.FrameType, inspect.currentframe()).f_code.co_name

print ('method_name = ' , str_name)

Test

obj = myClass()

obj.del_data()

Output

fun_name = del_data

import inspect

import types

from typing import cast

def del_data(self):

str_name=cast(types.FrameType, inspect.currentframe()).f_code.co_name

print ('fun_name = ' , str_name)

Test

del_data()

Output

fun_name = del_data

[function, arg, variable number] How to pass variable number of arguments to function (ref)

Code

def fun(*arg_list):

len_arg = len(arg_list)

print('len_arg = ', len_arg)

for var in arg_list:

print('var = ', var)

fun(1, 2, 3)

Result

jing@jing-pc:~/test/basic_test$ python fun_arg_var.py

len_arg = 3

var = 1

var = 2

var = 3

Code 2: Implement a function like print

def fun(*arg_list):

len_arg = len(arg_list)

print('len_arg = ', len_arg)

for var in arg_list:

print(var, end='')

print('')

fun('a = ', 1, ' b= ', 2, 3)

Result

jing@jing-pc:~/test/basic_test$ python fun_arg_var.py

len_arg = 5

a = 1 b= 23

[function, show_error]

def show_error(*list_msg):

if not global_set.bRelease:

msg = ''

for var in list_msg:

msg = msg + str(var)

print(msg)

[function, arg, array] How to pass an array to function (view)

[function, arg, 2d array] How to pass a 2d array to function (view)

x = [1,2]

print('x = ' + str(x)) # x is an 1d array

print('[x] = ' + str([x])) # [x] is an 2d array that created from the x (1d array)

[function, def] How to declare a function (view)
[function, invoke] How to call a function by its name

#code

def fun_A(arg_A):

print('fun_A::arg_A = ', arg_A)

def fun_B(arg_B):

print('fun_B::arg_B = ', arg_B)

method_to_call = globals()['fun_A']

method_to_call('myA')

method_to_call = globals()['fun_B']

method_to_call('myB')

#result

fun_A::arg_A = myA

fun_B:: arg_B = myB

[function, return, multi] How to return multiple values for a function (ref)

Code: Using Dictionary

def g(x):

y0 = x + 1

y1 = x * 3

y2 = y0 ** 3

return {'y0':y0, 'y1':y1 ,'y2':y2 }

# usage

retDict = g(1)

print(retDict)

print(retDict['y0'])

Result

10.12. Condition

[switch] How to implement switch in Python (ref)

def run():

switch = { 0 : self._Yushan,

1 : self._Taroko,

2 : self._Sheipa,

}

switch[park]()

def _Yushan(self):

print('Yusan')

def _Taroko(self):

print('Taroko')

def _Sheipa(self):

print('Sheipa')

driver.find_element_by_xpath('//*[@title="雪霸國家公園"]').click()

time.sleep(1) # Let the user actually see something!

switch = { MODE_RELEASE : run_release,

MODE_DEBUG : test_single_any,

MODE_BATCH: run_batch,

}

if dict_args['mode'] == MODE_RELEASE:

switch[dict_args['mode']](dict_args)

else:

switch[dict_args['mode']]()

#Code

def fun_1():

print('fun 1')

def fun_2():

print('fun 2')

def fun_3():

print('fun 3')

options = {1 : fun_1,

2 : fun_2,

3 : fun_3,

}

try:

options[2]()

options[0]() # <-- throw Key Error

except KeyError:

print('[Error] Key Error')

#Result

fun 2
[Error] Key Error

10.13. Loop

[for, loop] How to generate i in 0, 1, 2, 3, ... len(array) (ref)

for i in range(len(x))
for i in range (50)

[for, loop] How to generate i in 0, 1, 2, 3

range(start, stop, step])

[for, continue] How to use continue in loop? -- loop
[for, iterate, array] How to iterate over array (ref)

import numpy as np

myArray = np.arange(6)

print('myArray = ', myArray)

for x in myArray:

print('x = ', x)

Result

myArray = [0 1 2 3 4 5]

x = 0

x = 1

x = 2

x = 3

x = 4

x = 5

import numpy as np

myArray = np.arange(6)

print('myArray = ', myArray)

for x in np.nditer(myArray):

print('x = ', x)

Result

myArray = [0 1 2 3 4 5]

x = 0

x = 1

x = 2

x = 3

x = 4

x = 5

10.14. boolean

[boolean, not] How to use not operator in Python (view)

#code

bRelease = True

bShow = not bRelease # 關鍵片段 bShow = !bRelease

print('bShow = ', bShow)

#result

bShow = False

[boolean, or] How to use or operator? -- (ref)

if 1 < 2 and 4 > 2:
print("condition met")

if 1 > 2 and 4 < 10:
print("condition not met")

if 4 < 10 or 1 < 2:
print("condition met")

[boolean, None] How to determine None ? -- if x is None: (ref)

if file is None:

print('file == None')

10.15. List

[list, create ,1d, append] How to create a 1D list with 40 elements (view)

obj=[]

# append element

obj.append(element)

[list, create, append] How to create an list contains sequencs [[1, 2, 3, 4, 5], [2, 3, 4, 5, 6], ..., etc] (view)

result = [] # create a list

result.append(data[0:5]) # [[1, 2, 3, 4, 5] ]

result.append(data[1:6]) # [[1, 2, 3, 4, 5], [2, 3, 4, 5, 6]]

print(result)

[list, remove] How to remove the first element from the list (ref)

>>> l = ['a', 'b', 'c', 'd']

>>> l.pop(0)

'a'

>>> l

['b', 'c', 'd']

>>>

[list, copy] What's different between shallow copy and deep copy for a list? (ref)

# Shallow copy

Code

class A():

def __init__(self):

self.data = 1

lst_org = list()

lst_org.append(A())

lst_new = lst_org.copy()

lst_org[0]

Lst_new[0]

Result

>>> lst_org[0]

<__main__.A object at 0x7fd47cfd7278>

>>> lst_new[0]

<__main__.A object at 0x7fd47cfd7278>

# Deep copy

Code

import copy

class A():

def __init__(self):

self.data = 1

lst_org = list()

lst_org.append(A())

lst_new2 = copy.deepcopy(lst_org)

lst_org[0]

Lst_new2[0]

Result

>>> lst_org[0]

<__main__.A object at 0x7fd47cfd7278>

>>> lst_new2[0]

<__main__.A object at 0x7fd47cfd7400>

[list, list2array] How to create an array from a given list (view)

array = np.array(result) # conver to array.

# [[1 2 3 4 5]

# [2 3 4 5 6]]

print(array)

print(str(array[1].size)) # 5

print(array[0]) # array[0] = [1 2 3 4 5]

print(array[1]) # array[1] = [2 3 4 5 6]

[list, interleave] How to create a list by interleaving zeros to a given list

signals1= obj.get_ecg_data2(0, 2000, channel) # signals1 = [1, 2, 3]

listofzeros = [0] * len(signals1) # 產生相同長度的 0 list

inter_signals1 = [*sum(zip(listofzeros,signals1),())] # 把 0 交叉放進 signal1

signals = np.c_[inter_signals1, signals2] # signals = [1, 0, 2, 0, 3, 0]

[list, transpose] How to transpose a 2D list (ref)

list(map(list, zip(*lst_data)))

[list, add] How to sum of two lists (ref)

[x + y for x, y in zip(first, second)]

[list, add, scale] How to add a integer to each element of the list (ref)

[x+1 for x in mylist]

>>> map(lambda x:x+1, [1,2,3])

[2,3,4]

[list, dict, sorting] How to soring a dictionary list based on the spcified key (ref) (ref2)

newlist = sorted(list_to_be_sorted, key=lambda k: k['name'])

from operator import itemgetter

newlist = sorted(list_to_be_sorted, key=itemgetter('name'))

def do_sort(cls, lst_seg):

def get_key(seg):

return seg.start

return sorted(lst_seg, key=get_key)

10.16. Tuple

[tuple, create] How to create a tuple with only element?

The most important difference between a list and a tuple is mutability. Unlike lists, tuples are immutable i.e. they can’t be modified [ref].

Code

tuple(['hello'])

Output

('hello',)

10.17. Set

[set, create] How to create an empty set?

set_item_type = set()

set_item_type.add('1')

10.18. Dict

[dict, update] How to update the dictionary (view)
[dict] how to use dict (view)
[dict, diff] How to get the difference between dictionaries (ref) (best)

value = { k : second_dict[k] for k in set(second_dict) - set(first_dict) }

[dict, list2dict] How to conver a list to dict

mylist = [1, 2, 3, 4]

mydict = {}

mydict['A'] = mylist # 'A' : [1, 2, 3, 4]

print('mydict[A] = ', mydict['A'])

[dict2list] How to cover a dictionary data to a 2D list

LEAD_LIST = ['I', 'II']

dict_data ={}

dict_data['I] = [.....]

dict_data['II'] = [.......]

[dict_data[ch] for ch in LEAD_LIST]

[dict, merge] How to merge two dictionaries? (ref)

Code

x ={'key1':'value1', 'key2':'value2'}

y ={'key3':'value3', 'key4':'value4'}

z = {**x, **y}

print('z=', z)

Output

{'key1': 'value1', 'key2': 'value2', 'key3': 'value3', 'key4': 'value4'}

[dict, key] How to get the latest key?

a={'a':1, 'b':2}

list(a)[-1]

Output

'b'

[dict, key, index] How to get the key index for giving a key to a dictionary(ref)

index = list(mydict).index(mykey)

[dict, key, value] How to get the key for giving a name from dictionary (ref)

key =find_key(my_dict, value)

def find_key(self, input_dict, value):

return next((k for k, v in input_dict.items() if v == value), None)

[dict, key, sorted] How to get a sorted key list from dictionary

for key in sorted(mydict.keys()):

print(key)

[dict, dict2df] How to create a dataframe from dict

import pandas as pd

df = pd.DataFrame()

dict_data = {'name': 'my_name', 'id': '123'}

dict_temp = dict()

for key in dict_data.keys():

dict_temp[key] = [dict_data[key]]

temp_df = pd.DataFrame(dict_temp)

df = df.append(temp_df, ignore_index=True)

df.head()

Output

name id

0 my_name 123

dict_data = {'name': 'my_name2', 'id': '567'}

dict_temp = dict()

for key in dict_data.keys():

dict_temp[key] = [dict_data[key]]

temp_df = pd.DataFrame(dict_temp)

df = df.append(temp_df, ignore_index=True)

df.head()

Output

name id

0 my_name2 567

# setup column order

lst_col_order = ['id', 'name']

df = df[lst_col_order]

df.head()

Output

id name

0 567 my_name2

def get_dataframe_record(dict_data):

dict_temp = dict()

for key in dict_data.keys():

dict_temp[key] = [dict_data[key]]

return dict_temp

import pandas as pd

df = pd.DataFrame()

dict_data = {'name': 'my_name', 'id': '123'}

temp_df = pd.DataFrame(get_dataframe_record(dict_data))

df = df.append(temp_df, ignore_index=True)

df.head()

Output

name id

0 my_name 123

dict_data = {'name': 'my_name2', 'id': '567'}

temp_df = pd.DataFrame(get_dataframe_record(dict_data))

df = df.append(temp_df, ignore_index=True)

df.head()

Output

name id

0 my_name 123

1 my_name2 567

[dict, dict2df] How to conver a dict to dataframe

lst_row= ['row1', 'row2']

dict = {'col1':[], 'col2':[]}

dict['col1'] = [1, 2]

dict['col2'] = [3, 4]

pd.DataFrame(dict, index=lst_row)

print('dict=', dict)

Output

col1 col2

row1 1 3

row2 2 4

import pandas as pd

import numpy as np

df = pd.DataFrame(mydict) # dict to dataframe

df.to_csv(strFullFileName, sep=',') # save to csv

[dict, dict2dataframe, report] Use dataframe to report

lst_row_key = ['p_dur', 'qrs_dur']

dict = {'mean':[], 'std':[]}

dict['mean'] = [1, 2]

dict['std'] = [3, 4]

import pandas as pd

df = pd.DataFrame(dict, index=lst_row)

print('dict=', dict)

Output

mean std

p_dur 1 3

qrs_dur 2 4

[dict, dataframe2dict, df2dict] How to convert a dataframe to dict (ref)

'''

Dataframe:

col1 col2

row1 1 0.50

row2 2 0.75

'''

df = pd.DataFrame({'col1': [1, 2],

'col2': [0.5, 0.75]}, index=['row1', 'row2'])

dict = df.to_dict('list')

print('dict = ', dict)

# Result

dict = {'col1': [1, 2], 'col2': [0.5, 0.75]}

# remove list version

df = pd.DataFrame({'col1': [1, 2],

'col2': [0.5, 0.75]})

dict = df.to_dict('list')

for key in self.__dict_ecg_data.keys(): # remove []

dict[key] = dict[key][0]

print('dict = ', dict)

# Result

dict = {'col1': [1, 2], 'col2': [0.5, 0.75]}

import pandas as pd

df = pd.read_json(filename, lines=True)

dict = df.to_dict()

# df =

# col1, col2

# index = 0 v1 v2

# dict =

{col1: {0, v1}, col2: {0, v2}

[dict, two dim] How to create a two dimension dictionary

new_dic = {}

new_dic[1] = {}

new_dic[1][2] = 5

[list2file] How to write a list to file

def list2file(list, filename):

f = open(filename,'w')

for file in list:

f.write(file + '\n') # python will convert \n to os.linesep

f.close()

print('Saved. Filename = ', filename)

# Usage

list2file(list)

[dict, key, list] How to list the keys from a dictionary? (ref)

for key in newdict.keys():

print(key)

list(newdict.keys())

[dict, key, exist] How to determine a dictionary has a key?

dict_data = {}

if lead not in dict_data:

dict_data[lead] = []

dict_data[lead].append(node)

[dict, key] 動態給 key

hr = 50

k_value = 100

key_list = ['hr', 'k_value']

dict = {

'hr': hr,

'k_value':k_value

}

dict2 = {}

for key in key_list:

dict2[key] = eval(key)

dict 與 dict2 一樣

[dict, key, value] How to switch a dictionary key value paire (ref)

my_dict2 = {y:x for x,y in my_dict.items()}

[dict, key, name] How to change the dictionary key name (ref)

dictionary[new_key] = dictionary.pop(old_key)

[list, key-value] How to get a list from a dictionary where the value > 1

dict = {'I': 0, 'II': 0, 'III': 0, 'aVL': 0, 'aVF': 0, 'V1': 2, 'V2': 2, 'V3': 2, 'V4': 0, 'V5': 0, 'V6': 0}

lst_lead_ste = [key for key in dict if dict[key] > 1]

Output:

['V1', 'V2', 'V3']

[dict2csv] How to write a dict datatype to csv file

''''

-------------------------------

| file | col1 | col2 |

--------- --------- ---------

| file1 v1 v2 |

| file2 v3 v4 |

------------------------------ |

'''

lst_key = ['file', 'col1', 'col2']

list_dict_data = []

list_dict_data.append({'col1':'v1', 'col2':'v2', 'file': 'file1'})

list_dict_data.append({'col1':'v3', 'col2':'v4', 'file': 'file2'})

out_filename = 'test.csv'

import pandas as pd

df = pd.DataFrame(data=list_dict_data)

df = df[lst_key] # 按照指定的 column order

df.to_csv(out_filename, sep=',', index=False)

print(out_filename+':ok')

# csv2dict panda version

import pandas as pd

df = pd.read_csv(filename)

list_dict_data = df.to_dict('records')

Result

List_dict_data = [

{'col1':'v1', 'col2':'v2', 'file': 'file1'},

{'col1':'v3', 'col2':'v4', 'file': 'file2'},

]

import pandas as pd

import numpy as np

df = pd.DataFrame(mydict)

df.to_csv(strFullFileName, sep=',')

10.19. Panda, DataFrame

[lib, analysis] The data analysis library -- Pandas (ref)

[df, create] How to create a dataframe (ref)

import pandas as pd

df = pd.DataFrame()

dict_data = {'name':['my_name'], 'id':['123']}

temp_df = pd.DataFrame(dict_data)

df = df.append(temp_df, ignore_index=True)

df.head()

Output

name id

0 my_name 123

dict_data = {'name':['my_name2'], 'id':['567']}

temp_df = pd.DataFrame(dict_data)

df = df.append(temp_df, ignore_index=True)

df.head()

Output

name id

0 my_name 123

1 my_name2 567

[df, select, row, col] How to get the col serials (ref)

import pandas as pd

df = pd.read_csv(file.csv)

print('df = ', df)

lstfile = df.iloc[:,0]

print('lstfile = ', lstfile)

[csv, read, panda] read csv, named the column, and print

import pandas as pd

import numpy as np

df = pd.read_csv(strFullFileName, names=['name1', 'name2'])

print('df = ', df['name1'])

[df, access] How to access row

df = pd.read_csv(file, names=['1', 'ans'])

int_total = len(df['ans'])

cnt = 0

for str_row in df['ans']:

if str_row == str_ans:

cnt = cnt + 1

[dataframe2list] How to convert values to list (ref)

import pandas as pd

df = pd.read_csv(file.csv)

print('df = ', df)

lstfile = df.iloc[:,0].to_list()

print('lstfile = ', lstfile)

[panda, nan] How to determine nan for missing values (ref)

import pandas as pd

pd.isnull(s)

10.20. Array

10.20.1. Definition

[scipy array, survey] Lots of array example (ref) (basic)

# 注意: dimenaion 跟 shape 不一樣, 有各自的定義
[ndarray] the ndarray is the Numpy array
[axis] the order of indexing into the array (ref)

import numpy as np

a = np.array([[1,2],[3,4]])

# [ [1, 2]

[3, 4] ]

sum0 = np.sum(a, axis=0) # axis 0 (垂直) 的加起來. [1 + 3 = 4, 2 + 4 = 6]

sum1 = np.sum(a, axis=1) # axis 1 (水平) 的數值加起來

Print (a)

print (sum0)

print (sum1)

Result

[ [1, 2]

[3, 4] ]

[4 6]

[3 7]

[dimension] the number of axes (dimensions) of the array. the rank of axis (ref)
[dimension] How to get the dimension of an array -- x.ndim (view)
[shape] a tube indicating the number of elements along each axis of array (ref)
[shape] How to get the shape of an array -- print('array.shape = ', 'array.shape)

10.20.2. Create 1d

[0d, create, ()] How to create a zero-dimensional array (view)

x = np.array(1) # create a zero dimension array, that can be imutable access

x =123

[length, 1d] How to get the length of an array -- len(your_array)
[1d, create, 1x1] How to create an one-dimensional array with zero values (view)

x_1x=np.zeros(1)

Result

x_1x = [ 0.] x_1x shape= (1,)

[1d, create, 1x1] How to create an one-dimension array with shape (1, ) (view)

x_1x=np.ones(1)

x = np.array([1]) (ref)

Result

x_1x = [ 0.] x_1x shape= (1,)

[1d, create, 1x3] How to create an one-dimension array with shape (3, )

import numpy as np

x = np.array([1,2,3])

print('x.shape=', x.shape)

print('x.ndim = ', x.ndim)

Result

x.shape= (3,)

x.ndim = 1

[1d, create, 1xN, zeros] How to create 1xN zero array

Code

N = 100

x_1xN = np.zeros(N)

print('x_1xN = ', np.shape(x_1xN))

Result

x_1xN = (100,)

[1d, create, 1xN, zeros, int type] How to create a 1xN int zero array

Code

N = 100

x_1xN = np.zeros(N, 'int')

[1d, create, 1xN, linspace] How to setup a linespace array (view) (ref)

myArray = linespace(start, stop, N)

In [6]: linspace(0, 10, 5)
Out[6]: array([ 0. , 2.5, 5. , 7.5, 10. ])

[1d, create, 1xN, arange] How to get an array of evenly spaced values (view)

A_1234 = np.arange(1, 10, 1)
np.arange(start, end, interval)

[array, create ,1d] How to create a 1D array with 40 elements (view)

10.20.3. Create 2d

[2d, create, 3xn] How to create a 2D list with row = 3

data_3xn = [ [], [], [] ]

data_3xn[0] .append (填入資料到 channel 0)

[2d, create, rolxcol] How to create a 2D list with variable list elements? (view)

如果 row 是變數, 那就用 for-range

# [] ----> [.......]

# ...

data_rowxcol = [[] for y in range(row)]

data_rowxcol[0].append (填入資料到 channel 0)

data_rowxcol[1].append (填入資料到 channel 1)

data_rowxcol[2].append (填入資料到 channel 2)

[2d, create, 2x3] How to create a 2x3 array

a_2x3 = np.array([[1, 2, 3],

[2,3,4]])

print(a_2x3.shape)

Result

(2, 3)

[2d, create, 2d] How to create a 2D array (view)
[2d, create, shape] How to create a 2D array for giving a shape

N = 2

point_Nx2 = np.ones(shape = (N,2))

point_Nx2[0] = [1, 2]

point_Nx2[0] = [2, 3]

#point_Nx2[0] = np.array([1 ,2])

#point_Nx2[1] = np.array([2 ,3])

print('len(point_Nx2) = ', len(point_Nx2))

print('type(point_Nx2[0]) = ', type(point_Nx2[0]))

print('type(point_Nx2[1]) = ', type(point_Nx2[1]))

Result

len(point_Nx2) = 2

type(point_Nx2[0]) = <class 'numpy.ndarray'>

type(point_Nx2[1]) = <class 'numpy.ndarray'>

[2d, create] How to create a 2D array (view)

# 注意: dimenaion 跟 shape 不一樣

x = np.array([[1,2,3],[4,5,6], [1,2,3]]) # create two-dimensional array with shape (3, 3)

Result

x.shape= (3, 3)

[2d, create] How to construct an 2d array from two Python lists (view)

[2d, create, (0, 0)] How to create a two-dimensional array with shape (0,0)

x = np.ones( shape=(0, 0) )

print('x.shape=', x.shape)

print('x.ndim = ', x.ndim)

print('x = ', x) # access demo

[2d, create, (1, 1)] How to create a two-dimensional array with shape (1,1) (view)

x_1x1=np.zeros((1,1))

Result

x_1x1 = [[ 0.]] x_1x1 shape= (1, 1)

[2d, create, (2, 3)] How to create a two-dimensional array with shape (2, 3) (view)

x_2x3 =

[ [ 0. 0. 0.]

[ 0. 0. 0.] ]

x_2x3 shape= (2, 3)

[2d, create, (N, 1)] How to create a two-dimensional array with shape (N, 1) in random nums

N = 10

import numpy as np

x = np.random.random((N, 1))

print('shape x = ', np.shape(x))

Result

shape x = (10, 1)

[2d, create, concatenation] How to concate two 1D array to 2D array -- np.c_ (ref) -- insert column

>>> np.c_[np.array([1,2,3]), np.array([4,5,6])]

array([[1, 4],

[2, 5],

[3, 6]])

[array, col] How to grab an entire column, column vector (view)

Col0= Array2D[:,0]

print('Col0 = ', Col0)

[array, col, 20] How to get 20 elements from a column (view)

[2d, combine] How to generate a 2d array by combining two arrays -- c = np.vstack((a_2x3, b_2x3)) => c_4x3 (view)
[2d, 1D -> 2D, reshape] reshape((3, 2)) -- Create 2D array from a 1D array (view)

3x2 三個兩個成員的陣列

[2d -> 1d, reveal] How to create a contiguous flattened 1d array from input -- reveal (ref)

>>> x = np.array([[1, 2, 3], [4, 5, 6]])

>>> print(np.ravel(x))

[1 2 3 4 5 6]

10.20.4. Basic OPs

[code] How to break your long line python code into multiple lines? (ref)

=> backslash

#Code

a = '1' + '2' + '3' + \

'4' + '5'

[index] python array 接受 -1 index. 最後一項 (ref)

#code

xList = [1, 2, 3]

xArray = np.asarray(xList)

x1 = xArray[0]

print('x1 = ', x1)

x = xArray[-1]

print('x = ', x)

#Result

x1 = 1

x = 3

[size] The total number of elements of the array. This is equal to the product of the elements of shape
[dim, add] How to add an additional dimension to numpy array (ref)

有時候, 不想要批次丟進去處理, 只想要單筆資料進行預測. 這時, 你會需要把讀進來的單筆 sample x 做 dimension extension, 然後餵給訓練好的模型做預測. 下面的方法, 是把 (2, 3) shape 的 x, 變成 (1, 2, 3) 的 X.

Key

X = x[np.newaxis, ...] # 注意: 就是 ...

Code

import numpy as np

x = np.array([ [1,2,3],[4,5,6]])

X = x[np.newaxis, ...] # 注意: 就是 ...

print('x.shape = ', x.shape)

print('X.shape = ', X.shape)

Result

x.shape = (2, 3)

X.shape = (1, 2, 3)

[len] How to get array length (ref)
[append] How to append a number to numpy array

#Code

import numpy as np

list_a= [1, 2, 3]

array_a = np.array(list_a)

array_b = np.append(array_a, 4) # key point

print('array_a = ', array_a, 'type = ', type(array_a))

print('array_b = ', array_b, ' type = ', type(array_b))

#Result

array_a = [1 2 3] type = <class 'numpy.ndarray'>

array_b = [1 2 3 4] type = <class 'numpy.ndarray'>

[list, histogram] How to get the max occurence element for a given list

Code

import numpy as np

mylist1 = [1,1, 2,2,2,2,3,4,5,5,5]

mylist = list(set(mylist1))

histo = list()

for i in mylist:

histo.append(mylist1.count(i))

mylist[np.argmax(histo)]

Output
2

[array, mean] How to get the mean for a giving array (view)

[access, two list] How to iteratly two lists

Code

lst_data1 = [1, 2, 3, 4]

lst_data2 = [5, 6, 7, 8]

for (x, y) in zip(lst_data1, lst_data2):

print(x,y)

Result

If the list data was collected in a list, like this

Code

data1 = [1, 2, 3, 4]

data2 = [5, 6, 7, 8]

lst_data = list()

lst_data.append(data1)

lst_data.append(data2)

for (x, y) in zip(*lst_data):

print(x,y)

Result

1 5

2 6

3 7

[vstack] How to use vstck

#code

import numpy as np

xx = [1, 2, 3]

xx = np.vstack((xx, [4, 5, 6]))

xx = np.vstack((xx, [7, 8, 9]))

print('xx = ', xx)

#result

xx = [[1 2 3]

[4 5 6]

[7 8 9]]

[slice] How to get a slice from an array (ref)

#code

import numpy as np

a = np.reshape(np.arange(9),(3,3))

print('a = ', a)

print('a[:,1] = ', a[:,1])

#result

a = [[0 1 2]

[3 4 5]

[6 7 8]]

a[:,1] = [1 4 7]

[foreach] How to evaluate the elements for an array (view)
[trans, Nx1 -> 1xN] How to transfer a Nx1 array to 1xN array (view)

signals_Nx1 = signals

print(signals_Nx1)

signals_1xN = signals_Nx1.reshape(-1)

print(signals_1xN)

[transpose, 1xN -> Nx1] How to transfer a 1xN array to Nx1 array (view)

signals_Nx1 = signals

print(signals_Nx1)

signals_1xN = signals_Nx1.reshape(-1)

print(signals_1xN.shape)

print(signals_1xN)

signals_new_Nx1 = signals_1xN.reshape(20000,1)

print(signals_new_Nx1)

[transpose, list-> n x 1] How to trans a list to nx1 array where each element is 1 dimension

#code

list_K = [0] * 10

print('list_K = ', list_K)

y = np.expand_dims(list_K, axis=1)

print('y = ', y)

#result

list_K = [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

y = [[0]
[0]
[0]
[0]
[0]
[0]
[0]
[0]
[0]
[0]]

[transpose, list -> n x m] How to cut your long sample list into n vector where each element has m dimension

Purpose: 把一長串的 sample list 切割成整齊的 n 個 m 長度 sample 陣列

#code

dim = 2

A = [1, 2, 3, 4]

B = np.reshape(A, (-1, dim))

print('B = ', B)

#result

B = [[1 2]
[3 4]]

[access, N-by-1] How to access the element of N-by-1array

kk[1,0] = 123

[access, column] How to write data to the column of array (view)

A=[[ 0. 0.]

[ 0. 0.]

[ 0. 0.]]

Updated A = [[ 0. 0. ]

[ 0.84147098 0. ]

[ 0.90929743 0. ]

[ 0.14112001 0. ]

[-0.7568025 0. ]

[-0.95892427 0. ]

[-0.2794155 0. ]

[ 0.6569866 0. ]

[ 0.98935825 0. ]

[ 0.41211849 0. ]]

[array, add] How to get add between element of array (view)

C = A + B

[array, subarray] How to get sub-elements of array

取得前面 N-1 筆資料 (去除最後一筆資料):

a[ : -1]

取得後面 N-1 筆資料

A[1:]

[count, if] How to count the element if condition is true

Purpose: Calcuating the class ratio in traning set for evaluating the balance issue

#code

array = [1, 1, 0, 0, 0, 0, 1]

sum(1 for x in array if x == 0)

#result

[string array] How to create a string array (view)

NoisePool = ['a', 'b']

print('NoisePool[0]=', NoisePool[0])

[squeeze] What is squeeze for array -- 把外面 (最左邊) 拔掉 (view)

[aggregate, zip] How to aggregates elements from each of the iterables (ref)

x = [1, 2, 3]
y = [4, 5, 6]
zipped = zip(x, y)
list(zipped)
[(1, 4), (2, 5), (3, 6)]

[set, value, multi] How to setup a valutes to multiple indexs in array (ref)

# equivalent to a[0:6:2] = -1000;

# from start to position 6, exclusive, set every 2nd element to -1000

a[:6:2] = -1000

[reverse] How to reverse the permutation of the value of array (ref)

a[ : :-1]

[iterate] How to iterate all members of array (ref)

for i in a:

... print(i**(1/3.))

[slice, subarray, strip, 1d] How to strip a 4-element 1D subarray from an array -- data_1D(0: 4) (view)

list = [1, 2, 3,4, 5, 6, 7, 8, 9, 10]

print(list) # [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

print(list[0:5]) # [1, 2, 3, 4, 5]

print(list[1:6]) # [2, 3, 4, 5, 6]

[remove, 2d] 如何修改 2D array, 移除後面 row, 只留下 row 0 (ref)

# [[1 2 3 4 5]

[2 3 4 5 6]]

[subarray, strip, 1d] How to strip the last elemnt from an array? -- [:-1] (view)

word[:2] # character from the beginning to position 2 (excluded)

=> 'Py'

word[4:] # characters from position 4 (included) to the end

=> 'On'

[subarray, strip] How to extract elements from the begining to the position N (excluded) (view)

s = wave.ys[:N]

[subarray, strip] How to extract elements from the position N (include) to End (view)

s = wave.ys[N:]

[col] How to grab an entire column, column vector -- Col0= Array2D[:,0] (view)

[col] How to get the column vector from 2D numpy array (view)

[col, 20] How to get the column list from a 2D array (view)

x = points_2d[:,0] # retrive the column list from column 0

y = points-2d[:,1] # retrive the column list from column 1

[2d, column] How to get the column list from a 2D array (view)
[array, row] How to get the row vector from 2D array -- points_2d[0, :]

把 : 當 * 理解就可以了.

a = array([[1, 2, 3],

[4, 5, 6]])

>>> a[0,:]

array([1, 2, 3])

[array, transpose] How to transpose an 2d array (view)

array_2d = numpy.array([x, y])

result_2d = array_2d.transpose()

[mean] How to get the mean for a giving array (view)

[array, sum of square] 用內積來做串列的平方和 (view)
[array, rms] How to calcuating the power value of a signal, python (view)
[array, normalized, 單邊 feature scaling] How to normalize a wave array so the maximum amplitude is +amp or -amp (view)
[array, normalized, standard score, zscore] How to normalized a wave array using standard score way (view)

[two array, max difference] How to get the maximun difference value for giving two arrays

Code

A = [1, 2, 3]

A = np.asarray(A)

B = [2, 10, 2]

B = np.asarray(B)

value = max(abs(A - B))

print('A - B = ', A - B)

print('value = ', value)

Result

A - B = [-1 -8 1]

value = 8

[average] How to calcuate the avg value by specified index

import numpy as np

data = [2, 10, 2, 10, 2, 10]

data = np.asarray(data)

index = [0, 2, 4]

avg = np.mean(data[index])

print('calcuating the avg value by giving a index array from data = ', avg)

[length, numpy] How to get the length for a given numpy array -- var.size
[array, max] How to get maximum value for a given array (ref)

max = np.amax(d_Nx1)

print('max = ', max)

[array, max] How to get the index of the max for a given array (ref)

index_max = np.argmax(data_1xN)

[array, majority] How to get the majority value for a given array (view)

hisList = [(2, 5), (3, 5), (4, 4), (1, 3), (5, 3), (6, 2)]

maxHisValue = 2 , count = 5

[array, outlier] How to remove the outlier (view)

arr = [9.31509847,5.5, 7.42095722,5.12572001, 4.52767176, 2.31286147, 7.48680971, 5.3547486, 32.2906815]

final_list = remove_outlier(arr)

print('final_list = ', final_list)

[array/list, if-where] get a list from an another list or dictionary where condition is true

test_vector = [

{'grp': 0, 'type': 0, 'desc': 'TP Test'}

{'grp': 0, 'type': 1, 'desc': 'FP Test'},

]

grp = 0

test_vector_grp = [tu for tu in test_vector if (tu == grp)]

print('test_vector_grp = ', test_vector_grp)

10.21. List comprehension

[index, enumerate] enumerate the list element

header = ['str1', 'str2', 'str3']

for col_id, col_name in enumerate(header):

[index, select, array] 回傳 array values by given an index array

near_list = list(x for x in list_B if abs(x-loc) < th)

height_list = list(list_Y[int(x)] for x in near_list) # 回傳 list_Y[near_list]

[array, condition] 從一個 array 中, 取出符合條件的數值, 建立另一個 array

# 把距離 loc 範圍內的所有點全部找出來

loc

near_list = list(x for x in list_B if abs(x-loc) < th)

[create, array] 有幾個成員, 就有幾個數值

data_mxn =[data1, data2, data3, data4, data5]

sampling_mxn = [500 for y in range(len(data_mxn))]

[批次取值] 取出那些大於 value 陣列

lst_data = [1, 2, 3, 5]

value =3

lst_data_new = np.asarray(lst_data)[(lst_data > np.int64(value))]

print(lst_data_new)

Output

[5]

[match] 取出 list 中, match 的項目

'''

mylist = [

{'CAL20110', 1, 2, 3,4},

{'CAL20112', 1, 2, 3,4},

{''ANE20001'', 1, 2, 3,4},

....

'''

# 取出 match 的成員

matched_row = [row for row in mylist if row[0] == 'ANE20001']

[list, interval] How to get the interval list for given a number list

Code

a = [1, 2, 3, 5, 7, 8, 11, 23]

a = list(zip(a[:-1], a[1:]))

y = [x[1]-x[0] for x in a]

Output
[1, 1, 2, 2, 1, 3, 12]

[array, diff] How to calcuate different betwen elements in array (ref)

>>> t

[1, 3, 6]

>>> [j-i for i, j in zip(t[:-1], t[1:])]

[2, 3]

[max, lambda] Find max of element of list of dictionaries

max_dict_obj = max(lst_dict_obj, key=lambda x:abs(x['data']))

[min, lambda] Customizde min

V_min = min((x for x in list_B if x >= loc) , key=lambda x: x-loc))

[nearest, min, lambda] 給一個 list B, 從 list A 中, 取出跟 list B 最近的那些點

def get_nearest(list_A, list_B):

list_loc_nearest = []

for loc in list_A:

list_loc_nearest.append(min((x for x in list_B if x >= loc) , key=lambda x: x-loc))

#list_loc_nearest.append(min(list_B, key=lambda x:abs(x-loc)))

'''

print('list_loc_r = ', list_A)

print('list_loc_peak = ', list_B)

print('list_loc_nearest = ', list_loc_nearest)

'''

return np.asarray(list_loc_nearest)

[median] Get the median value from lists of dictionaries with the key

Key

np.median([x['data'] for x in lst_dictA])

Code

import numpy as np

lst_dictA = [ {'data': 1}, {'data': 2}, {'data': 3}]

np.median([x['data'] for x in lst_dictA])

Result

[avg ] 計算那些 valid = 1 的那些數值平均值

# 計算那些 myarray[0] = 1 對應的那些數值的平均值

myarray = [ [1, 0, 1],

[2, 3, 4]]

list_vk = [*zip(*myarray)] # 轉置成 [1, 2], [0, 3], [1, 4]

>>>

[(1, 2), (0, 3), (1, 4)]

#列出 x_pair[0] == 1 的那些 vk[1]

list_valid_key = list(vk[1] for vk in list_vk if vk[0] == 1)

>>>

[2, 4]

sum(list_valid_key)/len(list_valid_key)

def get_avg(list_valid_key):

list_vk = [*zip(*list_valid_key)]

list_valid_key = list(vk[1] for vk in list_vk if vk[0] == 1)

if len(list_valid_key) > 0:

return 1, sum(list_valid_key)/len(list_valid_key)

else:

return 0, 0

[批次改值] 把陣列中偶數的變成 1, 奇數變成 0 (ref)

# key: 在 for 前面, 加 if-else 條件

my_list = [1, 2, 3, 4, 5, 6, 7, 8]

new_list = [1 if (x%2)==0 else 0 for x in my_list] # get odd/event list

Output

new_list = [0, 1, 0, 1, 0, 1, 0, 1]

[批次改值] 把陣列中, 數值 > 5 的保留, 其餘設定變成 0 (ref)

# key: 在 for 前面, 加 if-else 條件

my_list = [1, 2, 3, 4, 5, 6, 7, 8]

new_list = [x if x > 5 else 0 for x in my_list]

Output

new_list = [0, 0, 0, 0, 0, 6, 7, 8]

[filter] 把 list 中, 內容在 a, b 之間的數值, 挑出來

# key: 在 for 後面, 加 if 條件

my_list = [1, 2, 3, 4, 5, 6, 7, 8]

new_list = [x for x in my_list if x > 5]

Output

new_list = [6, 7, 8]

[nearest] How to get the neaset value list for giving a value list (ref)

Key:

# 從 list_B 傳回與 mydata 最接近的數字

min(list_B, key=lambda x:abs(x-mydata)

def test_get_nearest():

list_A = [10, 20, 30, 40]

list_B = [11, 22, 33, 44]

get_nearest(list_A, list_B) # output: [11, 22, 33, 44]

# 從 list B 中, 找出最接近 list A 的數值串列

def get_nearest(list_A, list_B):

list_loc_nearest = []

for loc in list_A:

list_loc_nearest.append(min(list_B, key=lambda x:abs(x-loc)))

print('list_loc_nearest = ', list_loc_nearest)

def test_get_nearest():

list_loc_r = [10, 20, 30, 40]

list_loc_peak = [11, 22, 33, 44]

get_nearest(list_loc_r, list_loc_peak) # output: [11, 22, 33, 44]

10.22. Generator

[iterable, yield] What is yield in Python (ref)

Iteratorable datatype 可以讓我們存取資料, 可是這些資料都必須放在記憶體中, 這是一個問題. 如果我們自訂的大批複合型資料集合體, 也想擁有讓別人 for each 等 iteration 功能, 就直接使用 yield 建立一個集合體.
[建立資料集合] 用 yield 建立一個 iterable data type, 存放一堆可以 iterate 的資料集合

def createGenerator():

mylist = range(3)

for i in mylist:

yield i*i

然後你就可以存取

mygenerator = createGenerator() # create a generator

for i in mygenerator:

print(i)

10.23. Serial

[array2serail] How to conver array to serial (view)

ps = pd.Series(data)

[rs232] How to access RS232

10.24. Plot

[data] 如何選擇不同的圖來分析你的資料 (ref)

[plt, site] matplotlib official page (view) (installation) (plotfile_demo)

[plt, title] How to setup the window title

plt.plot(lst_st_amp)

plt.gcf().canvas.set_window_title('mytitle')

plt.show()

[plt, plt] How to show the current figurat at a specified location (ref)

# Quick
draw.move_figure(plt.gcf(), 500, 0)

# Function

def move_figure(cls, f, x, y):

import matplotlib

"""Move figure's upper left corner to pixel (x, y)"""

backend = matplotlib.get_backend()

if backend == 'TkAgg':

f.canvas.manager.window.wm_geometry("+%d+%d" % (x, y))

elif backend == 'WXAgg':

f.canvas.manager.window.SetPosition((x, y))

else:

# This works for QT and GTK

# You can also use window.setGeometry

f.canvas.manager.window.move(x, y)

# Usage

from draw_lib2 import DrawClass as draw

import matplotlib.pylab as plt

mark_size = 6

draw.draw_line(signals2, plt.gca(), title, 500)

draw.move_figure(plt.gcf(), 500, 0)

plt.show()

[plt, zoom] How to perset the zoom for the plt view

ax = plt.gca()

ax.set(xlim=(0, 10000), ylim=(-4000, 4000))

[matplotlib] The matplotlib homepage (ref) the artist api (ref)

[matplotlib, version] How to check the matplotlib version (view)

[matplotlib, animation] A tutorial (ref)
[matplotlib, upgrade] How to upgrade matplotlib (ref)
[plt, loc] How to move the plot to the location

import matplotlib.pyplot as plt

plt.gcf().canvas.manager.window.move(x, y)

plt.show()

Ugly

import matplotlib.pyplot as plt

from pylab import get_current_fig_manager

manager = get_current_fig_manager()

manager.window.move(500, 0)

plt.show()

[pandas] http://pandas.pydata.org/ (ref)

[plot.ly] the official site (ref)
[seaborn] the tutorial site (ref)
[ipython, jupyter] iPhython Node Sample (ref)

[color] Python color map (ref)

[mat, demo] Lots of plot examples (ref)
[progress bar] How to show the progress -- tqdm (ref) (doc)

Install

conda install -c conda-forge tqdm

Code

from tqdm import tqdm

With tqdm(total=100) as pbar
pbar.set_description('Processing')

for i in range(100):
pbar.update(1)

Result

[draw, line] How to draw a line

Key message:

import matplotlib.pyplot as plt

plt.plot(1 * data1)

plt.show()

Ex1:

import matplotlib.pyplot as plt

data1 = [...]

data2 = [...]

data3 = [...]

plt.plot(1 * data1)

plt.plot(2 * data2)

plt.plot(3 * data3)

plt.show()

Result

plt.plot(data1)

plt.plot(data2)

plt.plot(data3)

[draw, text] How to draw text (ref)

ax = fig.add_subplot(row_num_ax = 2, 1, 1)

text = ax.text(0,0, 'hello world', size=size_text)

bb = text.get_window_extent(fig1.canvas.get_renderer())

height = bb.height

[pdf, read] How to extract text from pdf

https://www.blog.pythonlibrary.org/2018/06/07/an-intro-to-pypdf2/

from PyPDF2 import PdfFileReader

def get_info(path):

with open(path, 'rb') as f:

pdf = PdfFileReader(f)

info = pdf.getDocumentInfo()

number_of_pages = pdf.getNumPages()

print(info)

author = info.author

creator = info.creator

producer = info.producer

subject = info.subject

title = info.title

def text_extractor(path):

with open(path, 'rb') as f:

pdf = PdfFileReader(f)

# get the first page

page = pdf.getPage(16)

print(page)

print('Page type: {}'.format(str(type(page))))

text = page.extractText()

print(text)

if __name__ == '__main__':

path = 'sample.pdf'

get_info(path)

text_extractor(path)

[pdf, plot] How to plot your data into pdf file (view)

#plt.show()

print('save to ', pdffile)

plt.savefig(pdffile)

觀念: 只要設定 plt.subplot(211) 就可以在任何地方畫圖

import matplotlib

matplotlib.use('PDF')

import matplotlib.pyplot as plt

from matplotlib.backends.backend_pdf import PdfPages

def drawLine(my_data, color='b'):

N = len(my_data)

t = np.linspace(0, N, N)

plt.plot(t, my_data, color)

return plt

def main():

# plot:

plt.figure(1, (8, 4))

with PdfPages('mypdf.pdf') as pdf:

plt.subplot(211) # for a grid 2x1 figure, draw on the index 1 axis

drawLine(df['I'])

plt.subplot(212) # for a grid 2x1 figure, draw the line on the index 2 axis

drawLine(df['II'])

pdf.savefig()

plt.close()

[pdf, title] How to control title's location

ax.set_title('I', y=0)

[plt, x, y] How to draw a (x, y) data point (ref)

X = list()

Y = list()

for record in lst_dict_stm_record:

print(record)

X.append(record['int_ref_in_uv'])

Y.append(record['int_ref_minus_algo'])

print('-- plot --')

import matplotlib.pyplot as plt

fig1 = plt.figure()

ax = plt.gca()

ax.plot(X, Y, 'bo')

plt.show()

[line, data] draw line for giving a serial data

FIG_X_inch = int(50)

FIG_Y_inch = int(2)

def draw_line(data, ax, color='black'):

N = len(data)

t = np.linspace(0, N, N)

ax.plot(t, data, color)

def draw_line_quick(data, color='blue', figsize=(FIG_X_inch, FIG_Y_inch), pdffile='draw_line.pdf'):

fig1 = plt.figure(figsize=figsize)

gs = gridspec.GridSpec(nrows=1, ncols=1)

gs.update(hspace=0.2)

ax1 = fig1.add_subplot(gs[0, 0])

draw_line(data, ax1, color)

if pdffile != None:

with PdfPages(pdffile) as pdf:

print('save to ', pdffile)

pdf.savefig()

plt.close()

[line, 2point] draw line for giving two points

def draw_line_2point_quick(data, p1, p2, color='red', pdffile='draw_line_2point.pdf'):

fig1 = plt.figure()

gs = gridspec.GridSpec(nrows=1, ncols=1)

gs.update(hspace=1)

ax = fig1.add_subplot(gs[0, 0])

draw_line(data,ax,color='blue') # draw data

ax.plot([p1[0], p2[0]], [p1[1], p2[1]], color=color)

if pdffile != None:

with PdfPages(pdffile) as pdf:

print('save to ', pdffile)

pdf.savefig()

plt.close()

[line, draw] How to draw a line by giving two points (view)

[line, dimension] How to show a 1D sequence and give show dimension (view)

[line, width] How to setup the line width

ax.plot(x, 'b', lw=1)

[3d, pixel] How to plot pixels in 3D (ref)
[marker] How to draw a cross marker on the figure (ref) (all valid markers)

colors = ('b', 'g', 'r', 'c', 'm', 'y', 'k')

ind = np.asarray([1, 100])

plt.plot(ind, my_data[ind], '+', color = 'red')

ind = np.asarray([1, 100])

plt.plot(ind, my_data[ind], 'r+')

'yo': yello circle markers

'bo': blue circle markers

[plot, text] How to plot a text on ax?

ax.text(x, y , text, bbox=dict(facecolor='red', alpha=1.0))

[marker, style] How many marker style

Reference: https://stackoverflow.com/questions/8409095/matplotlib-set-markers-for-individual-points-on-a-line

import matplotlib.pylab as plt

markers=['.',',','o','v','^','<','>','1','2','3','4','8','s','p','P','*','h','H','+','x','X','D','d','|','_']

descriptions=['point', 'pixel', 'circle', 'triangle_down', 'triangle_up','triangle_left', 'triangle_right', 'tri_down', 'tri_up', 'tri_left','tri_right', 'octagon', 'square', 'pentagon', 'plus (filled)','star', 'hexagon1', 'hexagon2', 'plus', 'x', 'x (filled)','diamond', 'thin_diamond', 'vline', 'hline']

x=[]

y=[]

for i in range(5):

for j in range(5):

x.append(i)

y.append(j)

plt.figure()

for i,j,m,l in zip(x,y,markers,descriptions):

plt.scatter(i,j,marker=m)

plt.text(i-0.15,j+0.15,s=m+' : '+l)

plt.axis([-0.1,4.8,-0.1,4.5])

plt.tight_layout()

plt.axis('off')

plt.show()

[marker, label, legend] How to plot the lengend for a marker? (ref)

X = list()

Y = list()

for record in lst_dict_stm_record:

print(record)

X.append(record['int_ref_in_uv'])

Y.append(record['int_ref_minus_algo'])

print('-- plot --')

import matplotlib.pyplot as plt

ax = plt.gca()

ax.plot(X, Y, '.', label='Difference')

plt.legend()

plt.show()

plt.plot([1,2,3], label="test1")

plt.plot([3,2,1], label="test2")

# Place a legend to the right of this smaller subplot.

plt.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)

[circle] How to draw circles (view)

[fig] How to get the current figure

plt.gcf()

[ax] How to get the current axis

plt.gca()

[ax, range] How to change the axes range (view)

[circle, examples] Lots of circle examples (ref)
[animation, move circle] How to move a circle (view)
[animation, move circle] How to move a circle around a center (view)

[animation, rain] How to use matplotlib animation (view)

[animation, examples] Lots of animation examles (ref)
[ax, clean] How to clean the screen -- gca(), ax.cla()
[ax, range] How to setup axis range -- ax.set_xlim(0,10), ax.set_ylim(0,10)
[opt, curve_fit] How to use curve_fit function to evalute a curve (view)

[date format] csv plot (view)

[subplot, y, x] 把圖畫在 x, y 位置上 (ref)

# plot related package

import matplotlib

matplotlib.use('PDF')

import matplotlib.pyplot as plt

from matplotlib.backends.backend_pdf import PdfPages

import matplotlib.gridspec as gridspec

def drawLine2(my_data, ax, color='r'):

N = len(my_data)

t = np.linspace(0, N, N)

ax.plot(t, my_data, color)

def test():

fig1 = plt.figure(1)

row_num = 9

col_num = 2

# 指定 figure 總共有 row_num, col_num

gs = gridspec.GridSpec(row_num, col_num)

gs.update(hspace=1) # unit: inch

with PdfPages(pdffile) as pdf:

#傳回一個剛剛加在 (0, 0) 位置的 axes

ax = fig1.add_subplot(gs[0, 0])

drawLine2(df['I'], ax) # 畫圖

# 傳回一個剛剛加在 (1, 0) 位置的 axes

ax = fig1.add_subplot(gs[1, 0])

drawLine2(df['II'], ax) # 畫圖

pdf.savefig()

plt.close()

test()

Result

[multi-axes] create axes, draw, and then add it to fig

最簡單的範例取得 figure 上的 ax list, 讓我們可以不斷的追加繪製圖形上去

關鍵

list_ax = fig.get_axes() # 回傳 axes list 以供之後要追加畫圖使用

'''

Usage:

data1 = ...

Data2 = ...

data_mxn = [data1, data2]

title_mxn = ['data1', data2']

# 第一次在 fig 1 上繪製訊號

fig1 = plt.figure(figsize=(FIG_X_inch, FIG_Y_inch))

list_ax = draw_line_mxn(fig1, data_mxn, title_mxn)

# 第二次在 fig 1 上位於 (1, 1) 的 ax 上, 再標記東西 (追加繪製)

_, array_loc = dict_ecg_feature['r_loc']

draw_loc(data_ecg, array_loc, '*', 5 , 'r', 'r_loc', list_ax[0])

'''

def draw_line_mxn(fig, data_nxm, title_nxm, color='blue'):

row = len(data_nxm)

y = 0

for data in data_nxm:

ax1 = fig.add_subplot(row, 1, y+1) # 建立 axes on fig

draw_line(data, ax1, title_nxm[y], color) # 在 axes 上畫圖

y = y + 1

list_ax = fig.get_axes() # 回傳 axes list 以供之後要追加畫圖使用

return list_ax

def draw_line(data, ax, title=None, color='black'):

N = len(data)

t = np.linspace(0, N, N)

ax.plot(t, data, color)

if title != None:

ax.set_title(title)

def draw_loc(data, loc_array, mark_style, mark_size, mark_color, label, ax):

data_array = np.asarray(data)

ax.plot(loc_array, data_array[loc_array.astype(int)], mark_style, color = mark_color, markersize=mark_size, label=label)

ax.legend(bbox_to_anchor=(1.05, 1), loc=2, borderaxespad=0.)

[multi-axes] How to create sub-plots share x-axis (ref)

fig, (ax0, ax1, ax2) = plt.subplots(nrows = 3, figsize = (15, 10), sharex = True)

[sine wave, pyplot] How to draw a sin wave with frequency f0 in N samples (view)

[distri, pyplot] How to draw the normal distribution in shape of N, (N, 1), (N, 4) samples (view)

A normal distribution in shape of (N, 4)

10.25. Process/Thread

[time] Calcuating the time for your code (ref)

import time

class Timer:

def __enter__(self):

self.start = time.clock()

return self

def __exit__(self, *args):

self.end = time.clock()

self.interval = self.end - self.start

with Timer() as t:

print('Your long run procedure')

print('Request took %.03f sec.' % t.interval)

[time, get] Get the time text

import time;

localtime = time.localtime(time.time())

name = 'result_' + str(localtime.tm_year) +'_' + str(localtime.tm_mon) + '_' + str(localtime.tm_mday) + '_' + str(localtime.tm_hour) + '_' + str(localtime.tm_min) + '_' + str(localtime.tm_sec) + '.pdf'

print('localtime = ' + localtime)

[subprocess, check_output] How to launch git-bash.exe to run script by checkout_out method

# Python Code

import subprocess import check_output

check_output(["C:\\Program Files\\Git\\git-bash.exe", "/c/Users/Jing/work/lab/update.bash"])

import subprocess import check_output

check_output(["C:\\Program Files\\Git\\git-bash.exe", "C:\\Users\\Jing\\work\\lab\\update.bash"])

#要執行的批次作業

# update.bash

-------

echo 'update source'

git pull

-------

[subprocess, shell, bash] How to calling an external command in Python -- subprocess (view)

from subprocess import check_output

def run_shell(lst_cmd):

try:

out = check_output(lst_cmd).decode('utf-8')

return 1, out, 'ok'

except Exception as e:

return 0, None, str(e)

# method 1

from subprocess import call

call(["ls", "-l"])

# method 2

import os

retvalue = os.system("some_command < input_file | another_command > output_file")

print retvalue

# ex2

lst_cmd = ['cd ' + st.db_root.as_posix(), 'epicmp -r e0105 -a atr st -S - -']

lng_cmd = ''

for cmd in lst_cmd:

lng_cmd = lng_cmd + cmd + ';'

import os

os.system(lng_cmd)

os.system vs. subprocess (ref)

The os.system(command) method executes the command in subshell.

The subprocess.Popen(command_args, …..) method executes the command args as new process (child process). You can send / receive data to stdout /stdin using communicate() of Popen object.

[process, exit] How to exit the program -- sys.exit(0) (ref)
[process, pool] How to create a multiprocess tasks to separatly core

import sys

from multiprocessing import Pool

class demo():

def __init__(self):

self.lst_file = ['100','101','103','105','106','108','109','111','112','113','114','115',

'116','117','118','119','121','122','123','124', '200', '201', '202', '203', '205', '207', '208', '209', '210', '212', '213', '214', '215',

'219', '220', '221', '222', '223', '228', '230', '231', '232', '233', '234']

def process(self, filename):

print('filename = ', filename)

def run_thread(self):

with Pool(5) as p:

p.map(self.process, self.lst_file)

if __name__ == '__main__':

app = demo()

app.run_thread()

[thread, sync] Avoiding random crashes when multithreading Qt (ref) (code)
[thread, pyqt5] Multithreading PyQt applications with QThreadPool (ref)
[thread, pyqt5] Long run task without impact GUI (view)
[thread] How to run a specified function every second in another thread(view)

10.26. Async

Trio (ref)

10.27. Google

[google, sheets] How do access google sheets (ref)

10.28. Others

[perf] Performance (ref)

def fun(x):

# long run

import timeit

print(timeit.timeit("fun()", setup="from __main__ import fun"))

x = np.random.randn(10000)

ind = detect_peaks(x)

print('Data with %d points and %d peaks\n' %(x.size, ind.size))

print('Performance (without the minimum peak distance parameter):')

print('detect_peaks(x)')

%timeit detect_peaks(x)

print('\nPerformance (using the minimum peak distance parameter):')

print('detect_peaks(x, mpd=10)')

%timeit detect_peaks(x, mpd=10)

[ctype] How to call c (ref)
[mail, attachment] How to get all attachments for a given email address (view)
[sys, exit] 回傳給自動測試 script 你的結果

if __name__ == "__main__":

if main():

sys.exit(0) # test success

else:

sys.exit(1) # test failure

[api, pyplot] Where is the pyplot api (ref)
[api, artists] Where is the artist api (ref)
[debug] How to debug the scipy

ipython --pylab
Print "patch.center=", patch.center

How to install Scipy (view)
LDAP (ref)
[git, tag] How to get the tag information from git (ref)

cmd = 'git describe --tags --match [0-9]*'.split()

try:

version = subprocess.check_output(cmd).decode().strip()

except subprocess.CalledProcessError:

print('Unable to get version number from git tags')

exit(1)

[check version, git, commit] How to check version? (ref)

def check_update():

import subprocess

try:

print('checking versin ...')

subprocess.check_output('git fetch'.split()) # update ref

local = subprocess.check_output('git rev-parse @'.split()) # return local HEAD id

remote = subprocess.check_output('git rev-parse @{u}'.split()) # return remote HEAD id

comm_base = subprocess.check_output('git merge-base @ @{u}'.split()) # return comm id

if local == remote:

desc = '版本檢查: 目前是最新版'

return 0, desc

elif local == comm_base:

desc = '版本檢查: 有新版本. 請執行 git pull 進行更新'

return 1, desc

elif remote == comm_base:

desc = '版本檢查: Need to push'

return 2, desc

else:

desc = '版本檢查: Diverged'

return 3, desc

except subprocess.CalledProcessError:

desc = '版本檢查: [Error] Unable to get git information'

return 4, desc

[process] How to launch a process (ref)

import subprocess

subprocess.check_output('git fetch'.split()) # update ref

[os, system] How to determine os?

Code

platform.system()

Result

Linux: Linux

Mac: Darwin

Windows: Windows

[os, Windows] How to determine Windows ?

@classmethod

def is_Windows(cls):

if 'nt' == os.name:

return 1

else:

return 0

[os, RaspberryPI] How to determine RaspberryPi

def is_Raspeberry(cls):

ok, out, msg = PosSystem.run_shell(['cat','/sys/firmware/devicetree/base/model'])

if not ok:

return 0, msg

if out is None or 'Raspberry' not in out:

return 0, 'Not Raspberry'

return 1, 'ok'

[os, execv] How to restart a Python script itself (ref)

Code

import sys

import os

import time

def main():

for i in range (0, 10):

print(i)

time.sleep(1)

# restart it self

print('\nRestart')

os.execv(sys.executable, ['python'] + sys.argv)

if __name__ == '__main__':

main()

Result

[timer] How to use timer

# 10 秒後, 顯示 ok

QtCore.QTimer.singleShot(10000, lambda : self.text_status.setText('ok'))

QtCore.QTimer.singleShot(200, self.check_version)

def check_version(self):

res, desc, local_id = check_update()

out = desc + ' 目前 id = ' + local_id.decode("utf-8").rstrip()

print(out)

self.text_status.setText(out)

[max, lambda] how to find the max value for a giving dictionary array

test_vector = [

{'grp': 0}, # total file: 25, current detect 21

{'grp': 2},

]

# 其中 x 是 iterating 過程中的 iterator, 計算最大值時,

# 使用自訂的 x['grp'] 代替 x

>>> max(test_vector, key=lambda x:x['grp'])

Output:

{'grp': 2}

>>> max(test_vector, key=lambda x:x['grp'])['grp']

Output

[max, dict] Get the dictionary with the max key value in a list of dictionary

Example 1

Code

lst_dict = list()

lst_dict.append({'age': 1, 'score':2})

lst_dict.append({'age': 2, 'score':1})

max_dict = max(lst_dict, key=lambda x:x['age'])

print(max_dict)

Output

{'age': 2, 'score': 1}

Example 2

Code

lst_dict = list()

lst_dict.append({'age': 1, 'score':2})

lst_dict.append({'age': 2, 'score':1})

max_dict = max(lst_dict, key=lambda x:x['score'])

print(max_dict)

Output

{'age': 1, 'score': 2}

[google, sheets] How to access google spreadsheet -- pygsheets (ref)
[google, sheets] How to access google spreadsheet -- gsheet(view) (ref)

Note:

安全性: secret.json 不能流出去, 所以應該要讓程式執行在 Google Cloud Function

[google, sheets, keys] How to create a new keys to the service account for the project

Step 1: Choose a service account

[IAM & admin] -> [Service accounts]: Choose one of the service account by email

Step 2: Download key

[Create KEY]

[gmail, send] How to send a gmail with Title (view) (ref)
[math] 四捨五入, 小數點後 (ref)

import decimal

decimal.getcontext().rounding = "ROUND_HALF_UP"

decimal.Decimal("3.545").quantize(decimal.Decimal("0.00"))

11. Image Processing

11.1. Library

Python Imaging Library (PIL)

11.2. FAQ

[install, opencv] How to install Anaconda with OpenCV (view)
[image, load & show] How to load and show an image (view)

[image, save] How to save an image to png file (view)

[image, plt, show] How to use matplot to show an image (view)

[image, colormap] How to use color map for an image data (view)

[convert, bgr -> rgb] How to conver the BGR image to RGB image (ref)
[camera, capture] How to capture a frame from default camera (view)

[video, playback] How to play a video (view)

[video, save, xvid] How to save a video in XVID (view)
[image, contour] How to get the contour from an image (view)

[image, contour, CHAIN_APPROX_NONE] How to use APPROX_NONE mode to get the contour from an image (view)

[image, contour, moment] How to get the all moments from an image (view)

[image, contour, features] How to get the contour features for an image (view)

[image, contour, approx] How to get an approximation contour for a giving contour (view)

[camera, calibration] How is the Camera calibration (draft)

12. Signal Processing

12.1. KB

示波器運作原理 (ref)
甚麼叫做設計系統? => 輸入 x[n] 已知, 輸出已知 y[n], 計算或產生 system

(Edit)

訊號處理算法

計算的數值範圍

[read] 資料讀進來後, 一律進行 normalize +Amp or -Amp (+1 or -1)
[計算] 在浮點數下進行計算 (移植才需要考慮 QF 形式)
[out] 一律 quanta 到 2^15-1 (ex: 以 16-bit 音訊)

移植 QF format 的作法

首先製作 float 數版本
一行替換後, 立刻比對

[q-format]

[operator] 請用人家的 + - * / 作法, 不要自己弄, 除非你很清楚你在做的東西
[migration] 從浮點數版本一行一行新增, 比對兩者數值是否一樣 (寫程式比對, 不一樣, 要跳出來看)
[精確度問題] 要考慮精確度遺失下, 造成的音質損失
[精確度問題] 面對精確度問題時, 應該要先變成大的, 然後再做運算. 這樣才不會有溢位的問題.

(long)x * y

[dsp, signal, sampling] 取樣理論 -- aliasing (ref) (blog)

Downsampling: 當輸入訊號的取樣頻率高於輸出訊號 1/2 時, 就需要在輸入訊號執行一個低通濾波器, 以消除 alias. Fc 為目標訊號 fs/2, 因為要處理前, 先把超過目標訊號的 frequency 做梳理截掉.

Aliasing can be a problem, however, when the input sampling frequency is greater than the output sampling frequency. In such cases, if the input signals contain frequency components at or above half of the output sampling frequency, the input signals should be low-pass filtered.

TI: 參考: https://www.ni.com/zh-tw/innovations/white-papers/06/acquiring-an-analog-signal--bandwidth--nyquist-sampling-theorem-.html

Downsampling: 當輸入訊號的取樣頻率高於輸出訊號 1/2 時, 就需要在輸入訊號執行一個低通濾波器, 以消除 alias. 因為高過 fs/2 的內容訊號將會被混疊, 產生雜訊 (see 圖8) Alias F1-F8 分散在10, 30, 40Hz, 污染原訊號. 所以要再輸入端, 先把原訊號超過 fs/2 的訊號先濾掉.

Upsampling: 當輸入訊號的取樣率低於輸出訊號時, Resampling noise 會發生在輸入訊號的取樣頻率一半的地方, 因此, 也要在輸出訊號前, 加一個低通濾波器以消除這些 noise. Fc 為原訊號 fs/2. 因為輸出訊號的內容不該多原訊號出來.

Conversely, if the output sampling frequency is substantially greater than the input sampling frequency, resampling noise introduced at frequencies in excess of half of the input sampling frequency can be removed by low-pass filtering the output signals.

[dsp, signal, sampling] How to build the secret rabbit library for resampling operation (view)
[signal, dc offset] How to remove DC offset signal (ref)

設計 highpass filter, 刪除 frequency 接近 0 的訊號成份

缺點: 方波的直流成份會被消除. 測試不會過.

設計 lowpass filter, 取得 frequency 接近 0 的訊號成份, 當作 dc offset value. 然後整個訊號減去它

[signal, kalman] (ref)

如何辨識一個訊號的亂度

Step 1: x=S(t), y=S(t+th),

Step 2: 然後在二維座標系統上, 畫出分佈點

Step 3: 分析分佈圖

範圍很小: 甚至是一點: 表示訊號 S 是很週期的訊號
範圍很大, 表示訊號 S 是很亂的訊號

12.2. Filter

[filter, design, tuto] A tutorial on adaptive filter (ref)

Linear system and signal, z-transform
Active control for noise cancelation

[filter]

Highpass: 0.05Hz, 0.08Hz, 0.1Hz
Lowpass: 30Hz, 40Hz, 45Hz, 70Hz, 75Hz, 100Hz, 150Hz, 200Hz
Bandstop: 50Hz, 60Hz
肌電濾波: 20Hz ~ 500Hz

[filter, IIR vs. FIR] What's different between IIR and FIR (view)
[filter, IIR, real-time] IIR low-pass, high-pass, and band-pass real-time process filters

Self-implementation (python&javascript) (c#)
C (ref)
C++ (ref)
Python (ref)
Java (ref)

[filter, notch, Q-factor] A Q-factor cauclator for notch filter (ref)

注意: Python iirnotch 的 Q-factor 參數: 計算方式是倒數, 即 W0/BW (參考文件為: BW/ W0)

[filter, lowpass, numpy] How to implement a lowpass filter in numpy (FIR) (ref)

Delay 的探討: 長度為 M 的 FIR filter delay 為 (N-1)/2, 故 N = 51 的 filter, delay 為 25. 若要補償 delay, 只要 shift 25 點即可.

[filter, lowpass, thinkdsp] How to implement a lowpass filter in ThinkDSP (view)

[filter, highpass, scipy] How to create a high-pass filter using scipy.signal (ref)

import numpy as np

import pandas as pd

from scipy import signal

import matplotlib.pyplot as plt

def sine_generator(fs, sinefreq, duration):

T = duration

nsamples = fs * T

w = 2. * np.pi * sinefreq

t_sine = np.linspace(0, T, nsamples, endpoint=False)

y_sine = np.sin(w * t_sine)

result = pd.DataFrame({

'data' : y_sine} ,index=t_sine)

return result

def butter_highpass(cutoff, fs, order=5):

nyq = 0.5 * fs

normal_cutoff = cutoff / nyq

b, a = signal.butter(order, normal_cutoff, btype='high', analog=False)

return b, a

def butter_highpass_filter(data, cutoff, fs, order=5):

b, a = butter_highpass(cutoff, fs, order=order)

y = signal.filtfilt(b, a, data)

return y

# usage

fps = 30

sine_fq = 10 #Hz

duration = 10 #seconds

sine_5Hz = sine_generator(fps,sine_fq,duration)

sine_fq = 1 #Hz

duration = 10 #seconds

sine_1Hz = sine_generator(fps,sine_fq,duration)

sine = sine_5Hz + sine_1Hz

filtered_sine = butter_highpass_filter(sine.data,10,fps)

plt.figure(figsize=(20,10))

plt.subplot(211)

plt.plot(range(len(sine)),sine)

plt.title('generated signal')

plt.subplot(212)

plt.plot(range(len(filtered_sine)),filtered_sine)

plt.title('filtered signal')

plt.show()

[filter, lowpass, scipy] How to implement a filter using scipy.signal (view) (view)

[filter, bandpass, hw] How to implement a bandpass filter using OP AMP (ref)

[filter, bandpass, ButterworthBandpass, scipy] How to use ButterworthBandpass (view)

[filter, bandpass, thinkdsp] How to implement a general bandpass filter (view)

[filter, B, D, W] What's frequency of B, D, W

B: 20 ~ 200Hz
D: 200 ~ 600Hz
W: low pass < 1KHz

[Mel-Filters] The Mel-Filters Analysis for a singal (ref. slides)

Human concentrates on only certain frequency components

[filtfilt, lfilter] What's different between lfilter and filtfilt (ref)

filtfilt: 不能用在 real-time filtering, 因為有 filtering backward in time 的 zero-phase 處理
lfilter: 可以是用 real-time filter, 但一定有 delay

12.3. Blind Source Separation

ICA (ref)

12.4. Noise

[noise, noise control]
[noise, noise management]
[noise, noise source]
[noise, basic] Distribution, Correlation, Relationship between power and frequency

Distribution: the distribution of a random signal (ex: Gaussian noise)
Correlation: is each value in the signal independent of the others, or are these dependencies between them? (ex: Brownian noise)

Relationship between Power and Fequency: (ex: pink noise)

the power is inversely related to frequency;

[noise, white noise] Noise with equal power at all frequencies

The integrated spectrum,

[noise, uniform] A noise signal that contains random values from a uniform distribution; that is, every value in the range is equally likely
[noise, uncorrelated noise] The samples are jointly independant of one another. Ex white noise. The values are independent; that is, knowing one value provides no information about the others
[noise, uncorrelated uniform noise] a.k.a. UU noise.

Waveform
Power Spectrum

[noise, brownian noise] each value is the sum of the previous value and a random “step”. The samples are dependant on the preceding value.

Wave form
How to create? generate uncorrelated random steps and then add them up.

[noise, ecg] ECG 訊號的雜訊總分類

ECG Filters: (Ref)

Baseline wonder removing (IIR)
Polynormial Filter
Powerline interference
Muscle noise filtering

[smooth] How to smooth a 1D signal (ref)

[noise, suppression] Noise Suppression using 1d media filter (ref)

from scipy.signal import medfilt

smooth = medfilt( array_data, 11)

[noise, suppression] Savitzky-Golay filter (blog)

[morphology, dilation] How to use ndimage grey morphology (view)

[noise, smooth]

[noise, filter] Removed the noise, Validation of Automated Arrhythmia Detection for Holter ECG (ref)

[noise, spline] Spline interpolation in 1-d: Procedural (ref)

[noise, level] Measure the noise level (ref)

STD

[noise, uncorrelated uniform, thinkdsp] How to create an uncorrelated uniform , UU noise (view)

Waveform: random

Power Spectrum : random

[noise, uncorrelated gaussian] How to create an Uncorrelated Gaussian Noise (view)

[noise, pink] How to create a pink noise (view)

[signal, filter] residual_analysis (ref)

找趨勢

12.5. I/O

[wav, file2wave, load, play, thinkdsp] How to load a wave file and play it using ThinkDSP (view)

[wav, file2data, load, play] How to load a wave file and play it (view)

[wav, file2data] How to load the wave signal from file, thinkdsp (view)

[wav, play] How to play a wave signal using thinkdsp module, thinkdsp (view)

[wav, save] How to save the wave signal to file, thinkdsp (view)

[wav, data2wave] How to save the wave signal to WAV file via wave package (ref)

from pathlib import Path

from io_lib import *

import wave, struct, math, random

from tqdm import tqdm

from db_EC57Class import db_EC57Class

class WaveUtility():

@classmethod

# Normalizes a wave array so the maximum amplitude is +amp or -amp.

def normalize(cls, ys, amp=1.0):

"""Normalizes a wave array so the maximum amplitude is +amp or -amp.

ys: wave array

amp: max amplitude (pos or neg) in result

returns: wave array

"""

high, low = abs(max(ys)), abs(min(ys))

return amp * ys / max(high, low)

@classmethod

def test_ecg2wave(cls):

home = str(Path.home()) #ex: /home/jing

ec57_db_root = Path(home+'/work/database_ec57')

int_ch = 0

db = db_EC57Class(ec57_db_root / 'edb', 'edb', 'e0103')

ok, fs, msg = db.get_fs()

ok, total, msg = db.get_sig_len()

total_in_ms = total/fs * 1000

ok, ecg_data, msg = db.get_ecg_data2(0, total_in_ms, int_ch)

if not ok:

msg = 'Error: ' + msg + '\nDetail:\n' + 'in function: get_ecg_data2'

print(msg)

return 0, msg

ecg_data_2 = resample_by_interpolation(ecg_data, fs, 500)

ecg_data_3 = WaveUtility.normalize(ecg_data_2, 32767)

lst_data = list()

lst_data.append(ecg_data_3)

WaveUtility.ecg2wave(lst_data)

print('done')

@classmethod

def ecg2wave(cls, lst_data):

for ecg_data in lst_data:

sampleRate = 500 # 44100.0 # hertz

obj = wave.open('sound.wav','w')

obj.setnchannels(1) # mono

obj.setsampwidth(2)

obj.setframerate(sampleRate)

with tqdm(total=len(ecg_data)) as pbar:

pbar.set_description('Processing...')

for i, data in enumerate(ecg_data):

pbar.update(1)

#print('.', end=''); sys.stdout.flush()

value = int(data)

data = struct.pack('<h', value)

obj.writeframesraw( data )

obj.close()

def main():

WaveUtility.test_ecg2wave()

if __name__ == '__main__':

main()

[wave, data2wave] How to write multi-channel data to wave file (view)

[wav, wave2data, read, nparray] How to read wave file to numpy array (view)

[wave, data2wave] How to create a wave object from nparray data (view)

wavObj = thinkdsp.Wave(ys = nparray, framerate = 8000)

wavObj.make_audio()

wavObj.write('my.wav')

[wave, wave2csv] How to save the wave object to csv file (view)
[wave, csv2wave] How to load a wave object from a csv file (view)
[wav, wave2Audio] How to create an audio object from a wave object (view)

audio = Audio(data=wave.ys, rate=wave.framerate)

[audio, data2Audio] How to create an audio object from data array (view)

audio = Audio(data=wave.ys, rate=wave.framerate)

[ecg, wave] How to show whdb ECG wave by thinkdsp (view)

12.6. Find Peak

[signal, peak, nparray, pickutils] How to find the peaks for a given np array data (view), [peak, detector] PickUtils (ref) (conda package) (doc)

[signal, peak, wav, peakutil] How to find the peaks for a given wave (view)

[peak, detector, pickutils] PickUtils (ref) (doc) (conda package)

[peak, detector] Detection of peaks in data** by Marcos Duarte (ref)(home)

[peak, analysis] 分析 peak 後, 策略取 peak (view)

Peak 可能一群落在同一個波組 => 在同一個山裡面的 peak, 取最高的那個
Peak 落在不同的波組 => 真正的山峰間的 peak

[peak] Removed the noise, Validation of Automated Arrhythmia Detection for Holter ECG (ref)

讓 threshold 隨著原始波型變化改變, 調整增加或減少.

應用場域

對於原始資料有山有谷的波型, 可以訂出一個動態 threshold, 找凸出物 boundary

Filter

Dynamic Threshold

Result

12.7. Others

[dsp, book] Hayes, Monson H., Statistical Digital Signal Processing and Modeling, John Wiley & Sons, 1996, 493–552.
[dsp, book] Haykin, Simon, Adaptive Filter Theory, Prentice-Hall, Inc., 1996
[dsp, book] Think DSP (ref) (chinese)

[dsp, book, api] ThinkDSP API (ref)
[dsp, lecture] DSP Lecture 1 (video)
[speech, course] CMU Speech Processing Course (ref)

[env, thinkdsp, env] How to setup the development environmnet for the book "Think DSP" (view)

[lib, analysis, LibROSA] The music and audio analysis library -- LibROSA, MFCC, Spectrum (ref) (ref2)

conda install -c conda-forge librosa

[lib, sound, python] Basic Sound Processing in Python (video)
[lib, dsp] Think DSP (ref)

[lib, dsp] Python Adaptive Signal Processing package (ref)

[lms, matlab, tutorial] Signal Enhancement Using LMS and Normalized LMS (ref)
[beamforming, doc] Using Acoustic Beamforming for Pass-By Noise Source Detection (ref)

Spatial resolution, dynamic resolution, cable length, acceptable frequency
Sound waves can be treated as planar, meaning that if the sound source were centered in front of the microphone array, it would reach all microphones at the same time.
Microphone array configuration

[wav, format] The PCM wave format (view)
[signal, cycle] What is cycle? --- a full repetition of the signal
[signal, period] What is period? --- the duration of each cycle
[signal, freq] What is frequency? -- the number of cycle per second
[signal, fft] 傅立葉轉換的原理 -- 把訊號貼到圓周上來尋找重複的 pattern, 當猜測訊號的 pattern 和待測訊號本身週期一致或有關係時 (即猜測剪下來的訊號長度與訊號本身週期一致時), 在圓這個模型所形成的圖形會 "對齊". 當猜測的頻率和真正訊號的頻率不同時, 就會散亂. 用重心位置來量化評估對齊和散亂的程度, 達到頻率預測的目的. (ref)

想出這樣的方式來自動找出頻率，真的很有創意，https://youtu.be/spUNpyF58BY

[signal, phase] What is the phase for a wave (view) -- Phase is the position of a point in time on a wavefrom cycle

phase = 2PI * freq * ts + offset

ys = amp * fun(phase)

[signal, phase] How to show a wave in different phrases (view)

[sampling rate] How to change the sampling rate

a = numpy.array([1,2,3,4,5,6,7,8,9,10])

factor = 1.5

x = map(int,numpy.round(numpy.arange(0,len(a),factor)))

sampled = a[x]

[music, pitch] What is pitch? --- a perceptual property to judge sounds as "higher" and "lower" in the sense associated with musical melodies. defines the specific frequencies of particular pitches. (ref)

(ref)
C4=261.63
D4=293.66
E4=329.63
F=293.66
G=392
A=440
B=493.88

[signal, pitch] How to create pitch signals (view)

[spectrum, integrated spectrum] is a function of frequency, f , that shows the cumulative power in the spectrum up to f

Purpose: we can see the relationship between power and frequency more clearly [ref: thinking in DSP charpter 4]

[signal, correlation] two variables correlation means that if you know that value of one, you have some infomation about the other.
[approximation, poly] How to use polynomial approximation (view)
[sound, pressure level, spl] A sound wave in a transmission medium causes a deviation (sound pressure, a dynamic pressure) in the local ambient pressure, a static pressure. It is usually expressed in db SPL. (ref)

silence;
audible sound;
atmospheric pressure;
sound pressure

[sound, sound pressure meter] A device that measure the presure level caused by sound.

[Ambient Microphone] A device used to pick up background sound in the conference room
[audio, acoustic duct model]
[audio, acoustic echo cancellation] 在免持聽筒的通訊環境中，從遠方傳來的語音經擴音器放出來後，原訊號會經由多重路徑的反射，進入近端的麥克風，再傳回遠端，造成遠端之使用者聽到自己的迴音，形成通話之困擾；迴音消除技術之功用即在於消除此聲學上反射造成之迴音，改善通訊品質. 做法: 是在近端的麥克風做訊號處理, 減去遠端的人聲音, 再傳出去. 遠端因此不會聽到自己的聲音 (ref: opensource)
[echo cancelation, psychoacoustic masking] 心理聲學遮蔽
[feedback, neutralization]
Pich shift

https://www.kvraudio.com/forum/viewtopic.php?p=6662670
[matlab] https://www.mathworks.com/examples/audio-system/mw/audio_product-audioPitchShifterExample-delay-based-pitch-shifter
[paper] http://dafx.labri.fr/main/papers/p007.pdf
http://iq12.com/old_blog/2009/08/25/real-time-pitch-shifting/

[anc, ambient noise sources]
[anc, reference mic] the measure the noise near the headset
[anc, error mic] perceive the error between the noise source and he inverse signal. Moniting the performance of the ANC system.
[anc, canceling loud speaker]
[anc, sampling delay] the delay caused by sampling rate
[mic, survey] Lots of microphone for recording environment sound (ref)

[mic, Omnidirectional] receiving signals from or transmitting in all directions.

WM-60A (ref), (freq. response)

[impulse response] a IRF of a dynamic system is its output when presented with a brief input signal, called an impluse (ref)
[snr] What's different between RMS and peak current value (view)
[snr, doc] A BLIND SIGNAL LOCALIZATION AND SNR ESTIMATION METHOD (ref)
[snr] How to calcuate the SNR value for a given signal and noise (view)

[signal, custom, class] How to create a class for customized signal (view)

[signal, custom, class] How to create a signal that frequence changes in time (view)

[signal, frame rate] How to get the framerate (view)
[signal, envelope, hilbert] How to get the Hilbert envelop for an AM module source (view)

[plot, color, thinkdsp] How to plot a line with color, thinkdsp -- wavevar.plot(color='#045a8d') or plot(color='0.7') (view)

[window, function] How to implement a moving window function (view)
[thinkdsp, wave] How to plot a wave object (view)

# 畫出波形檢視

wave1.segment(start=0, duration=1).plot() # 只看 1 秒鐘這範圍的波形 y

[wave, plot, pyplot] How to plot wave data using plot (view)

[plot, sign, sin, thinkdsp] How to plot a wave using thinkdsp module (view)

[signal, power spectrum, thinkdsp] How to get the power spectrum for a given signal (view)

[signal, power spectrum, scipy] How to get the power spectrum for a given signal (view)

[signal, power spectrum, tool] How to show the power spectrum from a wav file -- audacity

[signal, spectrum -> wav] How to convert the sepctrum to wave (view)

[signal, wave -> spectrogram, scipy] How to show the power spectrogram for a given wave (view)

[ecg, spectgrogram, scipy] 顯示 whdp ECG 搭配 specgrogram 一起看 (view)

[signal, wav -> spectrum, spectrogram, thinkdsp] How to show the power spectrum for a given wav using python (view)

[signal, spectrogram, tool, app] spek, ubuntu

[signal, envlope] How to create the Spectral envlop form for a given wave (view)
[signal, downsample, matlab] How to decrease the sampling rate (ref)
[audo] Plotting wave form and spectrogram

[wavelet] How to use wavelet decomposition to ECG

import sys

PACKAGE_PATH_THINKDSP = '/home/jing/work/emat_main/emat/ThinkDSP/code/'

sys.path.insert(0, PACKAGE_PATH_THINKDSP)

import thinkdsp

# ecg utility

PACKAGE_PATH_DRAW='/home/jing/work/emat_main/emat/'

sys.path.insert(0, PACKAGE_PATH_DRAW)

from utl_draw import *

import utl_draw

from utl_io import *

import pywt

import pywt.data

def reconstruction_plot(yyy, **kwargs):

"""Plot signal vector on x [0,1] independently of amount of values it contains."""

plt.plot(np.linspace(0, 1, len(yyy)), yyy, **kwargs)

def get_total_element(coeffs):

sum = 0

for coeff in coeffs:

sum += len(coeff)

return sum

def to_int(coeffs):

for coeff in coeffs:

i = 0

if coeff is None:

break

for data in coeff:

coeff[i] = int(data)

i += 1

return coeffs

def main():

bdatatype = 'wave'

# ECG file test

'''

data_ecg = get_ecg_binary_data_v2('../ecg_channel_1.raw')

sampling = 800

# data_ecg = data_ecg[0:2048]

'''

# audio file test

wav_heart = thinkdsp.read_wave('../audio_channel_2.wav')

data_ecg = wav_heart.ys

CUT_LEVEL = 6

# decomposition

w = pywt.Wavelet('sym5')

coeffs = pywt.wavedec(data_ecg, w, level=6)

print('w = ', w)

print('coeffs = ', coeffs)

print('len coeffs = ', len(coeffs))

print('total data = ', get_total_element(coeffs))

print('total data cut = ', get_total_element(coeffs[:-CUT_LEVEL]))

ratio = get_total_element(coeffs)/get_total_element(coeffs[:-CUT_LEVEL])

print('ratio = ', ratio)

#print('the latest level coeff = ', coeffs[:-4] + [None] * 1)

# reconstruction

data_ecg_full= pywt.waverec(coeffs, w)

data_ecg_leavingout = pywt.waverec(to_int(coeffs[:-CUT_LEVEL]) + [None] * CUT_LEVEL, w)

list_data_ecg = [data_ecg, data_ecg_leavingout]

lst_title = ['org', 'CUT high freq ' + str(CUT_LEVEL)]

bShow = True

if bShow:

for ecg in list_data_ecg:

plt.plot(ecg)

plt.legend(lst_title)

plt.text(0, 0, 'ratio = ' + str(ratio))

plt.show()

if bdatatype == 'wave':

wavObj = thinkdsp.Wave(ys = data_ecg_leavingout, framerate = 8000)

wavObj.write('result_cut'+ str(CUT_LEVEL) +'.wav')

if __name__ =='__main__':

main()

[Cepstral Analysis] The Cepstral Analysis for a signal (ref. slides)

Ocean wave analysis (view)

[benchmark, mse] How to calcuating the MSE (view)

[benchmark, power, rms] How to calcuating the power value of a signal (view)

[benchmark, calibration] 平均聲壓的測定方法_Measurement_Procedure -- CNS8466 (pdf)

聲壓變動檢驗的規格
放置 microphone 的數目
如何計算平均聲壓值
頻譜水平軸的刻度: 125Hz, 250Hz, ...
應該紀錄的項目
White noise 測試實驗設計

[matlab, signal, bandpass, filter, sos] How to design a bandpass filter by giving second-order section matrix values (view)

[matlab, signa, tool, filterdesigner] The matlab filter designer tool (ref)

13. Math

[linear algebra] 超棒的線性代數課程, 有豐富的 python 範例, 影片教學 (ref)
[log] What is the value of log(0) -- - (view)

[latex, tutorial] (ref)
[latex, linefeed] How to line feed in Latex -- \\
[lastex, blank[ How to instart a blank character -- \quad
[signal, autoregressive] the value of the noise at time t depends only on its previous values and on a random disturbance.
[signal, noisy signal] contains both the desired signal and an added noise component
[num, positive] Positive number (ref)

x > 0
A number is positive if it is greater than zero

[num, non-negative] Non-negtive number (ref)

x >= 0
A number is non-negative if it is greater than or equal to zero

[set, convex set] Convex Set (ref)

區域中的任兩點所形成的直線, 在那直線上的任一個點也屬於該集合
In Euclidean space, a convex set is the region such that, for every pair of points within the region, every point on the straight line segment that joins the pair of points is also within the region
(ok), (false)

[scale mapping] How to implement scale mapping (ref)

def LinearMap(aValue, aMin, aMax, bMin, bMax):

# function for linear mapping between two ranges

# inputs:

# aValue: the input value you want to map, range [aMin, aMax]

# bMin, bMax: the range of the resulting value

# output:

# the resulting value in range rout

# usage:

# v2 = LinearMap (0, -256, 255, -1, 1);

a = aMin;

b = aMax;

c = bMin;

d = bMax;

return ((c+d) + (d-c)*((2*aValue - (a+b))/(b-a)))/2;

Usage

v1 = 255

print('v1 = ', v1, ' v2 = ', LinearMap (v1, -256, 255, -1, 1))

v1 = -256

print('v1 = ', v1, ' v2 = ', LinearMap (v1, -256, 255, -1, 1))

v1 = 0

print('v1 = ', v1, ' v2 = ', LinearMap (v1, -256, 255, -1, 1))

v1 = 123

print('v1 = ', v1, ' v2 = ', LinearMap (v1, -256, 255, -1, 1))

Result

v1 = 255 v2 = 1.0

v1 = -256 v2 = -1.0

v1 = 0 v2 = 0.0019569471624266144

v1 = 123 v2 = 0.48336594911937375

[order, partial order] The partial order set (ref)

A set having comparable elements, where the relation "less than or equal" can be used
(ref)

[order, bound] What is the lower bound (ref)

if you have a partially ordered set K (a set having comparable elements, where the relation "less than or equal" can be used), the lower bound is an element of K which is less than or equal to every element of S.
S={2,4,8,12}, 1 is the lower bound of S

[relation, bound, infimum] What is the infimum

The maximum lower bound
S={2,4,8,12}, 2 is the infimum of S
You can find an infinity of lower bound, but there is only one infimum

[relation, bound, supremum] What is the supremum

The upper bound

[function, epigraph] Epigraph function (ref)

a function f : Rn→R is the set of points lying on or above its graph
(ref)
Think of it as filling it with water but the water cannot overflow so it adds up vertically when it reaches the limits of the function

[function, convex, def] Convex function (ref)

A function is convex if you can trace a line between two of its points without crossing the function line
,
Convex function has global minimum

[function, convex, check] How to check a function is convex (ref)

By checking the Hessian matrix whether a positive semi-definite matrix

[function, concave, def] A concave function (ref)

A function f is concave if −f is convex

[function, opt, def] What is optimazation problem (ref)

Basically, it says in mathematical notation that the optimal value is the infimum of f(x) with all the constraints respected

[function, opt, objective] What is the objective function

f is called the objective function (it is also sometimes called the cost function). By changing x (the optimization variable) we wish to find a value x∗ for which f is at its minimum.
An optimization problem is typically written:
(ref)

[function, opt, constraint, equ] What is the Equality Constraint

(ref)

[function, opt, constraint, inequ] What is the Inequqality Constraint?

(ref)

[function, opt, constraint, combine] What is the Combiation Constraint?

[function, opt, feasible region] What is the feasible region (ref)

A feasible region, feasible set, search space, or solution space is the set of all possible points (sets of values of the choice variables) of an optimization problem that satisfy the problem's constraints, potentially including inequalities, equalities, and integer constraints.

[function, opt, Lagrange multipliers] What is the Lagrange Multipliers (ref)

Idea:

在限制條件 g(x, y) 下, 尋找 objective function f(x, y) 的最低值, Lagrange 發現, 最低值會出現在沿著 g(x, y) 的途徑上, 兩個function 的 gradient 方向平行的地方.

Is a strategy for finding the local maxima and minima of a function subject to equality constraints. (Wikipedia)
What did Lagrange find? He found that the minimum of f(x,y) under the constraint g(x,y)=0 is obtained when their gradients point in the same direction
圓是 function = x^2 + y^2, 顏色越深數值越低. 右上角的直線是 x + y - 1 = 0 的 constraint. Lagrange 發現在最小值會出現在兩個圖的 gradient 方向一致的地方.
We can see that the is only one point where two vectors point in the same direction: it is the minimum of the objective function, under the constraint.
手動模擬 Lagrange: 先從最上面的那個點開始, 沿著藍色的 constraint 線, 如果往左, 你會發現 objective function 的 gradient 方向漸漸往左 (這不是好的方向), 沿著藍線往右移動, objective function 的 gradient 方向漸漸與白色的 constraint 方向差距縮小. 但是若一直往右移動, 會發現 objective function 的 gradient 又開始與 constraint 方向越差越遠. 當你到達一個點 gradient 差距最小的地方, 就是最小點. (ref)
Lagrange 基本上就是在找 (objective function 與 constraint function 相同方向或平行方向的 (x, y) 位置) (ref). λ 是 Lagrante multiplier, 為了處理兩個 gradient functions 不同大小的問題.

[opt, lagrange, paper] Lagrange Multipliers Tutorial in the Context of Support Vector Machines (ref)

[gradient] can be visualized as a vector field, with the arrow pointing in the direction where the function is increasing (function 增加的方向) (ref)
[linear algebra, vector] a vector has a magnitude and a direction
[linear algebra, normal, def] Normal vector

A vector that is orthogonal (perpendicular) to the plane (ref)

[linear algebra, hyperplane, normal] hyperplane 方程式 Wx +b 中, 為何 W 向量永遠 normal 於該 hyperplane?

因為這個 hyperplane 就是藉由這個 normal vector W 所定義出來的 (ref)

[linear algebra, orthognal] How to calcuate the orthogonal projection of a vector x onto y -- z=(u⋅x)u (view)

[linear algebra, positive definite] What is positive definite matrix (view) (ref)

The symmetric matrix A is positive definite.
All eigenvalues of A are positive.
All the leading principal minors of A are positive.
There exists nonsingular square matrix B such that A=B⊺B

[linear algebra, positive semi-definite] What is positive semi-definite (view) (ref)

Usage: 評估一個多變數 function 的一階微分 = 0 的位置, 為全域最小值. 只要判斷它的 Hessian 矩陣是否為 positive semidefinite 即可.
The symmetric matrix A is positive semi-definite.
All eigenvalues of A are non-negative.
All the principal minors of A are nonnegative.
There exists B such that A=B⊺B

[linear equ] 計算直線方程式 -- 兩點 (view)

[distance, line, point] The shortest distance between a point and a line (video)

Python (ref) (ref2)

[degree] How to get degree from a set of points

import math

p30 = np.poly1d(np.polyfit(x2, data2, 1))

print('p30 = ', p30)

print('p30[1] = ', p30[1])

dx=1000

dy=p30(1000) - p30(0)

float_slope = np.degrees(math.atan2(dy, dx))

14. Authentication

LDAP (ref)

15. Debug

[debug] How to debug the scipy

ipython --pylab
Print "patch.center=", patch.center

[vscode, break] How to setup a conditional break for a python code (ref)

Expression condition: The breakpoint will be hit whenever the expression evaluates to true.

16. PyQt5

[tool, designer] How to launch the QT designer?

Ubuntu: designer
Mac: open - a Designer

[pyqt5, version] How to check the pyqt version

from PyQt5.Qt import PYQT_VERSION_STR

print("PyQt version:", PYQT_VERSION_STR)

16.1. Signal & Slot

[signal, tutorial] qt4, Development/Tutorials/Python introduction to signals and slots (ref)
[signal, sender] How to get the signal emitter

self.sender0

16.2. Layout

[QLayout, contructor] What's different between QVBoxLayout() and QVBoxLayout(self)?

定義

QVBoxLayout(widet)

Constructs a new top-level horizontal box with parent parent.

QVBoxLayout():

Constructs a new horizontal box. You must add it to another layout.

用法:

套疊 layout 的情況: 如果你的 layout 是要放在別的 layout 裡面, 那你應該使用 QVBoxLayout(). 然後在外面使用 addLayout 放進來, 例如 layout_top.addLayout(layout).

最外層 widget layout 的情況: 如果你的 layout 是提供給主要 application 的 widget 使用, 那麼你可以使用 QVBoxLayout(self) 或者是
layout = QVBoxLayout(); self.setLayout(layout)

[QLayout, align] How to align the component from top (ref)

from PyQt5.QtWidgets import QVBoxLayout

from PyQt5.QtCore import Qt

layout = QVBoxLayout()

layout.setAlignment(Qt.AlignTop)

[QLayout, replace] How to replace a widget A to widget B in a layout?

Code:

item = layout.replaceWidget(A, B)

layout.removeItem(item)

A.setParent(None) # remove widget A, if you don't do this, you'll see the messup UI with A and B

Q: How many methods to remove a QWidget widget?

widget.setParent(None)
widget.deleteLater(); widget = None
import sip; sip.delete(widget); widget = None

[QLayout, change] How to replace the old layout for the widget (ref)

if self.layout():

QWidget().setLayout(self.layout()) # 1

self.layout = The_New_Layout(self)

[QLayout, update] How to notify the layout manager to update the geometric (ref) (setFixedSize)

self.updateGeometry()

注意:

當你的 Widget 執行 self.move 移動時, 呼叫 self.updateGeometry() 會呼叫管理它的 layout manager 出來進行重新排版的工作.

但是這時若你的 Widget 使用了 setFixedSize(), 那這樣呼叫 self.updateGeometry() 會失效

[QLayout, area] Get the content area for a layout manager

width = self.geometry().width() # 取得目前管理的區域大小

if width == 0:

return 0

width = self.contentsRect().width()

if width <= 0:

return 0

[QLayout, space] How to setup the margins and space between managed widgets (ref)

Definition

Spacing: the spacing between widgets inside the layout

Margin: the margins used around the layout

Code

layout.setSpacing(0)

layout.setContentsMargins(0,0,0,0)

[QGridLayout] How to use QGridLayout to manage components

self.layout = QGridLayout(self)

self.layout.addWidget(self.categoryWidget, row=0, col=0)

self.layout.addWidget(self.itemWidget, 2, 0)

self.setLayout(self.layout)

[QGridLayout, ratio] 如何建立兩個元件, 一個佔自己容器 1/4 高度, 另一個佔自己容器 3/4 高度.

Key

1. QGridLayout

2. setup addWidget

3. setup SizePolicy

Code

self.categoryWidget = QPushButton('1/4')

self.itemWidget = QPushButton('3/4')

self.layout = QGridLayout(self)

self.layout.addWidget(self.categoryWidget, 0, 0, 1, 0) # 從 row 0 開始, 佔據 1 格

self.layout.addWidget(self.itemWidget, 1, 0, 3, 0) # 從 row 1 開始, 佔據 3 格

self.setLayout(self.layout)

self.categoryWidget.setSizePolicy(QSizePolicy.Preferred, QSizePolicy.Minimum)

self.itemWidget.setSizePolicy(QSizePolicy.Preferred, QSizePolicy.Expanding)

Result

16.3. Style, Skin

[official] all compoent example (ref)
[comment] What is the comment for the sheetstyle?

/* */

[set] How to set the style for all QTableWidget objects

app = QApplication(sys.argv)

app.setStyleSheet("QTableWidget { background-color: yellow }" )

[load] How to load the style from file

app = QApplication(sys.argv)

styleFile = Path('./res/darkorange.stylesheet')

strstyle = str(open(styleFile,"r").read())

app.setStyleSheet(strstyle)

16.4. Drag & Drop

[drag, button] How to implement a drag move button

Ref, work

[drag, resize, button, layout] How to implement a drage movable button by customized layout (view)

[drag, label] How to implement a drag&drop function between application (ref)

[drag, button]

# source

def mousePressEvent(self, event):

if event.button() == QtCore.Qt.RightButton:

mimeData = QMimeData()

mimeData.setText(self.text())

# mimeData.setData('mybutton/text', self.text())

drag = QDrag(self)

drag.setMimeData(mimeData)

drag.setHotSpot(event.pos() - self.rect().topLeft())

dropAction = drag.exec_(Qt.MoveAction)

# drop target

[drag, button] (video) (ref)

[drag, button, QtQuick, GridView] (ref)

16.5. Cursor

[cursor, get] How to get QWidget cursor? (ref)

self.cursor()

[cursor, set] How to set the QCursor?

self.setCursor(Qt.SizeFDiagCursor)

16.6. Digital

[digital] 最簡單的方式顯示數字 (blog)

關鍵片段:

QLCDNumber class

# reference

# https://stackoverflow.com/questions/33843652/update-lcd-number-countdown

import time

from threading import Thread

from PyQt5.QtWidgets import (QApplication, QWidget, QMainWindow, QLCDNumber)

class Window(QMainWindow):

def __init__(self):

QMainWindow.__init__(self)

self.lcdnumber = QLCDNumber(self)

self.resize(400, 400)

t = Thread(target=self._countdown)

t.start()

def _countdown(self):

x = 10

for i in range(x,0,-1):

time.sleep(1)

self.lcdnumber.display(i)

if __name__ == "__main__":

app = QApplication([])

window = Window()

window.show()

app.exec_()

[digital, size] 最簡單的方式顯示數字 (view) (Ref)

from threading import Thread

import time

from PyQt5.QtWidgets import (QWidget, QLCDNumber, QVBoxLayout, QPushButton)

class Screen(QWidget):

def __init__(self, parent = None):

super(Screen, self).__init__(parent)

self.resize(1024, 768)

self.lcdnumber = QLCDNumber(self)

self.lcdnumber.setGeometry(0, 0, 1024, 768)

self.lcdnumber.setDigitCount(4)

self.lcdnumber.setMinimumHeight(100)

self.weight = 0.0

t = Thread(target=self._countdown)

t.start()

self.setLayout(QVBoxLayout())

def _countdown(self):

x = 100

for i in range(x,0,-1):

time.sleep(0.1)

self.__display()

self.weight = self.weight + 1

self.lcdnumber.display(self.weight)

def __display(self):

print('weight = ', self.weight)

16.7. Console

[console, curses] How to use curses (ref)(office)

from curses import wrapper, ascii

def main():

wrapper(draw)

def draw(screen):

screen.clear()

screen.addstr(0, 0, 'Press ESC to quit.')

while screen.getch() != ascii.ESC:

pass

def get_border(screen):

from collections import namedtuple

P = namedtuple('P', 'y x')

height, width = screen.getmaxyx()

return P(height-1, width-1)

if __name__ == '__main__':

main()

16.8. QWidget

[QWidget, title] How to get the Widget Text (ref)

wig.windowTitle()

[QWidget, size] How to get the size of the widget? (view)

width = self.geometry().width()

height = self.geometry().height()

widget = QWidget(this)

self.widget.frameSize().width()

[QWidget, size, set] How to set the maximun size for a QWidget instance?

from PyQt5.QtWidgets import QWidget

from PyQt5.QtCore import QSize

widget = QWidget()

widget.setMaximumSize(QSize(0,0))

[QWidget, size, max] How to show the maximum size of a QWidget instance

from PyQt5.QtWidgets import QWidget

class mywidget(QWidget):

def __init__(self, parent):

super().__init__(parent)

self.showMaximized()

[QWidget, event, resize] Handle the resize

wig.resizeEvent = self.on_resize

def on_resize(self, event):

print('on resize')

16.9. QPushButton

[QPushButton, customized] How to customized a QPushButton

class ItemButton(QPushButton):

def __init__(self, Text, parent=None):

super(ItemButton, self).__init__(Text, parent)

self.clicked.connect(self.on_click)

def on_click(self):

print('item_bt: ok, text, ', self.text())

Usage

self.bt = ItemButton('item_' + str(bt_index))

[QPushButton, image] How to create an image button (ref)

from PySide2.QtWidgets import QAbstractButton

from PySide2.QtGui import QPixmap, QPainter

class PicButton(QAbstractButton):

def __init__(self, pixmap, parent=None):

super(PicButton, self).__init__(parent)

self.pixmap = pixmap

print('__init__')

def paintEvent(self, event):

painter = QPainter(self)

painter.drawPixmap(event.rect(), self.pixmap)

print('paint')

def sizeHint(self):

print('sizeHint')

return self.pixmap.size()

class MyWidget(QtWidgets.QWidget):

def __init__(self):

# add button

self.bt_yushan = PicButton(QPixmap('./res/img/Yushan.png'))

class PicButton(QAbstractButton):

def __init__(self, pixmap, parent=None):

super(PicButton, self).__init__(parent)

self.pixmap = pixmap

self.enter = False

def paintEvent(self, event):

painter = QPainter(self)

painter.drawPixmap(event.rect(), self.pixmap)

if self.enter:

qp = QtGui.QPainter()

qp.begin(self)

self.drawRectangles(qp, event.rect())

qp.end()

def drawRectangles(self, qp, rec):

qp.setBrush(QtGui.QColor(000, 000, 255, 255/4)) # optical

qp.drawRect(rec.left(), rec.top(), rec.width(), rec.height())

def enterEvent(self, event):

self.enter = True

self.update()

def leaveEvent(self, event):

self.enter = False

self.update()

def sizeHint(self):

return self.pixmap.size()

[QPushButton, sender] How to get the calling object from clicked event emited?

Key:

self.sender()

Code:

bt = QPushButton('test)

bt.clicked.connect(self.digitClicked)

def digitClicked(self):

clickedButton = self.sender()

digitValue = int(clickedButton.text())

[QPushButton, style] How to setup QPushButton Style

button.setStyleSheet('QPushButton {font-size: 20px; background-color: yellow; color: red;}')

[QPushButton, shadow] How to setup a shadow effect for the QPushButton (ref)

from PyQt5.QtWidgets import QPushButton, QGraphicsDropShadowEffect

class myButton(QPushButton):

def __init__(self, Text, parent=None):

super(myButton, self).__init__(parent)

self.shadow = QGraphicsDropShadowEffect(self)

self.shadow.setBlurRadius(5)

self.setGraphicsEffect(self.shadow)

[QPushButton, text, html] How to set rich text button (ref)

class myButton(QButton):

def __init__(self, Text):

super(myButton, self).__init__(parent)

def setText(self, Text):

from PyQt5.QtGui import QTextDocument, QPixmap, QPainter, QIcon

from PyQt5.QtCore import QRectF

HtmlText = QTextDocument()

# HtmlText.setHtml(" {} ".format(Text))

HtmlText.setHtml(Text)

pixmap = QPixmap(HtmlText.size().width(), HtmlText.size().height())

pixmap.fill(Qt.transparent)

painter = QPainter(pixmap)

HtmlText.drawContents(painter, QRectF(pixmap.rect()))

myIcon = QIcon(pixmap)

self.setIcon(myIcon)

self.setIconSize(pixmap.rect().size())

painter.end()

# Usage

item_string = ' {} '.format('井民全')

item_string = item_string + ' {} {}'.format(12, 32)

bt1 = myButton()

bt1.setText(item_string)

[QPushButton, hover] How to get the mouse hover event for the QPushButton (ref)

class myButton(QPushButton):

def __init__(self, Text, parent=None):

super(myButton, self).__init__(parent)

self.setMouseTracking(True)

def mouseMoveEvent(self, event):

print('event.pos() = ', event.pos())

[QPushButton, paint] How to show a crossed-dash lines on button?

class myButton(QPushButton):

def __init__(self, Text, parent=None):

super().__init__(parent)

def paintEvent(self, event):

super().paintEvent(event)

qp = QPainter()

qp.begin(self)

#self.drawRectangles(qp, event.rect()) # given the button dimension

self.drawCross(qp, event.rect())

qp.end()

self.update() # 強迫重新繪製

def drawRectangles(self, qp, rec):

qp.setBrush(QColor(000, 000, 255))

qp.drawRect(rec.left(), rec.top(), rec.width(), rec.height())

def drawCross(self, qp, rec):

qp.setPen(QPen(Qt.black, 1, Qt.DashDotLine, Qt.RoundCap))

qp.drawLine(rec.left(), rec.top(), rec.right(), rec.bottom())

qp.drawLine(rec.right(), rec.top(), rec.left(), rec.bottom())

[QPushButton, paint] How to draw a rectangle on button (ref)

from PyQt5.QtGui import QPainter, QColor, QFont, QColor

def paintEvent(self, event):

painter = QPainter(self)

painter.drawPixmap(event.rect(), self.pixmap)

if self.enter:

qp = QPainter()

qp.begin(self)

self.drawRectangles(qp, event.rect()) # given the button dimension

qp.end()

print('paint: ', self.enter)

def drawRectangles(self, qp, rec):

qp.setBrush(QtGui.QColor(000, 000, 255, 255/4)) # optical

qp.drawRect(rec.left(), rec.top(), rec.width(), rec.height())

16.10. QStackedWidget

[qstackedwidget] How to use QStackedWidget

# create stacked widget

self.item_class_stack_widget = QStackedWidget(self)

self.lst_item_widget = list()

for i in range(0, 10):

item_widget = xxxWidget()

self.item_class_stack_widget.addWidget(item_widget)

self.lst_item_widget.append(item_widget)

# switch to the widget

self.item_class_stack_widget.setCurrentIndex(index)

16.11. QTabWidget

[QTabWidget, example] An example of QTabWidget for Config panel (view)

[QTabWidget, tab, height] How to get the QTabBar height for a QTabWidget? (ref)

tabs = QTabWidget()

...

H = tabs.tabBar().size().height()

[QTabWidget, event] How to catch the current changed event (ref)

tabs = QTabWidget()

tabs.currentChanged.connect(self.OnCurrentChanged)

def OnCurrentChanged(self):

print('Current Changed')

cur_index = self.tabs.currentIndex()

cur_tab = self.lst_tab[cur_index]

print('cur_tab {} changed'.format(self.tabs.tabText(cur_index))

[QTabWidget, index] How to get the current tab index

cur_index = self.tabs.currentIndex()

[QTabWidget, index, switch] How to switch the Tab (ref)

Key:

tabs = QtWidget()

tabs.setCurrentIndex(index)

16.12. QTableWidget

[QTableWidget, create] How to create a QTableWidget (view) (ref)

Quick

widget_table = QTableWidget()

widget_table.setColumnCount(int_col) # 不指定不會顯示

widget_table.setRowCount(int_row) # 不指定不會顯示

# 指定 header

widget_table.setHorizontalHeaderLabels( ['name', 'text book', 'course'])

item = QTableWidgetItem('井民全')

widget_table.setItem(0, 0, item) # y, x, item

[QtableWidget, refresh] How to refresh the QTableWidget column width to the contents (blog)

如果想自動根據資料的長度, 調整 QTableWidget 欄位的寬度,

widget_table.resizeColumnsToContents()

[QTableWidget, Item] How to update the Table content (view)

tem = QTableWidgetItem(str(data))

widget_table.setItem(row, col + int_col_start, item)

[QTableWidget, Item] How to select the Table Item

item_new = widget_table.item(widget_table.rowCount()-1, 0)

if item_new is None:

msg = '[Error] item_new not found.'

return 0, msg

item_new.setSelected(1)

[QTableWidget, header, col, name] How to setup the column name? (view)

self.tableWidget.setHorizontalHeaderLabels(['a', 'b', 'c'])

[QTableWidget, header, style, font] How to change the header font size (ref)

Code

QHeaderView { font-size: 42pt; }

[QTableWidget, header, style] How to set the font and size for a specified table column (ref)

from PyQt5.QtGui import QFont

from PyQt5.QtCore import QSize

table_wig = QTableWidget()

# table_wig.setRowCount(2)

table_wig.setColumnCount(len(self.lan_table_head))

self.set_table_col_head(table_wig)

table_wig.cellChanged.connect(self.on_changed)

table_wig.setAlternatingRowColors(1)

# font

font = QFont()

font.setFamily("Arial Black")

font.setPointSize(15)

table_wig.horizontalHeaderItem(0).setFont(font)

# header size

from PyQt5.QtWidgets import QAbstractScrollArea

from PyQt5.QtWidgets import QHeaderView

table_wig.setSizeAdjustPolicy(QAbstractScrollArea.AdjustToContents)

header = table_wig.horizontalHeader()

header.setSectionResizeMode(0, QHeaderView.ResizeToContents)

[QTableWidget, scroll] How to scroll to the button (ref)

widget_table.scrollToBottom()

[QTableWidget, checkbox] How to add a checkbox to table (ref)

from PyQt5.QtWidgets import QCheckBox

# setCellWidget 與 setItem 不同, setCellWidget 不會加入新的 item 給 Table*

item = QTableWidgetItem('pseudo item')

widget_table.setItem(row, i, item)

chkBoxItem = QCheckBox('test')

widget_table.setCellWidget(row, i, chkBoxItem)

# Note: *

# Verification:

# item_new = widget_table.item(row, i) # get item will return None.

[QTableWidget, qcheckbox] How to customized the checkbox that located in the QTable

# get cell widget from table

chkbox = self.table_wig.cellWidget(y, 0)

if chkbox.isChecked():

# set cell widget to the table

def ui_add_chkbox(self, y, x):

# chkBoxItem = QTableWidgetItem()

# chkBoxItem.setFlags(Qt.ItemIsUserCheckable | Qt.ItemIsEnabled)

# chkBoxItem.setCheckState(Qt.Unchecked)

# chkBoxItem.setSizeHint(QSize(15, 300))

# self.table_wig.setItem(y, x, chkBoxItem)

# new method

from PyQt5.QtWidgets import QCheckBox

chkBoxItem = QCheckBox('test')

chkBoxItem.stateChanged.connect(self.on_checkbox_clicked)

chkBoxItem.setStyleSheet(

"QCheckBox::indicator:unchecked {width: 25px;height: 25px;} " +

"QCheckBox::indicator:checked {width: 30px;height: 30px; background-color: orange;} " +

"QCheckBox::indicator:unchecked:hover {width: 30px;height: 30px; } "

)

self.table_wig.setCellWidget(y, x, chkBoxItem)

def on_checkbox_clicked(self, state):

if state == Qt.Checked:

print('checked')

else:

print('non checked')

[QTableWidget, focus] How to focus the latest item

last_item = self.table.item(self.table.rowCount() - 1 , 1)

self.table.scrollToItem(last_item)

[QTableWidget, scrollbar, width] How to enlarge the width for the vertical scrollbar (ref)

table_wig.setStyleSheet("""

QTableView::item:hover:!selected {

background: yellow;

border: 1px solid blue;

}

QScrollBar:vertical { width: 100px; }

"""

)

[cell, style] How to enable and setup the alternative color for a QTableWidget?

Code

table_wig.setAlternatingRowColors(1)

Style

QTableView {

font-size: 24px;

color: white;

alternate-background-color: gray;

/* background: #e8f4fc; */

}

[cell, style, color, focus] How to change the cell color when the mouse hover? (ref)

table_wig = QTableWidget()

# hover color

table_wig.setStyleSheet("""

QTableView::item:hover:!selected {

background: yellow;

border: 1px solid blue;

}

"""

)

[cell, style, bg-color] How to setup the QTableItemWidget bg color style?

Code

app.setStyleSheet("QTableView::item { background-color: yellow }")

[cell, style, font, size] How to setup the QTableItemWidget font size?

Code

app.setStyleSheet("QTableView{ font-size: 24px; }")

[cell, window, size] How to resize the QTableItemWidget width to fit the QTableWidget size? (ref)

from PyQt5.QtWidgets import QAbstractScrollArea

from PyQt5.QtWidgets import QHeaderView

table_wig = QTableWidget()

table_wig.setSizeAdjustPolicy(QAbstractScrollArea.AdjustToContents)

#table_wig.resizeColumnsToContents()

header = table_wig.horizontalHeader()

header.setSectionResizeMode(0, QHeaderView.ResizeToContents)

header.setSectionResizeMode(1, QHeaderView.ResizeToContents)

16.13. QMainWindow

16.14. QMenu

[QMenu] How to add a memu to your application? (ref)

# menu

mainMenu = self.menuBar()

mainMenu.setNativeMenuBar(False) # Mac OS treats menu differently. Get a consist outcome

fileMenu = mainMenu.addMenu('File')

editMenu = mainMenu.addMenu('Edit')

viewMenu = mainMenu.addMenu('View')

searchMenu = mainMenu.addMenu('Search')

toolsMenu = mainMenu.addMenu('Tools')

helpMenu = mainMenu.addMenu('Help')

exitButton = QAction(QIcon('exit24.png'), 'Exit', self)

exitButton.setShortcut('Ctrl+Q')

exitButton.setStatusTip('Exit application')

exitButton.triggered.connect(self.close)

fileMenu.addAction(exitButton)

[QAction] How to get the action by given its name (view)
[QAction] How to trigger a menu action

exitButton = QAction(QIcon('exit24.png'), 'Exit', self)

exitButton.trigger()

16.15. QMessagebox

[messagebox, question] How to create a question message box (ref)

from PySide2 import QtCore, QtWidgets, QtGui

from PySide2.QtWidgets import QDesktopWidget, QMessageBox

---

reply = QMessageBox.question(self, 'Continue?',

'<html style="font-size:16pt;"> 是否要展示自動填入隊員資料? </html>', QMessageBox.Yes, QMessageBox.No)

if reply == QMessageBox.Yes:

self.dict_arg['auto_fill_member_list_at_start_for_demo'] = True

else:

self.dict_arg['auto_fill_member_list_at_start_for_demo'] = False

---

[message, warning] How to create a warning message box

from PyQt5.QtWidgets import QMessageBox

QMessageBox.warning(None, 'Warning', 'Error_Printer_Exception')

[message, info] How to show an information message box

QtWidgets.QMessageBox.information(self, 'title', 'msg', QtWidgets.QMessageBox.Yes)

16.16. QLabel

[label, create] How to create a QLabel

label = QtWidgets.QLabel('{:<10}'.format(text)) #(text)

[label, style] How to setup the QLabel style

self.label.setStyleSheet('font-size: 20px;')

[label, circle] How to draw a circle label (ref)

Key:

Size = 100

Border-radius = 50px

16.17. QLineEdit

[qlineedit, create] How to create an QLineEdit for passwd

txtValue = QtWidgets.QLineEdit(self)

txtValue.setEchoMode(QtWidgets.QLineEdit.Password)

[QLineEdit, hint] How to makes the line edit display a grayed-out placeholder text

from PyQt5.QtWidgets import QLineEdit

edit = QLineEdit()

edit.setPlaceholderText('Search')

[QLineEdit, textEdited] How to handle the textEdited event?

from PyQt5.QtWidgets import QLineEdit

edit = QLineEdit()

edit.textEdited.connect(self.__on_textEdited__)

def __on_textEdited__(self, text):

print('text edited')

[QLineEdit, editingFinished] How to handle the editingFinished signal

label_value = QLineEdit()

label_value.setReadOnly(int_readonly)

label_value.editingFinished.connect(self.__fn_callback_on_finished__)

def __fn_callback_on_finished__(self):

print('self.sender() = ', self.sender().text())

16.18. QComBox

[combox, create]

cb_date = QComboBox()

cb_date.addItems([str(x) for x in range(1, 31)])

[QComboBox, hint] How to makes the line edit display a grayed-out placeholder text
[QComboBox, textEdited] How to handle the textEdited event?

Code:

edit = QComboBox()

edit.setEditable(1)

edit.lineEdit().setPlaceholderText('Search')

edit.lineEdit().textEdited.connect(self.__on_textEdited__)

def __on_textEdited__(self, text):

print('text edited')

Result:

16.19. QDialog

[qdialog, create] How to create a Modaless QDialog window (ref)

A Modeless Dialog is a dialog that operates independently of other windows. And a Modal Dialog is a dialog that blocks input to other visible windows in the same application.

Key

# (a) modeless dialog (non-blocks)

dlg= MyDlg()

dlg.show()

# (b) modal dialog (blocks)

dlg = MyDlg()

result = dlg.exec_() # no return until dialog closed

print('accept:', result == QDialog.Accepted)

dlg = MyDlg()

dlg.setModal(1)

dlg.show() # return immediatly

Modeless Dialog: that operates independently of other windows

import sys

from PyQt5.QtWidgets import QDialog, QApplication, QWidget

from UI_ItemMgrWidget import UI_ItemMgrWidget

class MyWidget(QWidget):

def __init__(self, parent = None):

super(MyWidget, self).__init__(parent)

...

class myDlg(QDialog):

def __init__(self):

super().__init__()

self.item_mgr = MyWidget(self)

# OK and Cancel buttons

self.buttons = QDialogButtonBox(

QDialogButtonBox.Ok | QDialogButtonBox.Cancel,

Qt.Horizontal, self)

self.buttons.accepted.connect(self.accept)

self.buttons.rejected.connect(self.reject)

self.layout.addWidget(self.buttons)

if __name__ == '__main__':

app = QApplication(sys.argv)

dlg = myDlg()

dlg.show()

sys.exit(app.exec_())

Modal Dialog: that blocks input to other visible windows in the same application

import sys

from PyQt5.QtWidgets import *

from PyQt5.QtCore import Qt

class MyWidget(QWidget):

def __init__(self, parent = None):

super(MyWidget, self).__init__(parent)

class MyDialog(QDialog):

def __init__(self, parent=None):

super().__init__()

self.layout = QVBoxLayout(self)

myWidget = MyWidget(self)

self.layout.addWidget(myWidget)

# OK and Cancel buttons

buttons = QDialogButtonBox(

QDialogButtonBox.Ok | QDialogButtonBox.Cancel,

Qt.Horizontal, self)

buttons.accepted.connect(self.accept)

buttons.rejected.connect(self.reject)

self.layout.addWidget(buttons)

self.resize(640, 480)

@classmethod

def ask(cls, str_title):

dlg = cls()

return dlg.exec_()

def main():

app = QApplication(sys.argv)

if MyDialog().ask('test dlg').exec():

print('OK')

else:

print('Cancel')

if __name__ == '__main__':

main()

[qdialog, bt, size] How to change the button size in a dialog

# OK and Cancel buttons

self.btDlg = QDialogButtonBox(

QDialogButtonBox.Ok | QDialogButtonBox.Cancel,

Qt.Horizontal, self)

self.btDlg.accepted.connect(self.accept)

self.btDlg.rejected.connect(self.reject)

# change the style

lst_bts = self.btDlg.buttons()

for bt in lst_bts:

bt.setMinimumSize(200, 100)

[qdialog, bt, text] How to change qdialog OK button text (ref)

msgBox = QMessageBox(None)

msgBox.setWindowTitle('確定')

msgBox.setText(self.lan['dlg_msg_yes_no_update'])

msgBox.setStandardButtons(QMessageBox.No | QMessageBox.Yes)

msgBox.setDefaultButton(QMessageBox.Yes)

msgBox.button(QMessageBox.Yes).setText('是')

msgBox.button(QMessageBox.Yes).setMinimumSize(100, 50)

msgBox.button(QMessageBox.No).setText('否')

msgBox.button(QMessageBox.No).setMinimumSize(100, 50)

msgBox.exec()

[qdialog, bt, override] How to override the accept method for QDialog (ref)

class MyDlgClass(QDialog):

super().__init__()

self.item_mgr = MyWidget(self)

# OK and Cancel buttons

self.buttons = QDialogButtonBox(

QDialogButtonBox.Ok | QDialogButtonBox.Cancel,

Qt.Horizontal, self)

self.buttons.accepted.connect(self.accept)

self.buttons.rejected.connect(self.reject)

self.layout.addWidget(self.buttons)

@pyqtSlot()

def accept(self):

self.thread_out = 1

super(MyDlgClass, self).accept()

@pyqtSlot()

def reject(self):

self.thread_out = 1

super(MyDlgClass, self).reject()

17. MongoDb

[doc] A tutorial for MongoDb (ref)
[db, check] How to check if database exists (ref)

client = MongoClient()

dbnames = client.list_database_names()

if 'mydbname' in dbnames:

print "It's there!"

[db, check] How to check if data exists

def check_exist(self, col_name, key, value):

col = self.client[self.db_name][col_name]

dict_rd = col.find_one()

if dict_rd[key] == value:

return 1, 1, 'found'

else:

return 1, 0, 'not found'

[db, add] How to add a record to database

db_name = db_name

db_url = db_url # 'mongodb://localhost:27017/'

client = pymongo.MongoClient(db_url)

col = client[db_name][col_name]

dict_data = {'name': 'rd2', 'passwd':'5678', 'role': 'admin'}

col.insert_one(dict_data)

[db, del] How to remove a recrod from database

db_name = db_name

db_url = db_url # 'mongodb://localhost:27017/'

client = pymongo.MongoClient(db_url)

col = client[db_name][col_name]

# dict_data = {'name': 'rd2', 'passwd':'5678', 'role': 'admin'} # full data

dict_data = {'name': 'rd2'} # 可以接收部分資料的索引

col.delete_one(dict_data)

[db, update] How to update a record

db_name = db_name

db_url = db_url # 'mongodb://localhost:27017/'

client = pymongo.MongoClient(db_url)

col = client[db_name][col_name]

# dict_data = {'name': 'rd2', 'passwd':'1234', 'role': 'admin'}

dict_data = {'name': 'rd2'} # 可以接收部分資料的搜尋

dict_data_new = {'$set': {'name': 'rd2', 'passwd':'5678', 'role': 'admin'}} # 可以接收部分資料value的更新

col.update_one(dict_data, dict_data_new, upsert=False)

[db, replace] How to replace a record

db_name = db_name

db_url = db_url # 'mongodb://localhost:27017/'

client = pymongo.MongoClient(db_url)

col = client[db_name][col_name]

# dict_data = {'name': 'rd2', 'passwd':'1234', 'role': 'admin'}

dict_data = {'name': 'rd2'} # 可以接收部分資料的搜尋

dict_data_new = {'name': 'rd2', 'passwd':'5678', 'role': 'admin'}

col.replace_one(dict_data, dict_data_new, upsert=False)

18. Web Automation

Examples (ref)
[find, caption] find element by caption

driver.find_element_by_xpath('//*[@title="雪霸國家公園"]').click()

[browser, maximum] How to maximum the browser

options = webdriver.ChromeOptions()

options.add_argument('--start-maximized')

self.driver = webdriver.Chrome(chrome_options=options)

[browser, scroll] Auto scroll the element to visiable (ref)

Method 1:

actions = ActionChains(self.driver)

actions.move_to_element(element).perform()

Method 2:

self.driver.execute_script("arguments[0].scrollIntoView();", element)

[webelement, scroll] How to scrll the web element to the center (ref)

str_script_center = 'var viewPortHeight = Math.max(document.documentElement.clientHeight, window.innerHeight || 0);' \

+ 'var elementTop = arguments[0].getBoundingClientRect().top;' \

+ 'window.scrollBy(0, elementTop-(viewPortHeight/2));'

self.driver.execute_script(str_script_center, element)

[ui-datepicker] How to setup yyyy-mm-date

self.click_id(strID)

time.sleep(self.delay_ui)

element = self.driver.find_element_by_class_name('ui-datepicker-year')

sele = Select(element)

sele.select_by_index(int(yyyy)-1929) # 0: 1929

time.sleep(self.delay_ui)

element = self.driver.find_element_by_class_name('ui-datepicker-month')

sele = Select(element)

sele.select_by_index(int(mm)) # 0: 一月

elements = self.driver.find_elements_by_xpath(".//*[@id='ui-datepicker-div']/table/tbody/tr/td/a")

for dates in elements:

if(dates.is_enabled() and dates.is_displayed() and str(dates.get_attribute("innerText")) == str(date)):

dates.click()

[wait] How to wait for ready (ref)

def _get_elm_id(self, strID):

#ele = self.driver.find_element_by_id(strID)

#return ele

from selenium.webdriver.common.by import By

from selenium.webdriver.support.ui import WebDriverWait

from selenium.webdriver.support import expected_conditions as EC

element = WebDriverWait(self.driver, 10).until(

EC.presence_of_element_located((By.ID, strID)) and

EC.element_to_be_clickable((By.ID, strID))

)

return element

[pop, alert] How to handle alrt window

def handle_alert_popup(self):

# wait for alert window

# prevent:

# Message: unexpected alert open: {Alert text :

from selenium.webdriver.common.by import By

from selenium.webdriver.support.ui import WebDriverWait

from selenium.webdriver.support import expected_conditions as EC

element = WebDriverWait(self.driver, 10).until(

EC.alert_is_present()

)

# accept

alert = self.driver.switch_to_alert()

alert.accept()

[html, parsing] BeautifulSoap -- 網頁剖析 Html (ref)
[WebElement, get value] How to get the WebElement value after sending keys

Key:

element.get_attribute('value')

Example

def _fill_text_verify(self, dict_arg):

strID = dict_arg['strID']

strText = dict_arg['strText']

element = self._get_elm_id(strID)

self.scroll_to_element(element)

if element.get_attribute('value') == strText:

return 1, 'ok'

else:

return 0, 'Not Matched: element.text = {}, strText = {}\n'.format(element.text, strText)

[WebElement, select all] How to clean the text first and then input text? (ref)

from selenium.webdriver.common.keys import Keys

element.send_keys(Keys.CONTROL, 'a')

element.send_keys(strText)

19. Optimal

[opt, time] 如何找出最需要優化的模組 (ref)(doc)(blogger)

Quick Guide

1. 執行 python3 -m cProfile -s cumtime main.py > profile.txt

2. 觀察 profile.txt 中, cumtime 排序最高的 module

3. 優化這個 module

4. goto 1

作法

Step 1: Install Cprofile

conda install -c indico cprofilev

Step 2: Run Cprofile to your code

python3 -m cProfile -s cumtime main.py > profile.txt # sorting by cumtime

Step 3: Analyzing the code

Open profile.txt

欄位說明:

ncalls: 執行次數

tottime: 這段 code 執行的時間 (不含呼叫子程式)

percall: = tottime/ncalls
cumtime: 這段 code 累積執行時間 (含呼叫子程式), 真正的執行時間

percall: = cumtime/ncalls

filename:lineno(function)

Ex:

3121219 function calls (3111644 primitive calls) in 2.214 seconds

ncalls tottime percall cumtime percall filename:lineno(function)

120 0.014 0.000 0.322 0.003 P.py:30(<listcomp>)

120 0.014 0.000 0.320 0.003 P.py:41(<listcomp>)

分析示範

1. 全部 3121219 個函式呼叫, 總共花了 2.214 秒

2. 查看最多 cumtime 的 fun:
P.py: 30, 這段程式在 listcomp 函式, 累積執行了 0.322 sec (cumtime). 其中 percall = 0.322 / 120 = 0.003 秒. 佔了 15% 的執行時. ==> 有必要優化

2021年9月3日 星期五

[python, main] Python FAQ

1. Coding Style & Tutorial

2. Utility

3. Pylint

4. Pytest

5. Transfer the script to exe

6. Tox

7. Environment

8. Package

8.1. Conda

8.2. pip

8.3. Import

8.4. Packaging to library

9. Language

9.1. Comment

10. Enum

10.1. Variable

10.2. Scope

10.3. Module

10.4. Class

10.5. Time

10.6. String

10.7. Exception

10.8. File I/O

10.9. Network

10.10. Device I/O

10.11. Function

10.12. Condition

10.13. Loop

10.14. boolean

10.15. List

10.16. Tuple

10.17. Set

10.18. Dict

10.19. Panda, DataFrame

10.20. Array

10.20.1. Definition

10.20.2. Create 1d

10.20.3. Create 2d

10.20.4. Basic OPs

10.21. List comprehension

10.22. Generator

10.23. Serial

10.24. Plot

10.25. Process/Thread

10.26. Async

10.27. Google

10.28. Others

11. Image Processing

11.1. Library

11.2. FAQ

12. Signal Processing

12.1. KB

12.2. Filter

12.3. Blind Source Separation

12.4. Noise

12.5. I/O

12.6. Find Peak

12.7. Others

13. Math

14. Authentication

15. Debug

16. PyQt5

16.1. Signal & Slot

16.2. Layout

16.3. Style, Skin

16.4. Drag & Drop

16.5. Cursor

16.6. Digital

16.7. Console

16.8. QWidget

16.9. QPushButton

16.10. QStackedWidget

16.11. QTabWidget

16.12. QTableWidget

16.13. QMainWindow

16.14. QMenu

16.15. QMessagebox

16.16. QLabel

2021年9月3日星期五