Holger Frey
8169daeb89
Added parser for the assay results xml file
...
The `sensospot_parser.parse_folder()` function now tries to parse the xml file first and will fall back to parsing csv files if an error occurs
2 years ago
Holger Frey
0a1da70021
added more example data for comparing xml to csv parsing
2 years ago
Holger Frey
9de0a09557
added more example xml data for testing
2 years ago
Holger Frey
819358977e
fixed test data for upcoming module xml_parser
2 years ago
Holger Frey
dd476bb19a
added more calls to "clenup_data_columns()" to remove double index columns after merges
2 years ago
Holger Frey
8c591e9f3c
removed debug print statement
2 years ago
Holger Frey
988c7562d9
renamed some functions in "csv_parser" module to have more explicit names
2 years ago
Holger Frey
8644a25d78
moved _cleanup_data_columns() function to "columns" module
...
by moving this function it can be easily reused in the upcoming "xml_parser" module
2 years ago
Holger Frey
99d0e0fa4e
renamed test cases and added a new one for xml parsing
2 years ago
Holger Frey
938b063069
removed the "recordtime" module
...
The "recordtime" module uses the assay result xml file for retrieving the Analyses.Datetime value.
This value is only available in the assay result xml.
Since the "parse_csv" module will only be a backup if this xml file is not present, a separate parser for the Analysis.Datetime is not needed anymore
2 years ago
Holger Frey
873091504c
renamed the "parser" module to "csv_parser"
...
The "csv_parser" module will be used as a backup if the analysis xml file is not present.
2 years ago
Holger Frey
1c259b49df
the `quiet` parameter is passed from the cli to the function
2 years ago
Holger Frey
791ad62398
cleaned up the cli interface
...
I was annoyed by the cli interface quite some time. It should now adhere more to what is expected from a POSIX cli
2 years ago
Holger Frey
c1bba778b9
removed debugging output
2 years ago
Holger Frey
78c5db031a
Added image name and recording datetime to the results
...
This two columns where added:
- Analysis.Image
- Analysis.Datetime
When parsing a folder, the image names and (if available) the date and time of the recording is added to the resulting data frame.
If the recording datetime is not available, the resulting column will contain NaN
2 years ago
cc133f1873
added build instructions for documentation
2 years ago
74920764e7
added mkdocs for documentation
2 years ago
6c749e0843
finally separate flake8 config file removed
2 years ago
efc1f2a043
removing .flake8 again for testing new precommit setup
2 years ago
dd04b0ece6
python3 -m venv --prompt sensospot .venv
...
.venv/bin/pip3 install --upgrade pip
Requirement already satisfied: pip in ./.venv/lib/python3.10/site-packages (22.2.2)
.venv/bin/pip3 install "flit>3.2"
Requirement already satisfied: flit>3.2 in ./.venv/lib/python3.10/site-packages (3.7.1)
Requirement already satisfied: docutils in ./.venv/lib/python3.10/site-packages (from flit>3.2) (0.19)
Requirement already satisfied: requests in ./.venv/lib/python3.10/site-packages (from flit>3.2) (2.28.1)
Requirement already satisfied: tomli-w in ./.venv/lib/python3.10/site-packages (from flit>3.2) (1.0.0)
Requirement already satisfied: flit_core>=3.7.1 in ./.venv/lib/python3.10/site-packages (from flit>3.2) (3.7.1)
Requirement already satisfied: tomli in ./.venv/lib/python3.10/site-packages (from flit>3.2) (2.0.1)
Requirement already satisfied: idna<4,>=2.5 in ./.venv/lib/python3.10/site-packages (from requests->flit>3.2) (3.3)
Requirement already satisfied: certifi>=2017.4.17 in ./.venv/lib/python3.10/site-packages (from requests->flit>3.2) (2022.6.15)
Requirement already satisfied: charset-normalizer<3,>=2 in ./.venv/lib/python3.10/site-packages (from requests->flit>3.2) (2.1.1)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in ./.venv/lib/python3.10/site-packages (from requests->flit>3.2) (1.26.12)
.venv/bin/flit install --pth-file
[?1049h[1;28r(B[m[4l[?7h[28;1H[?1049l
[?1l>Requirement already satisfied: pandas>=1.0.0 in ./.venv/lib/python3.10/site-packages (from -r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 1)) (1.4.3)
Requirement already satisfied: defusedxml>=0.6.0 in ./.venv/lib/python3.10/site-packages (from -r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 2)) (0.7.1)
Requirement already satisfied: tables>=3.6.1 in ./.venv/lib/python3.10/site-packages (from -r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 3)) (3.7.0)
Requirement already satisfied: click in ./.venv/lib/python3.10/site-packages (from -r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 4)) (8.1.3)
Requirement already satisfied: pytest>=4.0.0 in ./.venv/lib/python3.10/site-packages (from -r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 5)) (7.1.2)
Requirement already satisfied: pytest-cov in ./.venv/lib/python3.10/site-packages (from -r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 6)) (3.0.0)
Requirement already satisfied: pytest-mock in ./.venv/lib/python3.10/site-packages (from -r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 7)) (3.8.2)
Requirement already satisfied: pytest-randomly>=3.5.0 in ./.venv/lib/python3.10/site-packages (from -r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 8)) (3.12.0)
Requirement already satisfied: tox in ./.venv/lib/python3.10/site-packages (from -r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 9)) (3.25.1)
Requirement already satisfied: black in ./.venv/lib/python3.10/site-packages (from -r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 10)) (22.6.0)
Requirement already satisfied: flake8 in ./.venv/lib/python3.10/site-packages (from -r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 11)) (5.0.4)
Requirement already satisfied: flake8-comprehensions in ./.venv/lib/python3.10/site-packages (from -r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 12)) (3.10.0)
Requirement already satisfied: flake8-bandit in ./.venv/lib/python3.10/site-packages (from -r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 13)) (4.1.1)
Requirement already satisfied: isort>=5.0.0 in ./.venv/lib/python3.10/site-packages (from -r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 14)) (5.10.1)
Requirement already satisfied: keyring in ./.venv/lib/python3.10/site-packages (from -r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 15)) (23.8.2)
Requirement already satisfied: pre-commit in ./.venv/lib/python3.10/site-packages (from -r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 16)) (2.20.0)
Requirement already satisfied: python-dateutil>=2.8.1 in ./.venv/lib/python3.10/site-packages (from pandas>=1.0.0->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 1)) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in ./.venv/lib/python3.10/site-packages (from pandas>=1.0.0->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 1)) (2022.2.1)
Requirement already satisfied: numpy>=1.21.0 in ./.venv/lib/python3.10/site-packages (from pandas>=1.0.0->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 1)) (1.23.2)
Requirement already satisfied: numexpr>=2.6.2 in ./.venv/lib/python3.10/site-packages (from tables>=3.6.1->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 3)) (2.8.3)
Requirement already satisfied: packaging in ./.venv/lib/python3.10/site-packages (from tables>=3.6.1->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 3)) (21.3)
Requirement already satisfied: tomli>=1.0.0 in ./.venv/lib/python3.10/site-packages (from pytest>=4.0.0->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 5)) (2.0.1)
Requirement already satisfied: attrs>=19.2.0 in ./.venv/lib/python3.10/site-packages (from pytest>=4.0.0->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 5)) (22.1.0)
Requirement already satisfied: pluggy<2.0,>=0.12 in ./.venv/lib/python3.10/site-packages (from pytest>=4.0.0->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 5)) (1.0.0)
Requirement already satisfied: iniconfig in ./.venv/lib/python3.10/site-packages (from pytest>=4.0.0->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 5)) (1.1.1)
Requirement already satisfied: py>=1.8.2 in ./.venv/lib/python3.10/site-packages (from pytest>=4.0.0->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 5)) (1.11.0)
Requirement already satisfied: coverage[toml]>=5.2.1 in ./.venv/lib/python3.10/site-packages (from pytest-cov->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 6)) (6.4.4)
Requirement already satisfied: toml>=0.9.4 in ./.venv/lib/python3.10/site-packages (from tox->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 9)) (0.10.2)
Requirement already satisfied: filelock>=3.0.0 in ./.venv/lib/python3.10/site-packages (from tox->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 9)) (3.8.0)
Requirement already satisfied: virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0 in ./.venv/lib/python3.10/site-packages (from tox->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 9)) (20.16.3)
Requirement already satisfied: six>=1.14.0 in ./.venv/lib/python3.10/site-packages (from tox->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 9)) (1.16.0)
Requirement already satisfied: mypy-extensions>=0.4.3 in ./.venv/lib/python3.10/site-packages (from black->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 10)) (0.4.3)
Requirement already satisfied: platformdirs>=2 in ./.venv/lib/python3.10/site-packages (from black->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 10)) (2.5.2)
Requirement already satisfied: pathspec>=0.9.0 in ./.venv/lib/python3.10/site-packages (from black->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 10)) (0.9.0)
Requirement already satisfied: mccabe<0.8.0,>=0.7.0 in ./.venv/lib/python3.10/site-packages (from flake8->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 11)) (0.7.0)
Requirement already satisfied: pyflakes<2.6.0,>=2.5.0 in ./.venv/lib/python3.10/site-packages (from flake8->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 11)) (2.5.0)
Requirement already satisfied: pycodestyle<2.10.0,>=2.9.0 in ./.venv/lib/python3.10/site-packages (from flake8->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 11)) (2.9.1)
Requirement already satisfied: bandit>=1.7.3 in ./.venv/lib/python3.10/site-packages (from flake8-bandit->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 13)) (1.7.4)
Requirement already satisfied: identify>=1.0.0 in ./.venv/lib/python3.10/site-packages (from pre-commit->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 16)) (2.5.3)
Requirement already satisfied: pyyaml>=5.1 in ./.venv/lib/python3.10/site-packages (from pre-commit->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 16)) (6.0)
Requirement already satisfied: nodeenv>=0.11.1 in ./.venv/lib/python3.10/site-packages (from pre-commit->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 16)) (1.7.0)
Requirement already satisfied: cfgv>=2.0.0 in ./.venv/lib/python3.10/site-packages (from pre-commit->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 16)) (3.3.1)
Requirement already satisfied: stevedore>=1.20.0 in ./.venv/lib/python3.10/site-packages (from bandit>=1.7.3->flake8-bandit->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 13)) (4.0.0)
Requirement already satisfied: GitPython>=1.0.1 in ./.venv/lib/python3.10/site-packages (from bandit>=1.7.3->flake8-bandit->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 13)) (3.1.27)
Requirement already satisfied: setuptools in ./.venv/lib/python3.10/site-packages (from nodeenv>=0.11.1->pre-commit->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 16)) (58.1.0)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in ./.venv/lib/python3.10/site-packages (from packaging->tables>=3.6.1->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 3)) (3.0.9)
Requirement already satisfied: distlib<1,>=0.3.5 in ./.venv/lib/python3.10/site-packages (from virtualenv!=20.0.0,!=20.0.1,!=20.0.2,!=20.0.3,!=20.0.4,!=20.0.5,!=20.0.6,!=20.0.7,>=16.0.0->tox->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 9)) (0.3.6)
Requirement already satisfied: gitdb<5,>=4.0.1 in ./.venv/lib/python3.10/site-packages (from GitPython>=1.0.1->bandit>=1.7.3->flake8-bandit->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 13)) (4.0.9)
Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in ./.venv/lib/python3.10/site-packages (from stevedore>=1.20.0->bandit>=1.7.3->flake8-bandit->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 13)) (5.10.0)
Requirement already satisfied: smmap<6,>=3.0.1 in ./.venv/lib/python3.10/site-packages (from gitdb<5,>=4.0.1->GitPython>=1.0.1->bandit>=1.7.3->flake8-bandit->-r /var/folders/v5/ws7wg4wx6k76gx3t_ylh29lr0000gn/T/tmpk7adulvkrequirements.txt (line 13)) (5.0.0)
.venv/bin/pre-commit install --install-hooks
pre-commit installed at .git/hooks/pre-commit now installs also pre-commit hooks
2 years ago
da42f2358c
fixes for new findings of flake8
2 years ago
4e1d92ca41
modernized project layout and infrastructure
...
- bumped version to 0.8.0
- requires now at least Python 3.9
- tox tests run in Python verions 3.9 and 3.10
- moved package source to "src" directory
2 years ago
Holger Frey
614a031217
updated readme
3 years ago
Holger Frey
fde3513809
renamed package to 'sensospot_parser'
3 years ago
Holger Frey
4c69ef457b
added type hints and more docs to parser
3 years ago
Holger Frey
79f2f0e3b9
added more documentation
3 years ago
Holger Frey
81db693157
added typing information to module parameters
3 years ago
Holger Frey
64cee96485
simplified adding parsed parameters
3 years ago
Holger Frey
b86971cd5a
made the api import test nicer
3 years ago
Holger Frey
04b2c8b598
renamed test for package api
3 years ago
Holger Frey
38224770ad
renamed column constants
3 years ago
Holger Frey
1fa3c65d50
clarified column sets
3 years ago
Holger Frey
e165648123
added common index columns sets
3 years ago
Holger Frey
4fbb6410b3
cosmetic changes
3 years ago
Holger Frey
c7e8c3455a
parsing an empty folder without data raise a ValueError
3 years ago
Holger Frey
6804ae5cdd
simplified concatenating mulitple dataframes into one
3 years ago
Holger Frey
ad960108e1
fixed shell script name
3 years ago
Holger Frey
9801036911
clarification in readme
3 years ago
Holger Frey
6690b2fddc
added 'Analysis.Filename' to readme
3 years ago
Holger Frey
c03cff6ad4
added option to recurse one level down
3 years ago
Holger Frey
c1f1e2bb0e
added the possibility to redirect parsed data to stdout
3 years ago
Holger Frey
75ac4740a5
fixed sanity check
3 years ago
Holger Frey
e90d3367ea
removed unused column definitions
3 years ago
Holger Frey
2de4859ca8
removed typo
3 years ago
Holger Frey
0290c0a9ba
removed "utils" and "dynamic_range"
...
this two modules should be added to a new project. This project should concentrate on on just parsing the data.
3 years ago
Holger Frey
994a06a8f3
added more tests for apply_map()
3 years ago
Holger Frey
df0bf78b29
added apply_map() api function to readme
3 years ago
Holger Frey
791bc51deb
linted
3 years ago
Holger Frey
9359cc00af
added api function apply_map
...
this also lead to simplifications in _add_measurement_params and apply_exposure_map
3 years ago
Holger Frey
ab3fb1e040
fixed changelog
3 years ago
Holger Frey